Tokenizer

This repo contains C# and Typescript implementation of byte pair encoding(BPE) tokenizer for OpenAI LLMs, it's based on open sourced rust implementation in the OpenAI tiktoken. Both implementation are valuable to run prompt tokenization in .NET and Nodejs environment before feeding prompt into a LLM.

C# implementation

The TokenizerLib is built in .NET Standard 2.0, which can be consumed in projects on any version of .NET later than .NET Core 2.0 or .NET Framework 4.6.1.

You can download and install the nuget package of TokenizerLib here.

Example C# code to use TokenizerLib in your code:

using System.Collections.Generic;
using Microsoft.DeepDev;

var IM_START = "<|im_start|>";
var IM_END = "<|im_end|>";

var specialTokens = new Dictionary<string, int>{
                                            { IM_START, 100264},
                                            { IM_END, 100265},
                                        };
var tokenizer = await TokenizerBuilder.CreateByModelNameAsync("gpt-4", specialTokens);

var text = "<|im_start|>Hello World<|im_end|>";
var encoded = tokenizer.Encode(text, new HashSet<string>(specialTokens.Keys));
Console.WriteLine(encoded.Count);

var decoded = tokenizer.Decode(encoded.ToArray());
Console.WriteLine(decoded);

In production setting, you should pre-download the BPE rank file and call TokenizerBuilder.CreateTokenizer API to avoid downloading the BPE rank file on the fly. You can find the model to encoder and encoder to BPE rank file link mapping in: TokenizerBuilder.cs.

C# performance benchmark

PerfBenchmark result based on PerfBenchmark.csproj:

BenchmarkDotNet=v0.13.3, OS=Windows 11 (10.0.22621.1702)
Intel Core i7-1065G7 CPU 1.30GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=7.0.300-preview.23179.2
  [Host]     : .NET 6.0.16 (6.0.1623.17311), X64 RyuJIT AVX2
  DefaultJob : .NET 6.0.16 (6.0.1623.17311), X64 RyuJIT AVX2

| Method |    Mean |    Error |   StdDev |
|------- |--------:|---------:|---------:|
| Encode | 2.414 s | 0.0303 s | 0.0253 s |

Microsoft/Microsoft.DeepDev.TokenizerLibv1.3.3

Get Started

Readme

Tokenizer

C# implementation

C# performance benchmark

Typescript implementation

Contributing

Trademarks