The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
Get Started
$ dotnet add package TiktokenReadme
Tiktoken
This implementation aims for maximum performance, especially in the token count operation.
There's also a benchmark console app here for easy tracking of this.
We will be happy to accept any PR.
Implemented encodings
cl100k_baser50k_basep50k_basep50k_edit
Usage
var encoding = Tiktoken.Encoding.ForModel("gpt-4");
var tokens = encoding.Encode("hello world"); // [15339, 1917]
var text = encoding.Decode(tokens); // hello world
var numberOfTokens = encoding.CountTokens(text); // 2
var stringTokens = encoding.Explore(text); // ["hello", " world"]
var encoding = Tiktoken.Encoding.Get(Encodings.P50KBase);
var tokens = encoding.Encode("hello world"); // [31373, 995]
var text = encoding.Decode(tokens); // hello world
Benchmarks
You can view the reports for each version here
<!--BENCHMARKS_START-->
BenchmarkDotNet v0.13.12, macOS Sonoma 14.2.1 (23C71) [Darwin 23.2.0]
Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), Arm64 RyuJIT AdvSIMD
DefaultJob : .NET 8.0.0 (8.0.23.53103), Arm64 RyuJIT AdvSIMD
| Method | Categories | Data | Mean | Median | Ratio | Gen0 | Gen1 | Gen2 | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|---|---|---|
| SharpTokenV1_2_16_ | CountTokens | 1. (...)57. [19866] | 1,554,552.0 ns | 1,552,769.4 ns | 1.00 | 292.9688 | 146.4844 | - | 1846147 B | 1.00 |
| TiktokenSharpV1_0_9_ | CountTokens | 1. (...)57. [19866] | 1,242,157.7 ns | 1,241,657.7 ns | 0.80 | 253.9063 | 117.1875 | 3.9063 | 1570786 B | 0.85 |
| TokenizerLibV1_3_3_ | CountTokens | 1. (...)57. [19866] | 815,490.5 ns | 806,761.4 ns | 0.52 | 247.0703 | 98.6328 | 0.9766 | 1547678 B | 0.84 |
| Tiktoken_ | CountTokens | 1. (...)57. [19866] | 311,744.2 ns | 311,591.0 ns | 0.20 | 49.3164 | - | - | 309449 B | 0.17 |
| SharpTokenV1_2_16_ | CountTokens |
Support
Priority place for bugs: https://github.com/tryAGI/LangChain/issues
Priority place for ideas and general questions: https://github.com/tryAGI/LangChain/discussions
Discord: https://discord.gg/Ca2xhfBf3v