The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
Get Started
$ dotnet add package TiktokenReadme
Tiktoken
This implementation aims for maximum performance, especially in the token count operation.
There's also a benchmark console app here for easy tracking of this.
We will be happy to accept any PR.
Implemented encodings
o200k_basecl100k_baser50k_basep50k_basep50k_edit
Usage
using Tiktoken;
var encoder = ModelToEncoder.For("gpt-4o"); // or explicitly using new Encoder(new O200KBase())
var tokens = encoder.Encode("hello world"); // [15339, 1917]
var text = encoder.Decode(tokens); // hello world
var numberOfTokens = encoder.CountTokens(text); // 2
var stringTokens = encoder.Explore(text); // ["hello", " world"]
<!--BENCHMARKS_START-->
<!--BENCHMARKS_END-->