The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
Get Started
$ dotnet add package TiktokenReadme
Tiktoken
Due to the lack of a C# version of cl100k_base encoding (gpt-3.5-turbo), I have implemented a basic solution with encoding and decoding methods based on the official Rust implementation.
Currently, cl100k_base p50k_base has been implemented. Other encodings will be added in future submissions. If you encounter any issues or have questions, please feel free to submit them on the lssues."
Usage
using Tiktoken;
var encoding = Encoding.ForModel("gpt-3.5-turbo");
var tokens = encoding.Encode("hello world"); // [15339, 1917]
var text = encoding.Decode(tokens); // hello world
var encoding = Encoding.Get("cl100k_base");
var tokens = encoding.Encode("hello world"); // [15339, 1917]
var text = encoding.Decode(tokens); // hello world
Once the download is successful, subsequent uses will not require downloading again. You can set before using the encoder to modify the storage path of the downloaded tiktoken files, or you can pre-download the files to avoid network issues causing download failures.
<!--BENCHMARKS_START--> <!--BENCHMARKS_END-->