search
Found 21 packages
Tiktoken
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 2.1M / total
chatgptopenaitiktokentokensgpt-4
Tiktoken.Core
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 1.1M / total
chatgptopenaitiktokentokensgpt-4
Tiktoken.Encodings.o200k
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 1.1M / total
chatgptopenaitiktokentokensgpt-4
Tiktoken.Encodings.Abstractions
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 1.1M / total
chatgptopenaitiktokentokensgpt-4
Tiktoken.Encodings.cl100k
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 1.1M / total
chatgptopenaitiktokentokensgpt-4
Microsoft.KernelMemory.AI.Tiktoken✓
Provide tokenizers to allow counting content tokens for text and embeddings
v0.98.250508.3+bd8d34e
↙ 715.3K / total
TiktokenTokenizerRAGKernelMemory
pkg
TiktokenSharp
Token calculation for OpenAI models.
v1.2.1
↙ 561.7K / total
chatgptopenai
pkg
LLMSharp.Anthropic.Tokenizer
Anthropic Claude BPE Tokenizer unofficial implementation
v2.0.3
↙ 125.2K / total
anthropictokenizerclaudedotnettiktoken
Microsoft.ML.Tokenizers.Data.O200kBase✓
The Microsoft.ML.Tokenizers.Data.O200kBase includes the Tiktoken tokenizer data file o200k_base.tiktoken, which is utilized by models such as gpt-4o.
v2.0.0
↙ 1.8M / total
ML.NETMLMachineLearning
Tiktoken.Encodings.r50k
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 3.2K / total
chatgptopenaitiktokentokensgpt-4
Tiktoken.Encodings.p50k
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
v2.2.0
↙ 3.1K / total
chatgptopenaitiktokentokensgpt-4
Microsoft.ML.Tokenizers.Data.Cl100kBase✓
The Microsoft.ML.Tokenizers.Data.Cl100kBase class includes the Tiktoken tokenizer data file cl100k_base.tiktoken, which is utilized by models such as GPT-4.
v2.0.0
↙ 1.5M / total
ML.NETMLMachineLearning
Lofcz.Forks.Tiktoken
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken. This fork contains a printable version of Explore()
v1.1.6
↙ 760 / total
chatgptopenaitiktokentokensgpt-4
pkg
ManySpeech.Tiktoken
The fastest tokenizer for gpt2, multilingual and whisper inspired by Tiktoken.
v1.0.0
↙ 524 / total
chatgptopenaitiktokentokensgpt2
Microsoft.ML.Tokenizers.Data.P50kBase✓
The Microsoft.ML.Tokenizers.Data.P50kBase includes the Tiktoken tokenizer data file p50k_base.tiktoken, which is utilized by models such as text-davinci-002
v2.0.0
↙ 554.8K / total
ML.NETMLMachineLearning
pkg
LLMSharp.OpenAi.Tokenizer
Open AI Chat Completion Models (GPT 3.5/GPT 4) BPE Tokenizer unofficial implementation
v2.0.3
↙ 23.2K / total
openaigpt3.5gpt4
pkg
SemchunkNet.Tiktoken
Semchunk.Net tokenizer wrapper for the Tiktoken package.
v1.0.3
↙ 201 / total
BpeTokenizer
Package Description
v1.0.5
↙ 2.4K / total
BytePairEncodingBPEOpenAItiktoken.NET
Microsoft.ML.Tokenizers.Data.Gpt2✓
The Microsoft.ML.Tokenizers.Data.Gpt2 includes the Tiktoken tokenizer data file gpt2.tiktoken, which is utilized by models such as Gpt-2.
v2.0.0
↙ 14.0K / total
ML.NETMLMachineLearning
Microsoft.ML.Tokenizers.Data.R50kBase✓
The Microsoft.ML.Tokenizers.Data.R50kBase includes the Tiktoken tokenizer data file r50k_base.tiktoken, which is utilized by models such as text-davinci-001
v2.0.0
↙ 13.7K / total
ML.NETMLMachineLearning
page 1next →