13 packages tagged with “tiktoken”
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken.
Provide tokenizers to allow counting content tokens for text and embeddings
Anthropic Claude BPE Tokenizer unofficial implementation
Open AI Chat Completion Models (GPT 3.5/GPT 4) BPE Tokenizer unofficial implementation
Package Description
The fastest tokenizer for gpt2, multilingual and whisper inspired by Tiktoken.
The fastest tokenizer for GPT-3.5 and GPT-4 inspired by Tiktoken. This fork contains a printable version of Explore()