Tokenizers.DotNet
Nuget Package list
Requirements
Supported functionalities
How to use
- (1) Install the packages
- (2) Write the code
How to build
- (Recommended) Cross-platform build
- Building with local machine system
Release guide

Tokenizers.DotNet

.NET wrapper of HuggingFace Tokenizers library

Nuget Package list

Package	main	Description
Tokenizers.DotNet		Core library
Tokenizers.DotNet.runtime.win-x64		Native bindings for windows x64
Tokenizers.DotNet.runtime.win-arm64		Native bindings for windows arm64
Tokenizers.DotNet.runtime.linux-x64		Native bindings for linux x64
Tokenizers.DotNet.runtime.linux-arm64		Native bindings for linux arm64
Tokenizers.DotNet.runtime.osx-x64		Native bindings for macOS x64
Tokenizers.DotNet.runtime.osx-arm64		Native bindings for macOS arm64

Requirements

.NET 6 / .NET Standard 2.0 or above
(Build) Latest Rust

Supported functionalities

Download tokenizer files from Hugginface Hub
Load tokenizer file(.json) from local
Encode string to tokens
Decode tokens to string

How to use

(1) Install the packages

From the NuGet, install Tokenizers.DotNet package
And then, install Tokenizers.DotNet.runtime.<OS>-<ARCH> package too (e.a win-x64 or linux-arm64, check Nuget package list above).

(2) Write the code

Check following example code:

using Tokenizers.DotNet;

// Download skt/kogpt2-base-v2/tokenizer.json from the hub
var hubName = "skt/kogpt2-base-v2";
var filePath = "tokenizer.json";
var fileFullPath = await HuggingFace.GetFileFromHub(hubName, filePath, "deps");
Console.WriteLine($"Downloaded {fileFullPath}");

// Create a tokenizer instance
Tokenizer tokenizer;
try
{
    tokenizer = new Tokenizer(vocabPath: fileFullPath);
}
catch (TokenizerException e)
{
	Console.WriteLine(e.Message);
    return;
}
try
{
    var text = "음, 이제 식사도 해볼까요";
    Console.WriteLine($"Input text: {text}");
    var tokens = tokenizer.Encode(text);
    Console.WriteLine($"Encoded: {string.Join(", ", tokens)}");
    var decoded = tokenizer.Decode(tokens);
    Console.WriteLine($"Decoded: {decoded}");
}
catch (TokenizerException e)
{
    Console.WriteLine(e.Message);
    return;
}
Console.WriteLine($"Version of Tokenizers.DotNet.runtime.win: {tokenizer.GetVersion()}");
Console.WriteLine("--------------------------------------------------");

//// Download openai-community/gpt2 from the hub
hubName = "openai-community/gpt2";
filePath = "tokenizer.json";
fileFullPath = await HuggingFace.GetFileFromHub(hubName, filePath, "deps");

// Create a tokenizer instance
Tokenizer tokenizer2;
try
{
    tokenizer2 = new Tokenizer(vocabPath: fileFullPath);
}
catch (TokenizerException e)
{
    Console.WriteLine(e.Message);
    return;
}
try
{
    var text2 = "i was nervous before the exam, and i had a fever.";
    Console.WriteLine($"Input text: {text2}");
    var tokens2 = tokenizer2.Encode(text2);
    Console.WriteLine($"Encoded: {string.Join(", ", tokens2)}");
    var decoded2 = tokenizer2.Decode(tokens2);
    Console.WriteLine($"Decoded: {decoded2}");
}
catch (TokenizerException e)
{
    Console.WriteLine(e.Message);
    return;
}

Console.WriteLine($"Version of Tokenizers.DotNet.runtime.win: {tokenizer2.GetVersion()}");
Console.ReadKey();

How to build

(Recommended) Cross-platform build

You can use Docker to compile this library for Windows x64/arm64 and Linux x64/arm64

Run update_version.ps1 before running Docker to update the package version.

Windows:

PS > docker build -f Dockerfile -t ghcr.io/sappho192/tokenizers.dotnet:latest .
PS > docker run -v .\nuget:/out --rm ghcr.io/sappho192/tokenizers.dotnet:latest

Linux/MacOS:

$ docker build -f Dockerfile -t ghcr.io/sappho192/tokenizers.dotnet:latest .
$ docker run -v ./nuget:/out --rm ghcr.io/sappho192/tokenizers.dotnet:latest

Built packages will be in the nuget folder.

Building with local machine system

(Note that this has been confirmed only in Windows machine)

Prepare following stuff:
1. Rust build system (cargo)
2. .NET build system (dotnet 6.0 or above)
3. PowerShell (Recommend 7.4.2 or above)
Bump the version number in NATIVE_LIB_VERSION.txt
Run build_all_clean.ps1
1. To build Tokenizers.DotNet.runtime.<OS> only, run build_rust.ps1
2. To build Tokenizers.DotNet only, run build_dotnet.ps1

Each build artifacts will be in nuget directory.

Release guide

Run update_version.ps1 to update the version number
Commit with a new version number and add a tag. New tag should be end with -RC0, -RC1, ... until final check. BE SURE to check following files are committed:

NATIVE_LIB_VERSION.txt
Cargo.toml
Tokenizers.DotNet.nuspec
Tokenizers.DotNet.runtime.win-x64.nuspec
Tokenizers.DotNet.runtime.win-arm64.nuspec
Tokenizers.DotNet.runtime.linux-x64.nuspec
Tokenizers.DotNet.runtime.linux-arm64.nuspec

Update README.md and name a real version tag (not ending with -RC0, ...)

Tokenizers.DotNet
Nuget Package list
Requirements
Supported functionalities
How to use
- (1) Install the packages
- (2) Write the code
How to build
- (Recommended) Cross-platform build
- Building with local machine system
Release guide

Tokenizers.DotNet

.NET wrapper of HuggingFace Tokenizers library

Nuget Package list

Package	main	Description
Tokenizers.DotNet		Core library
Tokenizers.DotNet.runtime.win-x64		Native bindings for windows x64
Tokenizers.DotNet.runtime.win-arm64		Native bindings for windows arm64
Tokenizers.DotNet.runtime.linux-x64		Native bindings for linux x64
Tokenizers.DotNet.runtime.linux-arm64		Native bindings for linux arm64
Tokenizers.DotNet.runtime.osx-x64		Native bindings for macOS x64
Tokenizers.DotNet.runtime.osx-arm64		Native bindings for macOS arm64

Requirements

.NET 6 / .NET Standard 2.0 or above
(Build) Latest Rust

Supported functionalities

Download tokenizer files from Hugginface Hub
Load tokenizer file(.json) from local
Encode string to tokens
Decode tokens to string

How to use

(1) Install the packages

From the NuGet, install Tokenizers.DotNet package
And then, install Tokenizers.DotNet.runtime.<OS>-<ARCH> package too (e.a win-x64 or linux-arm64, check Nuget package list above).

(2) Write the code

Check following example code:

using Tokenizers.DotNet;

// Download skt/kogpt2-base-v2/tokenizer.json from the hub
var hubName = "skt/kogpt2-base-v2";
var filePath = "tokenizer.json";
var fileFullPath = await HuggingFace.GetFileFromHub(hubName, filePath, "deps");
Console.WriteLine($"Downloaded {fileFullPath}");

// Create a tokenizer instance
Tokenizer tokenizer;
try
{
    tokenizer = new Tokenizer(vocabPath: fileFullPath);
}
catch (TokenizerException e)
{
	Console.WriteLine(e.Message);
    return;
}
try
{
    var text = "음, 이제 식사도 해볼까요";
    Console.WriteLine($"Input text: {text}");
    var tokens = tokenizer.Encode(text);
    Console.WriteLine($"Encoded: {string.Join(", ", tokens)}");
    var decoded = tokenizer.Decode(tokens);
    Console.WriteLine($"Decoded: {decoded}");
}
catch (TokenizerException e)
{
    Console.WriteLine(e.Message);
    return;
}
Console.WriteLine($"Version of Tokenizers.DotNet.runtime.win: {tokenizer.GetVersion()}");
Console.WriteLine("--------------------------------------------------");

//// Download openai-community/gpt2 from the hub
hubName = "openai-community/gpt2";
filePath = "tokenizer.json";
fileFullPath = await HuggingFace.GetFileFromHub(hubName, filePath, "deps");

// Create a tokenizer instance
Tokenizer tokenizer2;
try
{
    tokenizer2 = new Tokenizer(vocabPath: fileFullPath);
}
catch (TokenizerException e)
{
    Console.WriteLine(e.Message);
    return;
}
try
{
    var text2 = "i was nervous before the exam, and i had a fever.";
    Console.WriteLine($"Input text: {text2}");
    var tokens2 = tokenizer2.Encode(text2);
    Console.WriteLine($"Encoded: {string.Join(", ", tokens2)}");
    var decoded2 = tokenizer2.Decode(tokens2);
    Console.WriteLine($"Decoded: {decoded2}");
}
catch (TokenizerException e)
{
    Console.WriteLine(e.Message);
    return;
}

Console.WriteLine($"Version of Tokenizers.DotNet.runtime.win: {tokenizer2.GetVersion()}");
Console.ReadKey();

How to build

(Recommended) Cross-platform build

You can use Docker to compile this library for Windows x64/arm64 and Linux x64/arm64

Run update_version.ps1 before running Docker to update the package version.

Windows:

PS > docker build -f Dockerfile -t ghcr.io/sappho192/tokenizers.dotnet:latest .
PS > docker run -v .\nuget:/out --rm ghcr.io/sappho192/tokenizers.dotnet:latest

Linux/MacOS:

$ docker build -f Dockerfile -t ghcr.io/sappho192/tokenizers.dotnet:latest .
$ docker run -v ./nuget:/out --rm ghcr.io/sappho192/tokenizers.dotnet:latest

Built packages will be in the nuget folder.

Building with local machine system

(Note that this has been confirmed only in Windows machine)

Prepare following stuff:
1. Rust build system (cargo)
2. .NET build system (dotnet 6.0 or above)
3. PowerShell (Recommend 7.4.2 or above)
Bump the version number in NATIVE_LIB_VERSION.txt
Run build_all_clean.ps1
1. To build Tokenizers.DotNet.runtime.<OS> only, run build_rust.ps1
2. To build Tokenizers.DotNet only, run build_dotnet.ps1

Each build artifacts will be in nuget directory.

Release guide

Run update_version.ps1 to update the version number
Commit with a new version number and add a tag. New tag should be end with -RC0, -RC1, ... until final check. BE SURE to check following files are committed:

NATIVE_LIB_VERSION.txt
Cargo.toml
Tokenizers.DotNet.nuspec
Tokenizers.DotNet.runtime.win-x64.nuspec
Tokenizers.DotNet.runtime.win-arm64.nuspec
Tokenizers.DotNet.runtime.linux-x64.nuspec
Tokenizers.DotNet.runtime.linux-arm64.nuspec

Update README.md and name a real version tag (not ending with -RC0, ...)

sappho192/Tokenizers.DotNetv1.4.0

Get Started

Readme

Tokenizers.DotNet

Nuget Package list

Requirements

Supported functionalities

How to use

(1) Install the packages

(2) Write the code

How to build

(Recommended) Cross-platform build

Building with local machine system

Release guide

sappho192/Tokenizers.DotNetv1.4.0

Get Started

Readme

Tokenizers.DotNet

Nuget Package list

Requirements

Supported functionalities

How to use

(1) Install the packages

(2) Write the code

How to build

(Recommended) Cross-platform build

Building with local machine system

Release guide