TinyTokenizer

A high-performance syntax tree library for .NET 8+ with fluent queries, pattern matching, and undo/redo editing — built on a zero-allocation tokenizer with SIMD optimization.

Features

• TinyAst — Red-green syntax tree with fluent queries, editing, and undo/redo
• Query API — CSS-like selectors with combinators, lookahead, and repetition
• SyntaxEditor — Batch mutations with atomic commit and full undo/redo support
• Syntax Nodes — Pattern-based AST matching (function calls, property access, etc.)
• Schema System — Unified configuration for tokenization + syntax node definitions
• TreeWalker — DOM-style filtered tree traversal
• High Performance — Zero-allocation parsing with SIMD-optimized SearchValues<char>
• Error Recovery — Gracefully handles malformed input and continues parsing

Installation

dotnet add package TinyTokenizer

Quick Start

TinyAst

using TinyTokenizer.Ast; // Parse source into a syntax tree var tree = SyntaxTree.Parse("function foo() { return 1; }"); // Query nodes with CSS-like selectors var idents = tree.Select(Query.AnyIdent); // [Ident("function"), Ident("foo"), Ident("return")] // Fluent mutations with undo support tree.CreateEditor() .Replace(Query.Ident("foo"), "bar") .Insert(Query.BraceBlock.First().InnerStart(), "console.log('enter');") .Commit(); // Undo/redo tree.Undo(); tree.Redo();

Pattern Matching with Schema

var tree = SyntaxTree.Parse("obj.method(x)", Schema.Default); var methodCalls = tree.Match<MethodCallSyntax>().ToList(); // [MethodCallSyntax { Object="obj", Method="method" }]

Low-Level Tokenization

For scenarios where you don't need a syntax tree:

using TinyTokenizer; var tokens = "func(a, b)".TokenizeToTokens(); // tokens contains: // - IdentToken("func") // - BlockToken("(a, b)") with children: // - IdentToken("a"), SymbolToken(","), WhitespaceToken(" "), IdentToken("b")

📚 Documentation

Full documentation on the Wiki →

TinyAst

Topic	Description
TinyAst Guide	Syntax tree API
Schema	Unified configuration
Query API	CSS-like node selectors
SyntaxEditor	Batch mutations, undo/redo
Syntax Nodes	Pattern-based matching
TreeWalker	DOM-style traversal
Trivia	Whitespace preservation

Low-Level Tokenization

Topic	Description
Token Types	SimpleToken vs Token, all types
Configuration	Operators, comments, symbols
Async Streaming	Stream/PipeReader APIs
Error Handling	ErrorToken and recovery

Reference

Topic	Description
Getting Started	Installation and basic usage
Architecture	Two-level design, red-green trees
API Reference	Types, enums, methods

Requirements

• .NET 8.0 or later
• CommunityToolkit.HighPerformance (automatically included)

License

MIT