WALnutDB is a simple, safe embedded database for .NET 8+. It’s built for devices with little RAM and slow flash (SD cards, eMMC). It uses Write-Ahead Logging (WAL) for power-loss durability, compacts data into sorted segment files (SST), and provides secondary indexes and time-series scans. Fully managed (no native deps), cross-platform, and async by default.
Highlights
Power-loss safety: WAL with per-frame CRC, truncation-tolerant replay.
Async I/O everywhere: designed for slow media and small RAM.
Document tables: pluggable serialization (e.g. System.Text.Json); GUIDs/strings/byte[] keys.
Secondary indexes: range scans over string/bytes/ints/floats/decimal (with scale).
Time-series mode: UTC-ordered keys, efficient range scans by time.
Checkpoint & SST: flush memtables to sorted segments, fast reads, WAL truncation.
Cross-platform: Linux/Windows/macOS; file-per-table for fault isolation.
No native dependencies: pure C# for easy deployment on IoT.
Roadmap: online compaction/defrag stats & swap, unique index enforcement, optional encryption-at-rest, richer query push-downs.
A simple, safe embedded database for .NET 8+. Built for devices with little RAM and slow flash (SD/eMMC). It uses Write-Ahead Logging (WAL) for power-loss durability, compacts data into sorted segment files (SST), and provides secondary indexes and time-series scans. Fully managed (no native deps), cross-platform, and async by default.
Highlights
Power-loss safety: WAL with per-frame CRC, truncation-tolerant replay.
Async I/O everywhere: designed for slow media and small RAM.
using System.Text.Json;
using WalnutDb;
using WalnutDb.Core;
using WalnutDb.Wal;
// Sample POCO
public sealed class User
{
[DatabaseObjectId] public string Id { get; set; } = "";
[DbIndex("Age")] public int Age { get; set; }
public string Name { get; set; } = "";
}
// Create database directory
var dir = Path.Combine(Path.GetTempPath(), "walnut-demo");
Directory.CreateDirectory(dir);
// WAL writer (group-commit, fsync per batch)
await using var wal = new WalWriter(Path.Combine(dir, "wal.log"));
// DB (managed, no native deps)
await using var db = new WalnutDatabase(
directory: dir,
options: new DatabaseOptions(),
manifest: new FileSystemManifestStore(dir),
wal: wal
);
// Open a table (default serializer is used unless you override it)
var users = await db.OpenTableAsync(new TableOptions<User>
{
GetId = u => u.Id, // string / Guid / byte[] supported
// Optional custom serializer:
//Serialize = u => JsonSerializer.SerializeToUtf8Bytes(u),
//Deserialize = b => JsonSerializer.Deserialize<User>(b.Span)!,
StoreGuidStringsAsBinary = true // optional optimization
});
// Upsert + Get
await users.UpsertAsync(new User { Id = "u1", Name = "Ada", Age = 37 });
var ada = await users.GetAsync("u1");
Console.WriteLine($"{ada?.Id}: {ada?.Name} ({ada?.Age})");
// Enumerate all
await foreach (var u in users.GetAllAsync())
Console.WriteLine($"{u.Id}: {u.Name} ({u.Age})");
// Durability checkpoint (flush mem→SST, truncate WAL)
await db.CheckpointAsync();
Secondary Indexes
Declare indexes:
public sealed class Product
{
[DatabaseObjectId] public string Id { get; set; } = "";
[DbIndex("Category")]
public string Category { get; set; } = "";
[DbIndex("Price", decimalScale: 2)] // decimals require a fixed scale
public decimal Price { get; set; }
}
Open table and scan ranges (decimal + string prefix):
using System.Text.Json;
using WalnutDb.Indexing;
var products = await db.OpenTableAsync(new TableOptions<Product> {
GetId = p => p.Id,
// Optional custom serializer:
//Serialize = p => JsonSerializer.SerializeToUtf8Bytes(p),
//Deserialize = b => JsonSerializer.Deserialize<Product>(b.Span)!
});
// Insert a few
await products.UpsertAsync(new Product { Id = "A", Category = "sensors", Price = 12.50m });
await products.UpsertAsync(new Product { Id = "B", Category = "sensors", Price = 18.75m });
await products.UpsertAsync(new Product { Id = "C", Category = "actuators", Price = 33.40m });
// Range by decimal index (inclusive/exclusive semantics)
var from = IndexKeyCodec.Encode(10m, decimalScale: 2);
var to = IndexKeyCodec.Encode(20m, decimalScale: 2);
await foreach (var p in products.ScanByIndexAsync("Price", from, to))
Console.WriteLine($"{p.Id} {p.Price:0.00}");
// Range by string prefix (e.g., category "sensors")
var start = IndexKeyCodec.Encode("sensors");
var end = IndexKeyCodec.PrefixUpperBound(start);
await foreach (var p in products.ScanByIndexAsync("Category", start, end))
Console.WriteLine($"{p.Id} {p.Category}");
Index hints
You can steer scans using IndexHint factories. (Descending Asc=false is planned; current implementation returns ascending.)
using WalnutDb.Indexing;
// Price in [10.00, 20.00), ascending, skip first 5, take 10
var hint = IndexHint.FromValues("Price", 10.00m, 20.00m, decimalScale: 2, asc: true, skip: 5, take: 10);
await foreach (var p in products.QueryAsync(x => true, hint))
Console.WriteLine($"{p.Id} {p.Price:0.00}");
// Prefix by category:
var hint2 = IndexHint.FromPrefix("Category", "sensors");
await foreach (var p in products.QueryAsync(_ => true, hint2))
Console.WriteLine($"{p.Id} {p.Category}");
Unique indexes
Mark a property with [DbIndex(Name, Unique = true)]. null values do not participate in uniqueness (SQL-like behavior).
public sealed class User
{
[DatabaseObjectId] public string Id { get; set; } = "";
[DbIndex("Email", Unique = true)]
public string? Email { get; set; }
}
// Insert A with email X, then checkpoint
await users.UpsertAsync(new User { Id = "A", Email = "x@example.com" });
await db.CheckpointAsync();
// Inserting B with the same email throws
await Assert.ThrowsAsync<InvalidOperationException>(() =>
users.UpsertAsync(new User { Id = "B", Email = "x@example.com" }));
// Deleting A releases the unique constraint immediately (in mem)
// and after checkpoint (for SST-backed entries)
await users.DeleteAsync("A");
await db.CheckpointAsync();
await users.UpsertAsync(new User { Id = "B", Email = "x@example.com" });
Time-Series
Provide a series id and a UTC timestamp. Keys are encoded so that lexicographic byte order matches chronological order.
public sealed class SensorSample
{
[DatabaseObjectId] public string Id { get; set; } = ""; // optional
public string DeviceId { get; set; } = "";
public DateTime Utc { get; set; } // must be UTC
public double Temperature { get; set; }
}
var ts = await db.OpenTimeSeriesAsync(new TimeSeriesOptions<SensorSample>
{
GetSeriesId = s => s.DeviceId,
GetUtcTimestamp = s => s.Utc,
// Optional custom serializer:
//Serialize = s => JsonSerializer.SerializeToUtf8Bytes(s),
//Deserialize = b => JsonSerializer.Deserialize<SensorSample>(b.Span)!,
});
// Append
await ts.AppendAsync(new SensorSample { DeviceId = "dev-1", Utc = DateTime.UtcNow, Temperature = 22.3 });
// Query by time range
var fromUtc = DateTime.UtcNow.AddMinutes(-10);
var toUtc = DateTime.UtcNow;
await foreach (var s in ts.QueryAsync("dev-1", fromUtc, toUtc))
Console.WriteLine($"{s.Utc:o} {s.Temperature:0.0}");
Transactions
Both automatic (implicit) and manual (explicit) transactions are supported.
// Auto: each call is an individual transaction
await users.UpsertAsync(new User { Id = "u2", Name = "Linus", Age = 49 });
// Manual: batch multiple ops → single WAL fsync
await using (var tx = await db.BeginTransactionAsync())
{
await users.UpsertAsync(new User { Id = "u3", Name = "Grace", Age = 44 }, tx);
await users.DeleteAsync("u1", tx);
await tx.CommitAsync(Durability.Group);
}
Encryption at rest
WALnutDB can encrypt values at rest (WAL + SST) with AES-GCM(256). Keys and index keys remain plaintext for sortability.
Encrypted: table values written to WAL and SST. Not encrypted: primary keys, index keys (they include the index value), filenames/metadata. Threat model: protects against offline reads of WAL/SST files. In-memory data is plaintext.
var dir = Path.Combine(Path.GetTempPath(), "walnut-enc");
await using var wal = new WalWriter(Path.Combine(dir, "wal.log"));
var key = Convert.FromHexString("00112233445566778899AABBCCDDEEFF00112233445566778899AABBCCDDEEFF");
await using var db = new WalnutDatabase(
dir,
new DatabaseOptions { Encryption = new AesGcmEncryption(key) },
new FileSystemManifestStore(dir),
wal);
var tbl = await db.OpenTableAsync(new TableOptions<MyDoc> { GetId = d => d.Id });
await tbl.UpsertAsync(new MyDoc { Id = "x", Secret = "hello" });
await db.CheckpointAsync(); // values written to SST are ciphertext
Crash-recovery with encryption: recovery verifies crc32 and replays committed frames, decrypting values on the fly before applying them to memtables.
Durability & Checkpoints
await db.FlushAsync() — fsync WAL (fast).
await db.CheckpointAsync() — flush memtables → SST and truncate the WAL.
Recovery reads wal.log, verifies the CRC of each frame, replays committed transactions, and safely stops at a torn tail.
For safe shutdown on embedded devices:
await db.CheckpointAsync(); // shortens recovery time and shrinks the WAL
Preflight Checks
Verify permissions, free space, and ability to acquire exclusive locks:
