High-performance memory-mapped file implementation of .NET's IList<T> interface for inter-process communication and large data handling
$ dotnet add package BruSoftware.ListMmfHigh-performance memory-mapped file implementation of .NET's IList<T> interface for inter-process communication and large data handling.
dotnet add package BruSoftware.ListMmf
using BruSoftware.ListMmf;
// Create or open a memory-mapped list with an appropriate type
// Use Int32 for most integers (±2.1B range), Int64 for larger/unknown ranges
var list = new ListMmf<int>("shared-list.mmf", DataType.Int32);
// Use it like any IList<T>
list.Add(42);
list.Add(100);
list.Add(255);
// Access elements
int value = list[0]; // 42
list[1] = 200; // Update value
// Share between processes - another process can open the same list
using var sharedList = new ListMmf<int>("shared-list.mmf", DataType.Int32);
Console.WriteLine(sharedList.Count); // 3
ListMmf uses fixed types and does NOT auto-upgrade like SmallestInt. Choose appropriate types upfront:
// ✅ GOOD: Use Int32 or Int64 for production data
var prices = new ListMmf<int>("prices.mmf", DataType.Int32); // ±2.1B range
var volumes = new ListMmf<long>("volumes.mmf", DataType.Int64); // ±9.2E+18 range
// ❌ AVOID: Small types risk overflow and data corruption
var prices = new ListMmf<short>("prices.mmf", DataType.Int16); // Only ±32K!
// If you must cast, use checked() to throw on overflow instead of corrupting data:
int realtimeValue = GetFromDataFeed();
try {
prices.Add(checked((short)realtimeValue)); // Throws OverflowException if too large
} catch (OverflowException) {
Logger.Error($"Value {realtimeValue} exceeds type range");
}
📘 See BEST-PRACTICES.md for detailed guidance on type selection and overflow handling.
// Inspect a window of data without allocating new arrays
ReadOnlySpan<int> recent = sharedList.AsSpan(start: 1, length: 2);
Console.WriteLine(recent[0]);
[!NOTE] Legacy callers can continue to use
GetRangebut the method now forwards toAsSpaninternally. Prefer theAsSpanoverloads for new code so the zero-copy semantics are obvious at call sites.
// Optimized for DateTime series with ordered data
var timeSeries = new ListMmfTimeSeriesDateTime("market-data");
// Add timestamps
timeSeries.Add(DateTime.UtcNow);
// Efficient search with automatic strategy selection (NEW in v1.0.8)
// Auto-detects if data is uniformly distributed and uses optimal algorithm
var index = timeSeries.LowerBound(targetDateTime); // 3-5x faster on uniform data
// Or choose explicit strategy for backtesting/analytics:
var index = timeSeries.LowerBound(targetDateTime, SearchStrategy.Interpolation); // O(log log n)
var index = timeSeries.LowerBound(targetDateTime, SearchStrategy.Binary); // O(log n)
var index = timeSeries.LowerBound(targetDateTime, SearchStrategy.Auto); // Smart auto-detect
Search Performance (for 2M-2B items):
See SEARCH-STRATEGIES.md for detailed usage guide.
// SmallestInt automatically uses the smallest storage size based on your data range
var optimizedList = new SmallestInt64ListMmf(DataType.Int24AsInt64, "optimized-data.bt");
// Stores using minimal bytes (Int24, Int40, etc.) and auto-upgrades when needed
optimizedList.Add(1000); // Stored as Int24 (3 bytes)
optimizedList.Add(10000000); // Auto-upgrades to Int32 (4 bytes)
When to use SmallestInt vs standard ListMmf:
See BEST-PRACTICES.md for detailed comparison.
Odd-byte structs such as UInt24AsInt64 and Int40AsInt64 expose zero-copy spans, but expanding them to full 64-bit integers previously required extra allocations or the SmallestInt wrappers. New extension methods keep conversions allocation-conscious:
using BruSoftware.ListMmf;
using var list = new ListMmf<UInt40AsInt64>("ticks.u40.mmf", DataType.UInt40AsInt64);
// Reuse a caller-owned buffer when iterating through large files
var chunkSize = 1_024;
var buffer = GC.AllocateUninitializedArray<long>(chunkSize);
long position = 0;
while (position < list.Count)
{
var toRead = (int)Math.Min(chunkSize, list.Count - position);
list.CopyAsInt64(position, buffer.AsSpan(0, toRead));
Process(buffer.AsSpan(0, toRead));
position += toRead;
}
// Pool-backed helper returns IMemoryOwner<long> trimmed to your requested length
using var owner = list.RentAsInt64(start: 0, length: chunkSize);
var span = owner.Memory.Span;
Consume(span);
These helpers work with ListMmf<T> writers and IReadOnlyList64Mmf<T> readers for all supported odd-byte types (24/40/48/56-bit, signed and unsigned). They expand values to long without per-element allocations and are ideal when you need repeated analysis passes. For one-off whole-file conversions, continue to use ListMmfWidthConverter.
longBruTrader and other downstream tools can now work purely with long values even when the on-disk representation uses odd-byte widths. The new factory returns an allocation-conscious adapter that exposes IListMmf<long> and IReadOnlyList64Mmf<long> while delegating storage to the original type.
using BruSoftware.ListMmf;
using System.IO.MemoryMappedFiles;
// Inspect values from a UInt24-backed file without rewriting it
using var bars = UtilsListMmf.OpenAsInt64("Closes.bt", MemoryMappedFileAccess.ReadWrite);
// Zero-copy reads reuse an internal pooled buffer for odd-byte widths
ReadOnlySpan<long> window = bars.AsSpan(start: bars.Count - 1_000, length: 1_000);
// Checked writes throw DataTypeOverflowException when the value no longer fits
try
{
bars.Add(checked((long)1_000_000));
}
catch (DataTypeOverflowException ex)
{
Console.WriteLine($"{ex.Message}\nUpgrade suggestion: {ex.SuggestedDataType}");
}
// Monitor capacity consumption (returns the larger of the positive/negative utilization ratios)
var status = ((IListMmfLongAdapter)bars).GetDataTypeUtilization();
Console.WriteLine($"{status.Utilization:P1} of {status.AllowedMax:N0} range in use");
// Optional: trigger a friendly warning when utilization crosses a threshold
((IListMmfLongAdapter)bars).ConfigureUtilizationWarning(0.90, info =>
{
Console.WriteLine($"WARNING: {info.Utilization:P0} of {info.AllowedMax:N0} capacity consumed");
});
UtilsListMmf.OpenAsInt64 automatically maps every supported DataType (including the odd-byte Int24/UInt24/Int40/UInt40/... variants) to its concrete ListMmf<T> and wraps it in the high-performance adapter. Writes remain O(1) with pooled buffers, and the adapter throws DataTypeOverflowException with upgrade guidance instead of silently truncating.
// Create read-only views for safe concurrent access
var readOnlyView = new ReadOnlyList64Mmf<double>("data", isReadOnly: true);
// Multiple readers can access simultaneously without locks
double sum = readOnlyView.Sum();
// Support for custom structs (must be blittable)
[StructLayout(LayoutKind.Sequential)]
public struct MarketTick
{
public long Timestamp;
public double Price;
public int Volume;
}
var tickData = new ListMmf<MarketTick>("market-ticks");
// Track performance metrics
var list = new ListMmf<long>("tracked-list");
list.ProgressReport += (sender, args) =>
{
Console.WriteLine($"Operation: {args.Operation}, Items: {args.ItemsProcessed}");
};
This library requires a 64-bit process to ensure atomic operations on 8-byte values without locking. This design choice enables:
The underlying storage uses Windows memory-mapped files, providing:
Data files are stored with metadata headers containing:
Example:
// Process A: Writer (exclusive)
using var writer = new ListMmf<long>("shared.mmf", DataType.Int64);
writer.Add(12345);
// Process B & C: Readers (concurrent, lock-free)
using var reader1 = new ListMmf<long>("shared.mmf", DataType.Int64);
using var reader2 = new ListMmf<long>("shared.mmf", DataType.Int64);
Console.WriteLine(reader1.Count + reader2.Count); // Safe
Copyright © Dale A. Brubaker 2022-2025
Licensed under the terms in LICENSE.txt
Contributions are welcome! Please feel free to submit issues and pull requests on GitHub.
For questions and support, please open an issue on the GitHub repository.