Loki Bulk Data Processor is a high-performance .NET library for fast, scalable bulk data ingestion using SQL Server's Bulk Copy engine. Now safe for repeated execution in ASP.NET Core Web APIs (v7.0+). Supports saving from model collections, DataTables, IDataReaders, and JSON streams. Ideal for ETL workflows, demo environments, and real-time processing. Includes mapping, structured logging (v6+), and flexible dependency injection. For tutorials and demos, visit the Loki Software YouTube channel: https://www.youtube.com/channel/UCZSEs5w0PEySBdb9uFE19FQ/playlists
Get Started
$ dotnet add package LokiBulkDataProcessorReadme
Loki Bulk Data Processor
The Loki Bulk Data Processor is a fast, efficient .NET library for bulk data inserts into SQL Server using SqlBulkCopy. Designed for ASP.NET Core, it supports:
IEnumerable<T>modelsDataTableIDataReader(v6.0.0+)JSON streams(v6.0.0+)Structured logging and mapping support(v6.0.0+)
Perfect for high-throughput ETL pipelines and real-time ingestion scenarios.
This project has been upgraded to target .NET 10.
Install
dotnet add package LokiBulkDataProcessor
Performance
Benchmark: 100,000 records inserted into an empty SQL Server table with 4 columns (1 primary key, 1 foreign key), on a 2.7GHz i7, 16 GB RAM, SSD.
| Method | Time Taken | Memory Usage |
|---|---|---|
| Loki Bulk Processor | 2.99 seconds | 266 MB |
| Entity Framework Core | 68.74 seconds | 1 GB+ |
Inserted 100,000 records into an empty table with 1 PK and 1 FK column.
⚠️ Entity Framework was used with
AddAsyncandSaveChangesAsync. Loki used a single bulk insert call.
Upgrade Guide
| From Version | What Changed | Action Required |
|---|---|---|
| ≤ 6.0.0 | Web API execution failed after the first request | Upgrade to v7.0.0+ ✅ |
| 6.0.0 | DataTableMapping renamed to DataMapping | Update references ⚠️ |
| 7.0.1 | Project upgraded to .NET 10 | No action required unless you target an earlier runtime ✅ |
Documentation
Everything you need to get started:
- 📝 Full Documentation (Wiki)
- 📺 YouTube Tutorial – Part 1 — Introduction & Setup
- 📺 YouTube Tutorial – Part 2 — Advanced Usage & Mapping
- 🧪 Example Projects on GitHub
Support
If this library saved you time or made your life easier, consider showing your support:
- ☕ Buy me a coffee
- ⭐ Star the GitHub repo to help others discover it
- 🗣️ Share your experience or contribute feedback via issues
Every bit helps keep the project going.