This is the core extension package. Always use it in conjunction with the corresponding database package (e.g., ETLBox.Postgres for PostgreSQL or ETLBox.SqlServer for SQL Server).
ETLBox.EntityFramework extends Entity Framework (Core) with additional bulk operations, including insert, update, upsert, delete, and merge. While SaveChanges() is already optimized for performance, these bulk extensions may offer improvements in certain scenarios.
These extensions are particularly useful when integrating ETL processes into your DbContext, especially when working with large datasets that cannot be fully loaded into memory.
For tutorials, examples, and documentation, visit:
https://ef.etlbox.net
ETLBox is a complete ETL (Extract, Transform, Load) library and data integration toolbox for .NET. It helps you create and manage data integration pipelines to extract data from different sources, transform it into the format you need, and load it into databases, APIs, or files. Whether you're migrating data, synchronizing systems, or automating workflows, ETLBox provides the components to handle these tasks efficiently.
ETLBox is a flexible alternative to traditional ETL tools like SQL Server Integration Services (SSIS) or Azure Data Factory. Unlike those tools, ETLBox does not rely on a graphical user interface, making it ideal for developers who prefer programmatic control. It supports parallel execution and streaming, allowing efficient processing of large data with minimal memory usage.
Feature Overview
Scalable Pipelines - Handle large data volumes with bulk operations and parallel processing.
Flexible Connectors - Connect to SQL, NoSQL, APIs, and flat files like CSV, Excel, and JSON.
Streaming Support - Processes data efficiently with minimal memory usage.
Transformations - Harmonize, filter, aggregate, and clean data with built-in tools.
Cloud-Ready - Deploy on Azure, AWS, Docker, and Kubernetes.
Custom Extensions - Build and integrate your own transformations and connectors.
Why Choose ETLBox?
ETLBox lets you build data pipelines that process structured and unstructured data from various sources, including databases, APIs, and flat files. It automates tasks like data transformation, validation, and cleaning, and supports bulk inserts, updates, and deletes. With its ability to handle streaming data and parallel workflows, ETLBox delivers performance and flexibility for modern data integration tasks. Developers can rely on built-in connectors or extend the library to fit custom requirements.
Using ETLBox
Example 1: Bulk Insert into SQL Database
string connString =
@"Source=.;Trusted_Connection=true;Initial Catalog=ETLBox;"
var destCon = new SqlConnectionManager(connString);
//Create the dataflow components
var source = new CsvSource(sourceCon, "SourceData.csv");
var dest = new DbDestination(destCon, "DestinationTable");
//Default mapping: Csv header and column name match (case-sensitive)
//Override mapping like this:
dest.ColumnMapping = new[] {
new DbColumnMap() {
DbColumnName = "Id", PropertyName = "OrderNumber"
}
};
//Link the source and destination, run the dataflow
source.LinkTo(dest);
await Network.ExecuteAsnyc(source);
Example 2: Migrate Between Databases
string mySqlConnString =
@"Server=localhost;Database=ETLBox;Uid=user;Pwd=password;";
string postgresConnString =
@"Server=.;Initial Catalog=ETLBox;Trusted_Connection=true;";
var sourceCon = new MySqlConnectionManager(mySqlConnString);
var destCon = new PostgresConnectionManager(postgresConnString);
//Create the dataflow components
var source = new DbSource(sourceCon, "SourceTableName");
var dest = new DbDestination(destCon, "DestTableName");
//Link the source and destination, run the dataflow
source.LinkTo(dest);
await Network.ExecuteAsync(source);
Example 3: Merge Data Between Tables
public class MyMergeRow : MergeableRow
{
[IdColumn]
public long Key { get; set; }
[CompareColumn]
[UpdateColumn]
public string Value { get; set; }
[DeleteColumn(true)]
public bool DeleteThisRow { get; set; }
}
//Create the dataflow components
DbSource<MyMergeRow> source = new (connection, "SourceTable");
DbMerge<MyMergeRow> mergeDest = new (connection, "DestinationTable");
merge.MergeMode = MergeMode.Full;
merge.CacheMode = CacheMode.Partial;
//Link the source and destination, run the dataflow
source.LinkTo(merge);
Network.Execute(source);
Licensing & Support
For our full list of code examples, tutorials, licensing information, and documentation visit: www.etlbox.net
