This is the DataFrame connector for ETLBox.
It extends ETLBox with support for processing and transforming data using DataFrames, enabling efficient handling of structured data.
ETLBox is a complete ETL (Extract, Transform, Load) library and data integration toolbox for .NET.
# Build scalable, code-first ETL pipelines for SQL, NoSQL, APIs, and flat files.
# Automate data movement, transformation, and synchronization with minimal memory usage.
# Ideal for data warehousing, migrations, and big data processing.
Simplify your data integration workflow:
ETLBox enables efficient, asynchronous data processing by reading from databases, APIs, and file formats like CSV, Excel, and JSON.
Transform data dynamically with row-based, batch, or lookup transformations, and read or write from/to multiple destinations in parallel.
Key Features:
* Stream large datasets efficiently without loading everything into memory
* Maximize performance with parallel, task-based data flow execution
* Connect to various data sources with built-in connectors or extend with custom components
For tutorials, examples, and documentation, visit:
https://www.etlbox.net/
ETLBox is a complete ETL (Extract, Transform, Load) library and data integration toolbox for .NET. It helps you create and manage data integration pipelines to extract data from different sources, transform it into the format you need, and load it into databases, APIs, or files. Whether you're migrating data, synchronizing systems, or automating workflows, ETLBox provides the components to handle these tasks efficiently.
ETLBox is a flexible alternative to traditional ETL tools like SQL Server Integration Services (SSIS) or Azure Data Factory. Unlike those tools, ETLBox does not rely on a graphical user interface, making it ideal for developers who prefer programmatic control. It supports parallel execution and streaming, allowing efficient processing of large data with minimal memory usage.
Feature Overview
Scalable Pipelines - Handle large data volumes with bulk operations and parallel processing.
Flexible Connectors - Connect to SQL, NoSQL, APIs, and flat files like CSV, Excel, and JSON.
Streaming Support - Processes data efficiently with minimal memory usage.
Transformations - Harmonize, filter, aggregate, and clean data with built-in tools.
Cloud-Ready - Deploy on Azure, AWS, Docker, and Kubernetes.
Custom Extensions - Build and integrate your own transformations and connectors.
Why Choose ETLBox?
ETLBox lets you build data pipelines that process structured and unstructured data from various sources, including databases, APIs, and flat files. It automates tasks like data transformation, validation, and cleaning, and supports bulk inserts, updates, and deletes. With its ability to handle streaming data and parallel workflows, ETLBox delivers performance and flexibility for modern data integration tasks. Developers can rely on built-in connectors or extend the library to fit custom requirements.
Using ETLBox
Example 1: Bulk Insert into SQL Database
string connString =
@"Source=.;Trusted_Connection=true;Initial Catalog=ETLBox;"
var destCon = new SqlConnectionManager(connString);
//Create the dataflow components
var source = new CsvSource(sourceCon, "SourceData.csv");
var dest = new DbDestination(destCon, "DestinationTable");
//Default mapping: Csv header and column name match (case-sensitive)
//Override mapping like this:
dest.ColumnMapping = new[] {
new DbColumnMap() {
DbColumnName = "Id", PropertyName = "OrderNumber"
}
};
//Link the source and destination, run the dataflow
source.LinkTo(dest);
await Network.ExecuteAsnyc(source);
Example 2: Migrate Between Databases
string mySqlConnString =
@"Server=localhost;Database=ETLBox;Uid=user;Pwd=password;";
string postgresConnString =
@"Server=.;Initial Catalog=ETLBox;Trusted_Connection=true;";
var sourceCon = new MySqlConnectionManager(mySqlConnString);
var destCon = new PostgresConnectionManager(postgresConnString);
//Create the dataflow components
var source = new DbSource(sourceCon, "SourceTableName");
var dest = new DbDestination(destCon, "DestTableName");
//Link the source and destination, run the dataflow
source.LinkTo(dest);
await Network.ExecuteAsync(source);
Example 3: Merge Data Between Tables
public class MyMergeRow : MergeableRow
{
[IdColumn]
public long Key { get; set; }
[CompareColumn]
[UpdateColumn]
public string Value { get; set; }
[DeleteColumn(true)]
public bool DeleteThisRow { get; set; }
}
//Create the dataflow components
DbSource<MyMergeRow> source = new (connection, "SourceTable");
DbMerge<MyMergeRow> mergeDest = new (connection, "DestinationTable");
merge.MergeMode = MergeMode.Full;
merge.CacheMode = CacheMode.Partial;
//Link the source and destination, run the dataflow
source.LinkTo(merge);
Network.Execute(source);
Licensing & Support
For our full list of code examples, tutorials, licensing information, and documentation visit: www.etlbox.net
