Getting Started

Installation

dotnet add package Parquet.Data.Ado

Basic Usage

// Connect to a Parquet file
using var connection = new ParquetConnection("path/to/file.parquet");
connection.Open();

// Create a command
using var command = connection.CreateCommand();
command.CommandText = "SELECT * FROM data";

// Execute and read the data
using var reader = command.ExecuteReader();
while (reader.Read())
{
    // Access data by column index or name
    var value = reader["column_name"];
    // Process the data...
}

Using SQL Queries

using var connection = new ParquetConnection("path/to/file.parquet");
connection.Open();

// Create an SQL command
using var sqlCommand = connection.CreateSqlCommand("SELECT column1, column2 FROM data WHERE column3 > 100");

// Execute the query and get a DataTable
var dataTable = await ParquetExtensions.ToDataTableAsync(sqlCommand.ExecuteReader());

Working with Virtual Columns

// Create a virtual column definition
var virtualColumn = new VirtualColumn("ComputedColumn", typeof(decimal), 0.0m);

// Open a reader with the virtual column
using var reader = ParquetDataReaderFactory.CreateWithVirtualColumns(
    "path/to/file.parquet",
    new[] { virtualColumn });

// Now the reader includes the virtual column
while (reader.Read())
{
    decimal computedValue = reader.GetDecimal(reader.GetOrdinal("ComputedColumn"));
    // Use the virtual column value...
}

Batch Processing

// Create a batch reader for large files
using var batchReader = new ParquetBatchReader("path/to/large_file.parquet");

// Process batches asynchronously
await foreach (var batch in batchReader.ReadAllAsync())
{
    Console.WriteLine($"Processing batch {batch.RowGroupIndex} with {batch.RowCount} rows");
    
    // Process the batch...
    foreach (var column in batch.Columns)
    {
        // Access column data...
    }
}

Converting DataTable to Parquet

// Create or obtain a DataTable
var dataTable = new DataTable("MyData");
// ... populate the table ...

// Export to Parquet
await dataTable.ExportToParquetAsync("output.parquet");

Key Components

• ParquetConnection: Manages connection to Parquet files
• ParquetCommand: Executes commands against Parquet files
• ParquetDataReader: Reads data from Parquet files
• ParquetSqlCommand: Executes SQL queries against Parquet files
• ParquetDataReaderFactory: Creates data readers with various options
• ParquetBatchReader: Processes large Parquet files in batches
• ParquetExtensions: Utility methods for Parquet operations

Advanced SQL Capabilities

The library includes a SQL parser that supports:

• SELECT statements with column selection
• WHERE clauses with complex conditions
• Basic comparison operators (=, <>, <, >, <=, >=)
• Logical operators (AND, OR, NOT)
• Pattern matching with LIKE
• Value checking with IN, BETWEEN, and IS NULL

Requirements

• .NET 9.0 or later
• Dependencies:
  • Parquet.Net (for Parquet file handling)
  • Microsoft.SqlServer.TransactSql.ScriptDom (for SQL parsing)

Testing and Extensibility

The GitHub repository includes extensive tests that demonstrate the library's functionality. These tests can be used as examples and modified to suit your specific requirements.

Part of SQLFlow

This library is a component of SQLFlow, a comprehensive Data Automation framework that enables seamless data integration, transformation, and analysis across various data sources.

License

This project is licensed under the MIT License - see the LICENSE file for details.