Modern, minimal, fast, zero allocation, reading and writing of separated values (csv, tsv etc.). Cross-platform, trimmable and AOT/NativeAOT compatible. Featuring an opinionated API design and pragmatic implementation targetted at machine learning use cases.

⭐ Please star this project if you like it. ⭐

🌃 Modern - utilizes features such as Span<T>, Generic Math (ISpanParsable<T>/ ISpanFormattable), ref struct, ArrayPool<T> and similar from .NET 7+ and C# 11+ for a modern and highly efficient implementation.

🔎 Minimal - a succinct yet expressive API with few options and no hidden changes to input or output. What you read/write is what you get. E.g. by default there is no "automatic" escaping/unescaping of quotes or trimming of spaces. To enable this see SepReaderOptions and Unescaping and Trimming. See SepWriterOptions for Escaping.

🚀 Fast - blazing fast with both architecture specific and cross-platform SIMD vectorized parsing incl. 64/128/256/512-bit paths e.g. AVX2, AVX-512 (.NET 8.0+), NEON. Uses csFastFloat for fast parsing of floating points. See detailed benchmarks for cross-platform results.

🌪️ Multi-threaded - unparalleled speed with highly efficient parallel CSV parsing that is up to 35x faster than CsvHelper, see ParallelEnumerate and benchmarks.

🌀 Async support - efficient ValueTask based async/await support. Requires C# 13.0+ and for .NET 9.0+ includes SepReader implementing IAsyncEnumerable<>. See Async Support for details.

🗑️ Zero allocation - intelligent and efficient memory management allowing for zero allocations after warmup incl. supporting use cases of reading or writing arrays of values (e.g. features) easily without repeated allocations.

✅ Thorough tests - great code coverage and focus on edge case testing incl. randomized fuzz testing.

🌐 Cross-platform - works on any platform, any architecture supported by NET. 100% managed and written in beautiful modern C#.

✂️ Trimmable and AOT/NativeAOT compatible - no problematic reflection or dynamic code generation. Hence, fully trimmable and Ahead-of-Time compatible. With a simple console tester program executable possible in just a few MBs. 💾

🗣️ Opinionated and pragmatic - conforms to the essentials of RFC-4180, but takes an opinionated and pragmatic approach towards this especially with regards to quoting and line ends. See section RFC-4180.

Example | Naming and Terminology | API | Limitations and Constraints | Comparison Benchmarks | Example Catalogue | RFC-4180 | FAQ | Public API Reference

Example

Naming and Terminology

Sep uses naming and terminology that is not based on RFC-4180, but is more tailored to usage in machine learning or similar. Additionally, Sep takes a pragmatic approach towards names by using short names and abbreviations where it makes sense and there should be no ambiguity given the context. That is, using Sep for Separator and Col for Column to keep code succinct.

Application Programming Interface (API)

Besides being the succinct name of the library, Sep is both the main entry point to using the library and the container for a validated separator. That is, Sep is basically defined as:

The separator char is validated upon construction and is guaranteed to be within a limited range and not being a char like " (quote) or similar. This can be seen in src/Sep/Sep.cs. The separator is constrained also for internal optimizations, so you cannot use any char as a separator.

⚠ Note that all types are within the namespace nietras.SeparatedValues and not Sep since it is problematic to have a type and a namespace with the same name.

To get started you can use Sep as the static entry point to building either a reader or writer. That is, for SepReader:

where .Reader() is a convenience method corresponding to:

where Sep? Auto => null; is a static property that returns null for a nullable Sep to signify that the separator should be inferred from the first row, which might be a header. If the first row does not contain any of the by default supported separators or there are no rows, the default separator will be used.

⚠ Note Sep uses ; as the default separator, since this is what was used in an internal proprietary library which Sep was built to replace. This is also to avoid issues with comma , being used as a decimal separator in some locales. Without having to resort to quoting.

If you want to specify the separator you can write:

or

Similarly, for SepWriter:

or

where you have to specify a valid separator, since it cannot be inferred. To fascillitate easy flow of the separator and CultureInfo both SepReader and SepWriter expose a Spec property of type SepSpec that simply defines those two. This means you can write:

where the writer then will use the separator inferred by the reader, for example.

API Pattern

In general, both reading and writing follow a similar pattern:

where each continuation flows fluently from the preceding type. For example, Reader() is an extension method to Sep or SepSpec that returns a SepReaderOptions. Similarly, Writer() is an extension method to Sep or SepSpec that returns a SepWriterOptions.

SepReaderOptions and SepWriterOptions are optionally configurable. That and the APIs for reader and writer is covered in the following sections.

For a complete example, see the example above or the ReadMeTest.cs.

⚠ Note that it is important to understand that Sep Row/Col/Cols are ref structs (please follow the ref struct link and understand how this limits the usage of those). This is due to these types being simple facades or indirections to the underlying reader or writer. That means you cannot use LINQ or create an array of all rows like reader.ToArray(). While for .NET9+ the reader is now IEnumerable<> since ref structs can now be used in interfaces that have where T: allows ref struct this still does not mean it is LINQ compatible. Hence, if you need store per row state or similar you need to parse or copy to different types instead. The same applies to Col/Cols which point to internal state that is also reused. This is to avoid repeated allocations for each row and get the best possible performance, while still defining a well structured and straightforward API that guides users to relevant functionality. See Why SepReader Was Not IEnumerable Until .NET 9 and Is Not LINQ Compatible for more.

⚠ For a full overview of public types and methods see Public API Reference.

SepReader API

SepReader API has the following structure (in pseudo-C# code):

That is, to use SepReader follow the points below:

1. Optionally define Sep or use default automatically inferred separator.
2. Specify reader with optional configuration of SepReaderOptions. For example, if a csv-file does not have a header this can be configured via: Sep.Reader(o => o with { HasHeader = false }) For all options see SepReaderOptions.
3. Specify source e.g. file, text (string), TextWriter, etc. via From extension methods.
4. Optionally access the header. For example, to get all columns starting with GT_ use: var colNames = header.NamesStarting("GT_"); var colIndices = header.IndicesOf(colNames);
5. Enumerate rows. One row at a time.
6. Access a column by name or index. Or access multiple columns with names and indices. Sep internally handles pooled allocation and reuse of arrays for multiple columns.
7. Use Span to access the column directly as a ReadOnlySpan<char>. Or use ToString to convert to a string. Or use Parse<T> where T : ISpanParsable<T> to parse the column chars to a specific type.

SepReaderOptions

The following options are available:

Unescaping

While great care has been taken to ensure Sep unescaping of quotes is both correct and fast, there is always the question of how does one respond to invalid input.

The below table tries to summarize the behavior of Sep vs CsvHelper and Sylvan. Note that all do the same for valid input. There are differences for how invalid input is handled. For Sep the design choice has been based on not wanting to throw exceptions and to use a principle that is both reasonably fast and simple.

· (middle dot) is whitespace to make this visible

¹ CsvHelper with BadDataFound = null

² Sep with Unescape = true in SepReaderOptions

Trimming

Sep supports trimming by the SepTrim flags enum, which has two options as documented there. Below the result of both trimming and unescaping is shown in comparison to CsvHelper. Note unescaping is enabled for all results shown. It is possible to trim without unescaping too, of course.

As can be seen Sep supports a simple principle of trimming before and after unescaping with trimming before unescaping being important for unescaping if there is a starting quote after spaces.

· (middle dot) is whitespace to make this visible

¹ CsvHelper with TrimOptions.Trim | TrimOptions.InsideQuotes

² Sep with SepTrim.All = SepTrim.Outer | SepTrim.AfterUnescape in SepReaderOptions

SepReader Debuggability

Debuggability is an important part of any library and while this is still a work in progress for Sep, SepReader does have a unique feature when looking at it and it's row or cols in a debug context. Given the below example code:

and you are hovering over reader when the break is triggered then this will show something like:

That is, it will show information of the source for the reader, in this case a string of length 55.

SepReader.Row Debuggability

If you are hovering over row then this will show something like:

This has the format shown below.

Note how this shows line number range [FromIncl..ToExcl], as in C# range expression, so that one can easily find the row in question in notepad or similar. This means Sep has to track line endings inside quotes and is an example of a feature that makes Sep a bit slower but which is a price considered worth paying.

GitHub doesn't show line numbers in code blocks so consider copying the example text to notepad or similar to see the effect.

Additionally, if you expand the row in the debugger (e.g. via the small triangle) you will see each column of the row similar to below.

SepReader.Col Debuggability

If you hover over col you should see:

Why SepReader Was Not IEnumerable Until .NET 9 and Is Not LINQ Compatible

As mentioned earlier Sep only allows enumeration and access to one row at a time and SepReader.Row is just a simple facade or indirection to the underlying reader. This is why it is defined as a ref struct. In fact, the following code:

can also be rewritten as:

where row is just a facade for exposing row specific functionality. That is, row is still basically the reader underneath. Hence, let's look at using LINQ with SepReader implementing IEnumerable<SepReader.Row> and the Row not being a ref struct. Then, you would be able to write something like below:

Given Row is just a facade for the reader, this would be equivalent to writing:

which hopefully makes it clear why this is not a good thing. The array would effectively be the reader repeated several times. If this would have to be supported one would have to allocate memory for each row always, which would basically be no different than a ReadLine approach as benchmarked in Comparison Benchmarks.

This is perhaps also the reason why no other efficient .NET CSV parser (known to author) implements an API pattern like Sep, but instead let the reader define all functionality directly and hence only let's you access the current row and cols on that. This API, however, is in this authors opinion not ideal and can be a bit confusing, which is why Sep is designed like it is. The downside is the above caveat.

The main culprit above is that for example ToArray() would store a ref struct in a heap allocated array, the actual enumeration is not a problem and hence implementing IEnumerable<SepReader.Row> is not the problem as such. The problem was that prior to .NET 9 it was not possible to implement this interface with T being a ref struct, but with C# 13 allows ref struct and .NET 9 having annotated such interfaces it is now possible and you can assign SepReader to IEnumerable, but most if not all of LINQ will still not work as shown below.

Calling Select should in principle be possible if this was annotated with allows ref struct, but it isn't currently.

If you want to use LINQ or similar you have to first parse or transform the rows into some other type and enumerate it. This is easy to do and instead of counting lines you should focus on how such enumeration can be easily expressed using C# iterators (aka yield return). With local functions this can be done inside a method like:

Now if instead refactoring this to something LINQ-compatible by defining a common Enumerate or similar method it could be:

In fact, Sep provides such a convenience extension method. And, discounting the Enumerate method, this does have less boilerplate, but not really more effective lines of code. The issue here is that this tends to favor factoring code in a way that can become very inefficient quickly. Consider if one wanted to only enumerate rows matching a predicate on Key which meant only 1% of rows were to be enumerated e.g.:

This means you are still parsing the double (which is magnitudes slower than getting just the key) for all rows. Imagine if this was an array of floating points or similar. Not only would you then be parsing a lot of values you would also be allocated 99x arrays that aren't used after filtering with Where.

Instead, you should focus on how to express the enumeration in a way that is both efficient and easy to read. For example, the above could be rewritten as:

To accomodate this Sep provides an overload for Enumerate that is similar to:

With this the above custom Enumerate can be replaced with:

Note how this is pretty much the same length as the previous custom Enumerate. Also worse due to how C# requires specifying types for out parameters which then requires all parameter types for the lambda to be specified. Hence, in this case the custom Enumerate does not take significantly longer to write and is a lot more efficient than using LINQ .Where (also avoids allocating a string for key for each row) and is easier to debug and perhaps even read. All examples above can be seen in ReadMeTest.cs.

There is a strong case for having an enumerate API though and that is for parallelized enumeration, which will be discussed next.

ParallelEnumerate and Enumerate

As discussed in the previous section Sep provides Enumerate convenience extension methods, that should be used carefully. Alongside these there are ParallelEnumerate extension methods that provide very efficient multi-threaded enumeration. See benchmarks for numbers and Public API Reference.

ParallelEnumerate is build on top of LINQ AsParallel().AsOrdered() and will return exactly the same as Enumerate but with enumeration parallelized. This will use more memory during execution and as many threads as possible via the .NET thread pool. When using ParallelEnumerate one should, therefore (as always), be certain the provided delegate does not refer to or change any mutable state.

ParallelEnumerate comes with a lot of overhead compared to single-threaded foreach or Enumerate and should be used carefully based on measuring any potential benefit. Sep goes a long way to make this very efficient by using pooled arrays and parsing multiple rows in batches, but if the source only has a few rows then any benefit is unlikely.

Due to ParallelEnumerate being based on batches of rows it is also important not to "abuse" it in-place of LINQ AsParallel. The idea is to use it for parsing rows, not for doing expensive per row operations like loading an image or similar. In that case, you are better off using AsParallel() after ParallelEnumerate or Enumerate similarly to:

As a rule of thumb if the time per row exceeds 1 millisecond consider moving the expensive work to after ParallelEnumerate/Enumerate,

SepWriter API

SepWriter API has the following structure (in pseudo-C# code):

That is, to use SepWriter follow the points below:

1. Optionally define Sep or use default automatically inferred separator.
2. Specify writer with optional configuration of SepWriterOptions. For all options see SepWriterOptions.
3. Specify destination e.g. file, text (string via StringWriter), TextWriter, etc. via To extension methods.
4. MISSING: SepWriter currently does not allow you to define the header up front. Instead, header is defined by the order in which column names are accessed/created when defining the row.
5. Define new rows with NewRow. ⚠ Be sure to dispose any new rows before starting the next! For convenience Sep provides an overload for NewRow that takes a SepReader.Row and copies the columns from that row to the new row: using var reader = Sep.Reader().FromText(text); using var writer = reader.Spec.Writer().ToText(); foreach (var readRow in reader) { using var writeRow = writer.NewRow(readRow); }
6. Create a column by selecting by name or index. Or multiple columns via indices and names. Sep internally handles pooled allocation and reuse of arrays for multiple columns.
7. Use Set to set the column value either as a ReadOnlySpan<char>, string or via an interpolated string. Or use Format<T> where T : IFormattable to format T to the column value.
8. Row is written when Dispose is called on the row.

   Note this is to allow a row to be defined flexibly with both column removal, moves and renames in the future. This is not yet supported.

SepWriterOptions

The following options are available:

Escaping

Escaping is not enabled by default in Sep, but when it is it gives the same results as other popular CSV librares as shown below. Although, CsvHelper appears to be escaping spaces as well, which is not necessary.

Separator/delimiter is set to semi-colon ; (default for Sep)

· (middle dot) is whitespace to make this visible

\r, \n are carriage return and line feed special characters to make these visible

¹ Sep with Escape = true in SepWriterOptions

Async Support

Sep supports efficient ValueTask based asynchronous reading and writing.

However, given both SepReader.Row and SepWriter.Row are ref structs, as they point to internal state and should only be used one at a time, async/await usage is only supported on C# 13.0+ as this has support for "ref and unsafe in iterators and async methods" as covered in What's new in C# 13. Please consult details in that for limitations and constraints due to this.

Similarly, SepReader only implements IAsyncEnumerable<SepReader.Row> (and IEnumerable<SepReader.Row>) for .NET 9.0+/C# 13.0+ since then the interfaces have been annotated with allows ref struct for T.

Async support is provided on the existing SepReader and SepWriter types similar to how TextReader and TextWriter support both sync and async usage. This means you as a developer are responsible for calling async methods and using await when necessary. See below for a simple example and consult tests on GitHub for more examples.

Note how for SepReader the FromTextAsync is suffixed with Async to indicate async creation, this is due to the reader having to read the first row of the source at creation to determine both separator and, if file has a header, column names of the header. The From*Async call then has to be awaited. After that rows can be enumerated asynchronously simply by putting await before foreach. If one forgets to do that the rows will be enumerated synchronously.

For SepWriter the usage is kind of reversed. To* methods have no Async variants, since creation is synchronous. That is, StreamWriter is created by a simple constructor call. Nothing is written until a header or row is defined and Dispose/DisposeAsync is called on the row.

For reader nothing needs to be asynchronously disposed, so using does not require await. However, for SepWriter dispose may have to write/flush data to underlying TextWriter and hence it should be using DisposeAsync, so you must use await using.

To support cancellation many methods have overloads that accept a CancellationToken like the From*Async methods for creating a SepReader or for example NewRow for SepWriter. Consult Public API Reference for full set of available methods.

Additionally, both SepReaderOptions and SepWriterOptions feature the bool AsyncContinueOnCapturedContext option that is forwarded to internal ConfigureAwait calls, see the ConfigureAwait FAQ for details on that.

Limitations and Constraints

Sep is designed to be minimal and fast. As such, it has some limitations and constraints:

• Comments # are not directly supported. You can skip a row by: foreach (var row in reader) { // Skip row if starts with # if (!row.Span.StartsWith("#")) { // ... } } This does not allow skipping lines before a header row starting with # though. In Example Catalogue a full example is given detailing how to skip lines before header.

Comparison Benchmarks

To investigate the performance of Sep it is compared to:

• CsvHelper - the most commonly used CSV library with a staggering downloads on NuGet. Fully featured and battle tested.
• Sylvan - is well-known and has previously been shown to be the fastest CSV libraries for parsing (Sep changes that 😉).
• ReadLine/WriteLine - basic naive implementations that read line by line and split on separator. While writing columns, separators and line endings directly. Does not handle quotes or similar correctly.

All benchmarks are run from/to memory either with:

• StringReader or StreamReader + MemoryStream
• StringWriter or StreamWriter + MemoryStream

This to avoid confounding factors from reading from or writing to disk.

When using StringReader/StringWriter each char counts as 2 bytes, when measuring throughput e.g. MB/s. When using StreamReader/StreamWriter content is UTF-8 encoded and each char typically counts as 1 byte, as content usually limited to 1 byte per char in UTF-8. Note that in .NET for TextReader and TextWriter data is converted to/from char, but for reading such conversion can often be just as fast as Memmove.

By default only StringReader/StringWriter results are shown, if a result is based on StreamReader/StreamWriter it will be called out. Usually, results for StreamReader/StreamWriter are in line with StringReader/StringWriter but with half the throughput due to 1 byte vs 2 bytes. For brevity they are not shown here.

For all benchmark results, Sep has been defined as the Baseline in BenchmarkDotNet. This means Ratio will be 1.00 for Sep. For the others Ratio will then show how many times faster Sep is than that. Or how many times more bytes are allocated in Alloc Ratio.

Disclaimer: Any comparison made is based on a number of preconditions and assumptions. Sep is a new library written from the ground up to use the latest and greatest features in .NET. CsvHelper has a long history and has to take into account backwards compatibility and still supporting older runtimes, so may not be able to easily utilize more recent features. Same goes for Sylvan. Additionally, Sep has a different feature set compared to the two. Performance is a feature, but not the only feature. Keep that in mind when evaluating results.

Runtime and Platforms

The following runtime is used for benchmarking:

• NET 9.0.X

NOTE: Garbage Collection DATAS mode is disabled since this severely impacts (e.g. 1.7x slower) performance for some benchmarks due to the bursty accumulated allocations. That is, GarbageCollectionAdaptationMode is set to 0.

The following platforms are used for benchmarking:

• AMD EPYC 7763 (Virtual) X64 Platform Information OS=Ubuntu 22.04.5 LTS (Jammy Jellyfish) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
• AMD Ryzen 7 PRO 7840U (Laptop on battery) X64 Platform Information OS=Windows 11 (10.0.22631.4460/23H2/2023Update/SunValley3) AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics, 1 CPU, 16 logical and 8 physical cores
• AMD 5950X (Desktop) X64 Platform Information (no longer available) OS=Windows 10 (10.0.19044.2846/21H2/November2021Update) AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
• AMD 9950X (Desktop) X64 Platform Information OS=Windows 10 (10.0.19044.3086/21H2/November2021Update) AMD Ryzen 9 9950X, 1 CPU, 32 logical and 16 physical cores
• Apple M1 (Virtual) ARM64 Platform Information OS=macOS Sonoma 14.7.1 (23H222) [Darwin 23.6.0] Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores

Reader Comparison Benchmarks

The following reader scenarios are benchmarked:

• NCsvPerf from The fastest CSV parser in .NET
• Floats as for example in machine learning.

Details for each can be found in the following. However, for each of these 3 different scopes are benchmarked to better assertain the low-level performance of each library and approach and what parts of the parsing consume the most time:

• Row - for this scope only the row is enumerated. That is, for Sep all that is done is: foreach (var row in reader) { } this should capture parsing both row and columns but without accessing these. Note that some libraries (like Sylvan) will defer work for columns to when these are accessed.
• Cols - for this scope all rows and all columns are enumerated. If possible columns are accessed as spans, if not as strings, which then might mean a string has to be allocated. That is, for Sep this is: foreach (var row in reader) { for (var i = 0; i < row.ColCount; i++) { var span = row[i].Span; } }
• XYZ - finally the full scope is performed which is specific to each of the scenarios.

Additionally, as Sep supports multi-threaded parsing via ParallelEnumerate benchmarks results with _MT in the method name are multi-threaded. These show Sep provides unparalleled performance compared to any other CSV parser.

The overhead of Sep async support is also benchmarked and can be seen with _Async in the method name. Note that this is the absolute best case for async given there is no real IO involved and hence no actual asynchronous work or continuations (thus no Task allocations) since benchmarks run from memory only. This is fine as the main purpose of the benchmark is to gauge the overhead of the async code path.

NCsvPerf PackageAssets Reader Comparison Benchmarks

NCsvPerf from The fastest CSV parser in .NET is a benchmark which in Joel Verhagen own words was defined with:

My goal was to find the fastest low-level CSV parser. Essentially, all I wanted was a library that gave me a string[] for each line where each field in the line was an element in the array.

What is great about this work is it tests a whole of 35 different libraries and approaches to this. Providing a great overview of those and their performance on this specific scenario. Given Sylvan is the fastest of those it is used as the one to beat here, while CsvHelper is used to compare to the most commonly used library.

The source used for this benchmark PackageAssetsBench.cs is a PackageAssets.csv with NuGet package information in 25 columns with rows like:

For Scope = Asset the columns are parsed into a PackageAsset class, which consists of 25 properties of which 22 are strings. Each asset is accumulated into a List<PackageAsset>. Each column is accessed as a string regardless.

This means this benchmark is dominated by turning columns into strings for the decently fast parsers. Hence, the fastest libraries in this test employ string pooling. That is, basically a custom dictionary from ReadOnlySpan<char> to string, which avoids allocating a new string for repeated values. And as can be seen in the csv-file there are a lot of repeated values. Both Sylvan and CsvHelper do this in the benchmark. So does Sep and as with Sep this is an optional configuration that has to be explicitly enable. For Sep this means the reader is created with something like:

What is unique for Sep is that it allows defining a pool per column e.g. via SepToString.PoolPerCol(...). This is based on the fact that often each column has its own set of values or strings that may be repeated without any overlap to other columns. This also allows one to define per column specific handling of ToString behavior. Whether to pool or not. Or even to use a statically defined pool.

Sep supports unescaping via an option, see SepReaderOptions and Unescaping. Therefore, Sep has two methods being tested. The default Sep without unescaping and Sep_Unescape where unescaping is enabled. Note that only if there are quotes will there be any unescaping, but to support unescape one has to track extra state during parsing which means there is a slight cost no matter what, most notably for the Cols scope. Sep is still the fastest of all (by far in many cases).

PackageAssets Benchmark Results

The results below show Sep is the fastest .NET CSV Parser (for this benchmark on these platforms and machines 😀). While for pure parsing allocating only a fraction of the memory due to extensive use of pooling and the ArrayPool<T>.

This is in many aspects due to Sep having extremely optimized string pooling and optimized hashing of ReadOnlySpan<char>, and thus not really due the the csv-parsing itself, since that is not a big part of the time consumed. At least not for a decently fast csv-parser.

With ParallelEnumerate (MT) Sep is >2x faster than Sylvan and up to 9x faster than CsvHelper.

At the lowest level of enumerating rows only, that is csv parsing only, Sep hits a staggering 21 GB/s on 9950X. Single-threaded.

AMD.EPYC.7763 - PackageAssets Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.7.PRO.7840U.w.Radeon.780M - PackageAssets Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.9.5950X - PackageAssets Benchmark Results (Sep 0.9.0.0, Sylvan 1.3.9.0, CsvHelper 33.0.1.24)

AMD.Ryzen.9.9950X - PackageAssets Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Apple.M1.(Virtual) - PackageAssets Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Cobalt.100 - PackageAssets Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

PackageAssets Benchmark Results (SERVER GC)

The package assets benchmark (Scope Asset) has a very high base load in the form of the accumulated instances of PackageAsset and since Sep is so fast the GC becomes a significant bottleneck for the benchmark, especially for multi-threaded parsing. Switching to SERVER GC can, therefore, provide significant speedup as can be seen below.

With ParallelEnumerate and server GC Sep is >4x faster than Sylvan and up to 18x faster than CsvHelper. Breaking 8 GB/s parsing speed on package assets on 9950X.

AMD.EPYC.7763 - PackageAssets Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.7.PRO.7840U.w.Radeon.780M - PackageAssets Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.9.5950X - PackageAssets Benchmark Results (SERVER GC) (Sep 0.9.0.0, Sylvan 1.3.9.0, CsvHelper 33.0.1.24)

AMD.Ryzen.9.9950X - PackageAssets Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Apple.M1.(Virtual) - PackageAssets Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Cobalt.100 - PackageAssets Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

PackageAssets with Quotes Benchmark Results

NCsvPerf does not examine performance in the face of quotes in the csv. This is relevant since some libraries like Sylvan will revert to a slower (not SIMD vectorized) parsing code path if it encounters quotes. Sep was designed to always use SIMD vectorization no matter what.

Since there are two extra chars to handle per column, it does have a significant impact on performance, no matter what though. This is expected when looking at the numbers. For each row of 25 columns, there are 24 separators (here ,) and one set of line endings (here \r\n). That's 26 characters. Adding quotes around each of the 25 columns will add 50 characters or almost triple the total to 76.

AMD.EPYC.7763 - PackageAssets with Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.7.PRO.7840U.w.Radeon.780M - PackageAssets with Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.9.5950X - PackageAssets with Quotes Benchmark Results (Sep 0.9.0.0, Sylvan 1.3.9.0, CsvHelper 33.0.1.24)

AMD.Ryzen.9.9950X - PackageAssets with Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Apple.M1.(Virtual) - PackageAssets with Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Cobalt.100 - PackageAssets with Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

PackageAssets with Quotes Benchmark Results (SERVER GC)

Here again are benchmark results with server garbage collection, which provides significant speedup over workstation garbage collection.

AMD.EPYC.7763 - PackageAssets with Quotes Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.7.PRO.7840U.w.Radeon.780M - PackageAssets with Quotes Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.9.5950X - PackageAssets with Quotes Benchmark Results (SERVER GC) (Sep 0.9.0.0, Sylvan 1.3.9.0, CsvHelper 33.0.1.24)

AMD.Ryzen.9.9950X - PackageAssets with Quotes Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Apple.M1.(Virtual) - PackageAssets with Quotes Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Cobalt.100 - PackageAssets with Quotes Benchmark Results (SERVER GC) (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

PackageAssets with Spaces and Quotes Benchmark Results

Similar to the benchmark related to quotes here spaces and quotes " are added to relevant columns to benchmark impact of trimming and unescape on low level column access. That is, basically " is prepended and appended to each column. This will test the assumed most common case and fast path part of trimming and unescaping in Sep. Sep is about 10x faster than CsvHelper for this. Sylvan does not appear to support automatic trimming and is, therefore, not included.

AMD.EPYC.7763 - PackageAssets with Spaces and Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.7.PRO.7840U.w.Radeon.780M - PackageAssets with Spaces and Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.9.5950X - PackageAssets with Spaces and Quotes Benchmark Results (Sep 0.9.0.0, Sylvan 1.3.9.0, CsvHelper 33.0.1.24)

AMD.Ryzen.9.9950X - PackageAssets with Spaces and Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Apple.M1.(Virtual) - PackageAssets with Spaces and Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Cobalt.100 - PackageAssets with Spaces and Quotes Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Floats Reader Comparison Benchmarks

The FloatsReaderBench.cs benchmark demonstrates what Sep is built for. Namely parsing 32-bit floating points or features as in machine learning. Here a simple CSV-file is randomly generated with N ground truth values, N predicted result values and nothing else (note this was changed from version 0.3.0, prior to that there were some extra leading columns). N = 20 here. For example:

For Scope=Floats the benchmark will parse the features as two spans of floats; one for ground truth values and one for predicted result values. Then calculates the mean squared error (MSE) of those as an example. For Sep this code is succinct and still incredibly efficient:

Note how one can access and parse multiple columns easily while there are no repeated allocations for the parsed floating points. Sep internally handles a pool of arrays for handling multiple columns and returns spans for them.

The benchmark is based on an assumption of accessing columns by name per row. Ideally, one would look up the indices of the columns by name before enumerating rows, but this is a repeated nuisance to have to handle and Sep was built to avoid this. Hence, the comparison is based on looking up by name for each, even if this ends up adding a bit more code in the benchmark for other approaches.

As can be seen below, the actual low level parsing of the separated values is a tiny part of the total runtime for Sep for which the runtime is dominated by parsing the floating points. Since Sep uses csFastFloat for an integrated fast floating point parser, it is >2x faster than Sylvan for example. If using Sylvan one may consider using csFastFloat if that is an option. With the multi-threaded (MT) ParallelEnumerate implementation Sep is up to 23x faster than Sylvan.

CsvHelper suffers from the fact that one can only access the column as a string so this has to be allocated for each column (ReadLine by definition always allocates a string per column). Still CsvHelper is significantly slower than the naive ReadLine approach. With Sep being >4x faster than CsvHelper and up to 35x times faster when using ParallelEnumerate.

Note that ParallelEnumerate provides significant speedup over single-threaded parsing even though the source is only about 20 MB. This underlines how efficient ParallelEnumerate is, but bear in mind that this is for the case of repeated micro-benchmark runs.

It is a testament to how good the .NET and the .NET GC is that the ReadLine is pretty good compared to CsvHelper regardless of allocating a lot of strings.

AMD.EPYC.7763 - FloatsReader Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.7.PRO.7840U.w.Radeon.780M - FloatsReader Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

AMD.Ryzen.9.5950X - FloatsReader Benchmark Results (Sep 0.9.0.0, Sylvan 1.3.9.0, CsvHelper 33.0.1.24)

AMD.Ryzen.9.9950X - FloatsReader Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)

Apple.M1.(Virtual) - FloatsReader Benchmark Results (Sep 0.12.0.0, Sylvan 1.4.3.0, CsvHelper 33.1.0.26)