.NET Incremental Generator that creates modified copies of unit and/or integration tests to update expected baselines in original tests, automating baseline creation and accelerating test development.
.NET Incremental Generator that creates modified copies of unit and/or integration tests to update expected baselines in original tests, automating baseline creation and accelerating test development.
Calculator class with a bug introduced by the buggyDelta variable:
public class Calculator
{
public static AddResult Add(int a, int b)
{
var buggyDelta = 1;
var result = a + b + buggyDelta;
return new AddResult
{
Value = result,
ValueAsHexString = "0x" + result.ToString("x"),
};
}
}
public class AddResult
{
public int Value { get; set; }
public string ValueAsHexString { get; set; } = string.Empty;
}
And its corresponding test with an expected baseline that matches the Calculator.Add buggy behavior:
public class CalculatorTest
{
[Xunit.Fact]
public void AddTest()
{
//Arrange
// incorrect `expected` baseline value will be overwritten with correct `actual` value
// after manual execution of auto-generated AddTestStormPetrel test.
var expected = new AddResult
{
Value = 5, //incorrect value example
ValueAsHexString = "0x5"
};
//Act
var actual = Calculator.Add(2, 2);
//Assert
actual.Should().BeEquivalentTo(expected);
}
}
When
The developer configures the test project with StormPetrel.Generator as per the Getting Started.
Then
A new test method, CalculatorTestStormPetrel.AddTestStormPetrel, is generated. This method is a specially modified copy of the original CalculatorTest.AddTest to overwrite its expected baseline.
CalculatorTestStormPetrel.AddTestStormPetrel is auto-generated after enabling Storm Petrel functionality.
When
The developer fixes buggyDelta to 0
and executes CalculatorTestStormPetrel.AddTestStormPetrel test.
Then
CalculatorTest.AddTest code is populated with correct expected baseline value, i.e. its code becomes
public class CalculatorTest
{
[Xunit.Fact]
public void AddTest()
{
//Arrange
// incorrect `expected` baseline value will be overwritten with correct `actual` value
// after manual execution of auto-generated AddTestStormPetrel test.
var expected = new AddResult
{
Value = 4,
ValueAsHexString = "0x4"
};
//Act
var actual = Calculator.Add(2, 2);
//Assert
actual.Should().BeEquivalentTo(expected);
}
}
So that
The developer should only review expected baseline changes (no manual modification) what typically saves development time.
AddTestGetExpected static method call returning incorrect expected baselines based on arguments;
AddTestGetExpected method possible variations per its comments.
public class CalculatorTestTheory
{
[Xunit.Theory]
[Xunit.InlineData(1, 5)]
[Xunit.InlineData(2, 2)]
[Xunit.InlineData(2, 3)]
public void AddTest(int a, int b)
{
//Arrange
var expected = AddTestGetExpected(a, b);
//Act
var actual = Calculator.Add(a, b);
//Assert
actual.Should().BeEquivalentTo(expected);
}
/// <summary>
/// Possible variations of AddTestGetExpected static method are:
/// - Method body may have pattern matches within pattern matches.
/// - Method body may have `switch` and/or `if` expressions with return statements returning expected baselines.
/// - The method may be placed in another class and/or file.
/// </summary>
private static AddResult AddTestGetExpected(int a, int b) => (a, b) switch
{
(1, 5) => new AddResult(), // should be overwritten with correct expected baseline after
// CalculatorTestTheoryStormPetrel.AddTestStormPetrel execution
(2, 2) => new AddResult(),
(2, 3) => new AddResult(),
_ => throw new InvalidOperationException(),
};
}
When
The developer executes CalculatorTestTheoryStormPetrel.AddTestStormPetrel test.
Then
CalculatorTestTheory.AddTestGetExpected code is populated with correct expected baseline values, i.e. its code becomes
private static AddResult AddTestGetExpected(int a, int b) => (a, b) switch
{
(1, 5) => new AddResult
{
Value = 6,
ValueAsHexString = "0x6"
}, //should be overwritten with correct expected baseline
(2, 2) => new AddResult
{
Value = 4,
ValueAsHexString = "0x4"
},
(2, 3) => new AddResult
{
Value = 5,
ValueAsHexString = "0x5"
},
_ => throw new InvalidOperationException(),
};
So that
The developer should only review expected baseline changes (no manual modification) what typically saves development time.
Use Case Variations
Expected variable expression is static property assignment
The attributes should be configured against data source methods or properties, not fields.
Snapshots of Snapshot Testing
HTML, JSON, binary or whatever expected snapshots can be hardcoded in tests code. See couple examples in SnapshotTest.
Expected expression decorators
Refer to the classes in the Scand.StormPetrel.Generator.Utils namespace and their usage in the files of the Test.Integration.Generator.Utils.XUnit project to decorate the expressions. The example includes:
CustomDumperDecorator. Replaces DateTime.ParseExact(...) with new DateTime(...) or a constant value.
DumperFactory. Combines implementations of IGeneratorDumper to have custom C# representations of expected baselines. It demonstrates how to remove redundant assignments and apply constant expressions, verbatim strings, raw string literals, implicit object creation, collection expressions and other C# syntax decorations to the baselines.
Built-in decorators include
Collection initializer
Replaces C# syntax nodes (e.g., array creation) with collection expressions.
Implicit object creation
Replaces C# constructor calls with implicit object creation syntax.
Raw string literals
Replaces regular strings with raw string literals. Optionally can add the literal comments like // lang=json according to the configuration.
Removal of property or field assignments
Removes property of field assignments according to the configuration.
Verbatim strings or custom literal values
Replaces regular strings with verbatim strings, regular literal values with custom literal values (e.g., regular numbers with hex numbers) according to the configuration.
Expected expression is inlined within an assertion
StormPetrel allows you to omit the expected variable entirely or use values that do not match the expected variable regex pattern. Instead, you can directly inline expected expressions of the following kinds:
Literal expressions (e.g., numeric or string literals);
Object creation expressions;
Prefix unary expressions (e.g., -1, !false);
Tuple expressions;
Non-Regex Variables: local variables or test method arguments that bypass the expected variable regex check. See examples in NoMatchToExpectedVarRegexTest.
Refer to the sections below for more details about supported assertions and examples of expected expressions.
Supported expressions include Assert.Equal(123, actual);, Assert.StrictEqual("string value", actual);, and Assert.Equivalent("string value", actual);. See more examples in NoExpectedVarAssertTest.
NUnit assertions
Supported expressions include Assert.That(actual, Is.EqualTo(123)); and Assert.That(actual, Is.EquivalentTo(new MyClass{...}));. See more examples in NoExpectedVarAssertThatTest.
MSTest assertions
Supported expressions include Assert.AreEqual(123, actual);. See more examples in NoExpectedVarAssertTest.
Shouldly expressions
Supported expressions include actual.ShouldBe(123); and actual.ShouldBeEquivalentTo(new object()); according to Shouldly documentation. See more examples in NoExpectedVarShouldlyTest.
Custom assertions that replicate the signatures above
You can also implement and use custom assertions. Ensure they replicate the signatures of the assertions mentioned above so that StormPetrel can detect them effectively. See CustomAssertTest for custom assertion examples.
See AutoFixtureTest for AutoFixture examples. Find AutoFixture.Xunit2.InlineAutoDataAttribute in the build script to see AutoFixture attributes configuration example for the environment variable what can be configured at process, user or machine level.
Playwright
See PlaywrightTest for Playwright test project configuration and examples. This project also showcases the Storm Petrel setup, enabling both traditional tests and file snapshot tests within the same project and even the same test class.
Other examples
The preceding sections highlight key test examples. While not exhaustive, Storm Petrel can also provide compatibility for the following frameworks/libraries in a similar manner:
Actual value is built via mocks or callbacks or other algorithms
Storm Petrel injects baseline update logic immediately after the last actual or expected value assignment.
When actual values are generated via:
Mocks (e.g., Moq/NSubstitute)
Callbacks
Complex algorithms
The baseline rewriting behavior may require test adjustments for reliable updates.
See BuildActualTest for:
✅ Working patterns for baseline updates
⚠️ Edge cases requiring modifications
Ref struct types support
Storm Petrel provides partial support for ref struct types (e.g., Span<T>, ReadOnlySpan<T>, UTF-8 string literals).
Supported Scenarios
Actual Value as ref struct with Conversion:
Use methods like ToArray(), Encoding.UTF8.GetString(), or other methods to convert ref struct values to regular types (e.g., arrays or strings) for assertions.
Expected Value as Regular Type:
The expected value must be a non-ref struct (e.g., string, T[]).
Direct ref struct Assertions. Reason: Storm Petrel and Dump Libraries API limitations against ref struct types.
See RefStructTest.Unsupported for examples and more explanations.
Getting Started
To utilize the StormPetrel tests, add the following NuGet Package references to your test project:
Scand.StormPetrel.Generator.
Object to C# code dumper. Typically, it represents actual test instance as C# code. See DumperExpression in Configuration for more details.
Option A. VarDump. Must be referenced in the case of StormPetrel.Generator default configuration (no appsettings.StormPetrel.json file in the test project). May be additionally configured.
Option C. Custom implementation of IGeneratorDumper interface. May be developed and configured.
Option D. Scand.StormPetrel.FileSnapshotInfrastructure. Typically, it represents the actual test instance as a checksum and writes the instance bytes to a snapshot file in the IGeneratorRewriter implementation. It may be referenced and configured according to its settings.
Configuration
Optional Visual Studio Extension
Use Visual Studio extension for configuring Storm Petrel. It provides a UI to set up test project settings (adds necessary NuGet packages and appsettings.StormPetrel.json file to the project), enabling automatic rewriting of expected baselines.
Alternative: Manually install packages and configure settings.
Optional JSON File
The StormPetrel Generator introduces several interfaces and classes to the Scand.StormPetrel.Generator.TargetProject namespace of the test project. These can be utilized alongside an optional JSON file to customize the rewriting of expected baselines. Key interfaces and classes include:
IGenerator, Generator;
IGeneratorBackuper, GeneratorBackuper;
IGeneratorDumper, GeneratorDumper;
IGeneratorRewriter, GeneratorRewriter.
Optionally appsettings.StormPetrel.json file (its Build Action should be C# analyzer additional file) can be added to a test project to configure Storm Petrel functionality.
The file name should match StormPetrel regex pattern and can be conditionally applied for Debug, Release, or other build configurations as demonstrated in Test.Integration.ObjectDumper.XUnit. Very first file is used in the case of multiple matches.
The file changes are applied on the fly and can have the following settings:
{
"$schema": "https://raw.githubusercontent.com/Scandltd/storm-petrel/main/generator/assets/appsettings.StormPetrel.Schema.json", // [optional] string, path to json schema.
"TargetProjectGeneratorExpression": "...", // [optional] string, configures the default `Generator`. An expression for the `IGenerator` instance.
"GeneratorConfig": // [optional] object to configure `Generator` behavior.
{
"BackuperExpression": "...", // [optional] string, instantiates `GeneratorBackuper` by default. An expression for the `IGeneratorBackuper` instance. Set to 'null' to skip creating backup files.
"DumperExpression": "...", // [optional] string, instantiates `GeneratorDumper` by default. An expression for the `IGeneratorDumper` instance. `GeneratorDumper` references [VarDump](https://www.nuget.org/packages/VarDump) stuff. Use
// - "new Scand.StormPetrel.Generator.TargetProject.GeneratorDumper(CustomCSharpDumperProvider.GetCSharpDumper())" to have `VarDump` with custom options. Need to implement `CustomCSharpDumperProvider.GetCSharpDumper()` method in this case.
// - "new Scand.StormPetrel.Generator.TargetProject.GeneratorObjectDumper()" expression for `GeneratorObjectDumper` instance which references [ObjectDumper.NET](https://github.com/thomasgalliker/ObjectDumper) stuff.
// - "new Scand.StormPetrel.Generator.TargetProject.GeneratorObjectDumper(CustomOptionsProvider.GetDumpOptions())" to have `ObjectDumper.NET` with custom options. Need to implement `CustomOptionsProvider.GetDumpOptions()` method in this case.
// - "new CustomClassImplementingIGeneratorDumper()" or similar expression to have totally custom implementation of dumping of an instance to C# code.
"RewriterExpression": "..." // [optional] string, instantiates `GeneratorRewriter` by default. An expression for the `IGeneratorRewriter` instance.
},
"IsDisabled": false, // [optional] boolean, false is by default. Indicates whether the generator should create 'StormPetrel' classes.
// Even if set to 'false', the generator still adds classes like 'IGeneratorDumper', 'GeneratorDumper' to avoid test project compilation failures
// in the case when custom classes uses them.
"IgnoreFilePathRegex": "...", // [optional] string, empty by default. Regular Expression to exclude certain paths from 'StormPetrel' class generation.
"IgnoreInvocationExpressionRegex": "...", // [optional] string, empty by default. Regular Expression to detect invocation expressions to not execute StormPetrel rewriting for.
// The property can be utilized in the case of custom IGeneratorRewriter implementations (e.g. when expected baseline is not stored in C# code but binary file as in File Snapshot Testing approach).
"IsAddNullableEnable": true, // [optional] boolean, true is by default. Indicates whether the generator should add `#nullable enable` and related directives under `#if !NETFRAMEWORK || SCAND_STORM_PETREL_NULLABLE_ENABLE` condition
// to auto-generated StormPetrel' files. This allows to avoid related compiler warnings/errors since auto-generated code has nullable reference types disabled regardless project global settings.
"Serilog": "...", // [optional] Logging configuration using Serilog (https://github.com/serilog/serilog-settings-configuration?tab=readme-ov-file#serilogsettingsconfiguration--).
// Defaults to logging warnings to the test project's Logs folder. Set to 'null' to disable logging.
// Use the '{StormPetrelRootPath}' token to indicate the target test project root path.
"TestVariablePairConfigs": [ // [optional] array of objects. Configures naming pairs for actual/expected variables/expressions to generate correct expected baselines.
{
"ActualVarNameTokenRegex": "[Aa]{1}ctual", // Default configuration object for actual-expected variable pairs. Assumes pair names like (expected, actual), (myExpected, myActual), (expectedOne, actualOne), (ExpectedTwo, ActualTwo), etc.
"ExpectedVarNameTokenRegex": "[Ee]{1}xpected", // Corresponds to the `ActualVarNameTokenRegex` for pairing.
},
{ // Default configuration object for 'Expected expression is inlined within an assertion' cases.
"ActualVarNameTokenRegex": null, // `"ActualVarNameTokenRegex": null` means any actual expression. Specify a regex to have more specific match of actual expressions if necessary.
"ExpectedVarNameTokenRegex": null, // `"ExpectedVarNameTokenRegex": null` indicates that Storm Petrel should analyze assertion expressions according to 'Expected expression is inlined within an assertion' cases.
}
// "TestVariablePairConfigs" elements order matters. For example, it detects `(expectedString, actualString)` pair (not `(expectedString, actualString.ToUpperInvariant())`)
// in a test containing `Assert.Equal(expectedString, actualString.ToUpperInvariant());` assertion what simultaneously corresponds to both default configuration elements above.
// Explicitly change the elements order in your file to prevail `(expectedString, actualString.ToUpperInvariant())` over `(expectedString, actualString)` if need.
]
}
Optional Environment Variable
To enable custom test attribute support in Scand.StormPetrel.Generator, you may set the SCAND_STORM_PETREL_GENERATOR_CONFIG environment variable. This JSON configuration will be read via Environment.GetEnvironmentVariable() during incremental generation. Configuration format:
{
"CustomTestAttributes": [ // Optional array of custom test attribute configurations
{
"TestFrameworkKindName": "XUnit", // Required: "XUnit", "NUnit", or "MSTest" (case-insensitive)
"FullName": "Xunit.UIFactAttribute", // Required: Full namespace-qualified attribute name (case-sensitive)
"KindName":"Test", // Required: "Test", "TestCase", or "TestCaseSource" (case-insensitive)
"XUnitTestCaseSourceKindName": "..." // Required for xUnit TestCaseSource: "MemberData" or "ClassData"
}
]
}
Configure the variable at process, user or machine level. IDE and/or incremental generator dotnet processes restart may be required to apply the configuration change.
See:
Does Scand.StormPetrel help developers track incorrect expected baselines in tests?
Developers can already track incorrect expected baselines without Scand.StormPetrel. They should manually review the changes made by Scand.StormPetrel to the expected baselines and decide if they are correct. This is the same approach used when Scand.StormPetrel is not involved.
Does Scand.StormPetrel violate Test Driven Development (TDD) practices?
We believe it does not. Here are the corresponding TDD steps with comments explaining why it does not violate the practices:
Write a test for the next bit of functionality you want to add. Scand.StormPetrel does not assist in creating test case input parameters.
It cannot help create the expected baseline at this step because there is no actual result yet. Therefore, the expected baseline is either:
A default instance
A manually created instance
Write the functional code until the test passes. Scand.StormPetrelcan automatically update the default or manually created instance with the actual value here. The developer should then review the changes to decide if the test truly passes.
What is the suggested configuration of Scand.StormPetrel for typical CI/CD and development environments?
The suggested configuration in the appsettings.StormPetrel.json file is:
{
"$schema": "https://raw.githubusercontent.com/Scandltd/storm-petrel/main/generator/assets/appsettings.StormPetrel.Schema.json", //Specify the path to observe schema suggestions in IDEs.
"GeneratorConfig": {
"BackuperExpression": null, //No need to backup because developers typically keep backups under Git or other Version Control Systems.
"DumperExpression": ..., //According to your requirements.
"RewriterExpression": ... //According to your requirements.
},
"IsDisabled": false, //Keep `true` under Git control for CI/CD to speed up test compilation and execution time.
//Keep `false` on the developer's machine, which should not be tracked by Git. StormPetrel tests are always available.
//Keep `true` on the developer's machine. Can be changed ad hoc to `false` to compile StormPetrel tests.
"Serilog": null //Avoid logging.
}
An alternative is to have conditionally applied appsettings.StormPetrel.Debug.json and appsettings.StormPetrel.Release.json files. The only suggested files difference is:
"IsDisabled": false for Debug version to have StormPetrel tests always available while development.
"IsDisabled": true for Release version to not have StormPetrel tests in CI/CD processes.
Can test project compilation fail after enabling Scand.StormPetrel? How can I avoid the failure?
Scand.StormPetrel relies on code syntax, not semantics. Therefore, it cannot properly generate StormPetrel test methods in all cases, and test project compilation might fail. You can detect the original test file causing the failure and add it to the "IgnoreFilePathRegex" property in the configuration to avoid the compilation error while still using Scand.StormPetrel for other tests.
Scand.StormPetrel constantly changes my expected baseline because the actual result has new property values (e.g., CreatedOn with the current time, auto-incremented Id, random Guid, etc.) every StormPetrel test call. How can I avoid these constant changes?
It is likely that you also ignore the property in the test assertion; otherwise, the test would fail.
An option is to always have a default value for the property while dumping it to C# code using Scand.StormPetrel.
This can be implemented via custom configuration or the implementation of IGeneratorDumper.
See an example of how this is implemented via the GetDumpOptions method in Test.Integration.XUnit/Utils and configured in Test.Integration.XUnit/appsettings.StormPetrel.json.
What Operating Systems and IDEs can I use to run StormPetrel tests?
You can definitely use Windows or Linux with:
Command line tools like dotnet test ... --filter "FullyQualifiedName~StormPetrel".
At SCAND, we specialize in building advanced .NET solutions to help businesses develop new or modernize their legacy applications. If you need help getting started with Storm Petrel or support with implementation, we're ready to assist. Whether you're refactoring or rewriting, our team can help solve any challenges you might face. Visit our page to learn more, or reach out for hands-on guidance.
