Play Nicely - SpecFlow Executor

The Play Nicely SpecFlow projects provide Gherkin language bindings to PlayNicely.Executor artefacts. These bindings allow you to define scenarios, including pre-conditions, actions and expected outcomes in domain specific language. Scenarios are designed to support building release tests for Play Nicely npm package based projects.

Getting Started

To start using these bindings in a SpecFlow project, install this package, or one of the more specific packages, as a dependency. Then add or update the specflow.json so that the project picks up the bindings.

With this done, you can use the bindings to specify your scenarios.

Install the package

From the Package Manager Console:

Install-Package PlayNicely.SpecFlow.Executor

Or use one of the more specific PlayNicely.SpecFlow packages.

Add or Update specflow.json

For SpecFlow to pick up the bindings from external assemblies, they have to be configured in the project's specflow.json file. If the project doesn't already have it, add the item to the root of your SpecFlow project.

Add the external assemblies to the file:

{ "stepAssemblies": [ { "assembly": "PlayNicely.SpecFlow.Executor" } ] }

You can now use the step definitions and bindings from the package.

Creating Test Scenarios

Test scenarios can be specified using the step definitions defined within this, and the more specific, packages. To create those scenarios, the Gherkin language bindings need to make use of several artefacts from the PlayNicely.Executor projects.

Several hooks and ScenarioContext extension methods have been defined to simplify this interaction. They ensure a test environment can be easily configured, and support collection of execution results, which are used for any testing assertions. They are explained in more depth here.

ℹ️ Working examples
The PlayNicely.NpmNpx and PlayNicely.Sass projects are real examples of how to set up your projects and configure test scenarios.

Step Definitions

This package provides the following step definitions.

Given

• Given command (command) is installed
  Ensures that command is on the $PATH in the test environment.
• Given command (command) is uninstalled
  If the command is found on the $PATH, its directory will be removed from the $PATH in the test environment. This simulates the command being unavailable (uninstalled). For a more complete description of how this works see this page.

When

• When command (command line) is executed
  Executes a process based on the command line provided. The command line is parsed based on the Linux shell rules, \ is the escape character and both " and ' are accepted as quotes.

Then

• Then the command should [succeed|fail]
  Test whether the process exited successfully or not.
• Then the command should return exitcode (exit code)
  Test the exit code value returned by the process.
• Then working directory should contain [directory|file|path] (relative/path)
  Test to ensure a directory, file, or either exists relative to the test execution working directory.
• Then working directory should not contain [directory|file|path] (relative/path)
  Test to ensure a directory, file or either does not exist relative to the test execution working directory.
• Then [stderr|stdout] should contain (text)
  Test that the standard error, or output, stream of the process contains text.
• Then [stderr|stdout] should match (regex)
  Test the standard error, or output, stream of the process to ensure regex matches.

Tags, Hooks and Extensions

This package defines a tag, @test‑environment, that simplifies creation of required objects, collection of scenario results and output of any diagnostic information, in the event of test failure.

⚠️ No tag, no definitions
Without this tag defined on a scenario or feature, many of the step definitions within this project will not be available.

This simplification is achieved using scoped step definitions, hooks and ScenarioContext extension methods.

Test Environment Setup

The Given steps of these packages are scoped with the @test‑environment tag.

If a feature, or scenario, has been tagged with @test‑environment, the Given definitions and the hook EnvironmentBuilderHooks are in scope. This ensures that a TestEnvironmentBuilder object is instantiated before each scenario and available within the ScenarioContext dictionary, the Given steps rely on this object.

The TestEnvironmentBuilder can be required by any step definition using the _scenarioContext.RequireEnvironmentBuilder() extension method.

Execution and Results

The When steps of these packages are also scoped with the @test‑environment tag. These steps use the defined TestEnvironmentBuilder to build the ITestEnvironment. With the environment built, a ITestEnvironmentRunner<T> is created, and executed against that environment.

ℹ️ Run a process
For this package, it is a basic command line defined process runner.

The ExecutionResultExtensions class provides extension methods to collect ExecutionResult objects (in the ScenarioContext) and retrieve them from any Then steps.

The extension methods support the base ExecutionResult and also any generic variant of ExecutionResult<T>.

Failed Test Diagnostics

The ExecutionResultHooks class is always in scope and runs after every scenario. If a test fails, and an ExecutionResult has been collected, the details of that result will be reported as part of the test output.

Managing Object Lifecycle

The DisposableExtensions class has extension methods that MarkForDisposal any IDisposable objects created or collected during testing. The DisposalHooks class will dispose any objects that are MarkForDisposal after each scenario.

Constants

The SpecFlowConsts class defines constants for tags and binding order.

ℹ️ Need to be very specific about hook order?
You shouldn't need to worry about these constants, unless you write a package that depends on this one, and you need to ensure the order of the SpecFlow hooks.