Captures output of a program to assist with debugging.
This project is tailored towards AWS services, and is not suitable for general purpose diagnostics.
Loads logs from services via a shared Correlation Id.
Can pull data from CloudWatch, S3, and DynamoDB.
stdout verb [options] --nolog
--nolog | -nl: Disable logging to the console.
CloudWatch
stdout cw --key appname --cid c6b8c804-34cb-4cf7-b762-d24b644831e9
--key | -k: The name of the configuration in app settings, that defines the log groups to query, and general filter rules.
--cid | -c: The Correlation Id to filter the logs by.
stdout cw --key appname --action createCustomer --actionkey appname
--key | -k: The name of the configuration in app settings, that defines the log groups to query, and general filter rules.
--action | -a: The name of the action to load correlation data from.
--actionkey | -ak: The name of the Load source to use from the configuration in app settings.
S3
stdout s3 --key appname --cid c6b8c804-34cb-4cf7-b762-d24b644831e9
--key | -k: The name of the configuration in app settings, that defines the bucket, and path prefix.
--cid | -c: The Correlation Id is part of (or all) of the key, that target files are found under the prefix, and correlation id.
stdout s3 --key appname --action createCustomer --actionkey appname
--key | -k: The name of the configuration in app settings, that defines the bucket, and path prefix.
--action | -a: The name of the action to load correlation data from.
--actionkey | -ak: The name of the Load source to use from the configuration in app settings.
stdout s3 --key appname --path assets/plaintext/c6b8c804-34cb-4cf7-b762-d24b644831e9
--key | -k: The name of the configuration in app settings, that defines the bucket, and path prefix.
--path | -p: The prefix key path of a S3 bucket, to retrieve files from (i.e. a static path, not merged with a correlation id at runtime).
DynamoDB
stdout db --key appname --cid c6b8c804-34cb-4cf7-b762-d24b644831e9
--key | -k: The name of the configuration in app settings, that defines the table name, and index to use.
--cid | -c: The Correlation Id stored in a table's index. Used to get the item's pk, and sk values; in order to load the db item.
stdout db --key appname --action createCustomer --actionkey appname
--key | -k: The name of the configuration in app settings, that defines the table name, and index to use.
--action | -a: The name of the action to load correlation data from.
--actionkey | -ak: The name of the Load source to use from the configuration in app settings.
stdout db --key appname --partitionkey pk_value
--key | -k: The name of the configuration in app settings, that defines the table name, and index to use.
--partitionkey | -pk: The Partition Key for an item.
stdout db --key appname --partitionkey pk_value --sortkey sk_value
--key | -k: The name of the configuration in app settings, that defines the table name, and index to use.
--partitionkey | -pk: The Partition Key for an item.
--sortkey | -sk: The Sort Key for an item.
Load
stdout ld --key appname --action action_name
--key | -k: The name of the configuration in app settings, that defines the data sources to load from.
--action | -a: The name of the action to load correlation data from.
Query
stdout qy --key appname
--key | -k: The name of the configuration in app settings, that defines the data sources to load from.
AppSettings
The appsettings.json file is found at the tool's installed location: %USERPROFILE%\.dotnet\tools
From there the relative path is: .store\md.stdout.cli\{VERSION}\md.stdout.cli\{VERSION}\tools\{RUNTIME}\any
Where {VERSION} is the installed package's version, i.e "2.0.0".
Where {RUNTIME} is the installed package's runtime, i.e. "net8.0".
Defaults are applied to all Sources that don't override the property value with their own.
The "app names" under Sources are matched to the --key command line argument.
If sensible defaults can be applied to all (or most) sources, then you would only need to set what's different for each source.
Custom settings can be applied for each "app" under Sources.
Each verb: cw, s3, and db, have their own Defaults, and Sources sections in app settings.
CloudWatch
Display: How you'd like to view the output; console or web browser (optional: console).
LogGroups: An array of log group names from AWS CloudWatch (required).
Limit: The maximum number of logs to return for a query (optional: 25).
RelativeHours: The number of hours to look backwards from "now" (optional: 1).
IsPresentFieldName: Selects logs with a particular field that must exist (optional: omitted from query).
CorrelationIdFieldName: The field name that contains an Id, that groups all logs together for a particular request (optional: omitted from query).
Fields: The CloudWatch fields to select from the query (optional: @timestamp, @message).
Filters: Clauses to add to the query, each filter will be in the form: "and key = value" (optional: omitted from query).
S3
Display: How you'd like to view the output; console or web browser (optional: console).
Bucket: The S3 buckname name, where your logging / debugging output files are stored (required).
Prefix: The key path where your files for a particular request can be found under. The Correlation Id from the command line is merged with <CID> (optional: when not using a correlation id).
ContentType: The expected file contents, used for pretty printing / formatting; only json, and text are supported for now (optional).
Files: The filenames to download, if found under the prefix path (optional: downloads all matches).
DynamoDB
Display: How you'd like to view the output; console or web browser (optional: console).
TableName: The name of the DynamoDB table (required).
PartitionKeyName: The table's Partition Key name (required).
SortKeyName: The table's Sort Key name (optional).
IndexName: The table's index name, where the correlation id makes up part of the index's pk, or sk (optional: when not using a correlation id).
IndexPartitionKeyName: The name of the index's Partition Key (optional: when not using a correlation id).
IndexPartitionKeyMask: The format of the index's pk, the correlation id from the command line is merged with <CID> (optional: when not using a correlation id).
IndexSortKeyName: The name of the index's Sort Key (optional).
IndexSortKeyMask: The format of the index's sk, the correlation id from the command line is merged with <CID> (optional).
Projection: The item(s) attribute(s) to select (optional: returns all attributes).
Load
Display: How you'd like to view the output; console or web browser (optional: console).
StdOut:Key:Application: The name of the program / service that stored the correlation Id (i.e. customer_service) (required).
StdOut:Key:Environment: The stage the request ran in (optional: empty).
StdOut:Key:User: An identifier for the user that ran the request (optional: empty).
StdOut:Sources:Disk:RootPath: The local disk root path, where the key is used to get the correlation Id (optional: skip Disk config).
StdOut:Sources:S3:Bucket: The name of the bucket in S3, where the key is used to get the correlation Id (optional: skip S3 config).
StdOut:Sources:S3:Prefix: A prefix to prepend to the search key (optional: empty).
StdOut:Sources:DynamoDb:TableName: The name of the DynamoDB table (optional: skip DynamoDB config).
StdOut:Sources:DynamoDb:PartitionKeyName: The name of the Partition Key in the DynamoDB table (where the key is stored) (optional: skip DynamoDB config).
StdOut:Sources:DynamoDb:SortKeyName: The name of the Sort Key in the DynamoDB table (where the key's action is split out, and stored) (optional: depending on the table setup).
Nuget Package
Where as the .NET CLI: md.stdout.cli, is used to read logs, tables, and text assets given a correlation Id.
This package: md.stdout, is used to store correlation Ids, for a given service, environment, user, and action; to retrieve later on.
The main purpose is to help find the correlation Id after some action happens, on some environment, in some program, by some user.
For this combination (known as a StorageKey), only the latest correlation Id is kept; any existing correlation Id for said key is overwritten.
Store
/// <summary>Save a "key" to storage (Disk, S3, or DynamoDB), with a payload containing the request's correlation Id.</summary>
/// <param name="config">Settings for data storage services, in order to persist the last used correlation Id.</param>
/// <param name="key">A deterministic value, which allows you to find a the last used correlation Id, for a particular application, and action.</param>
/// <param name="correlationId">The unique identifier assigned to this request, that allows you to track logs across multiple services.</param>
/// <returns>A valid response, if the payload was stored under the configured data source, with the key created from the merged parameters.</returns>
Task<Response<Unit>> Store(StdConfig config, StorageKey key, string correlationId);
Load
/// <summary>Load a "key" from storage (Disk, S3, or DynamoDB), to get the most recent correlation Id stored for said key.</summary>
/// <param name="config">Settings for data storage services, to retrieve the last used correlation Id.</param>
/// <param name="key">A deterministic value, which allows you to find a the last used correlation Id, for a particular application, and action.</param>
/// <returns>The most recent correlation Id for the given key, and source(s). Otherwise NotFound, or an invalid response on error.</returns>
Task<Response<Either<string, NotFound>>> Load(StdConfig config, StorageKey key);
Query
/// <summary>Query a "key" (without an action) from storage (Disk, S3, or DynamoDB), to get all actions using the application, environment, and user prefixes.</summary>
/// <param name="config">Settings for data storage services, to retrieve the last used correlation Id.</param>
/// <param name="key">A deterministic value, which allows you to find a the last used correlation Id, for a particular application.</param>
/// <returns>0 to many keys, that can be used in the Load() method, to gather the related correlation Ids.</returns>
Task<Response<StorageKey[]>> Query(StdConfig config, StorageKey key);
AppSettings
Since md.stdout is a package in your program, you're free to add the settings however you prefer; below is just one option.
The package won't read directly from any configuration source.
You can either pass the config model directly as an overload option on the IStdOut methods, or use the dependency injection extension method (below):
// Use one provided by a framework (asp.net), or create your own application; or web builder.
var builder = Host.CreateApplicationBuilder();
// Load both: "Sources", and "Key" using the configuration builder.
var options = new StdConfigOptions();
builder.Configuration.GetSection(StdConfigOptions.SECTION_NAME).Bind(options);
// Provide the config values to StdOut.
builder.Services
.AddStdOutServices(
opt => {
opt.Sources = options.Sources;
opt.Key = options.Key;
}
);
var host = builder.Build();
host.Services.AddContainerExpressionsLogging();
var stdout = host.Services.GetRequiredService<IStdOut>();
Only one source is required for stdout to store, load, and query correlation Ids.
Both S3, and DynamoDb sources have the option to delete old entries (i.e. PurgeObjectVersions, and TTL).
Each source has an OperationDissect property.
By default when a source has been configured, it will be used for each api operation (i.e. store, load, and query).
If you only want a particular source to say: read, and query, but not to store; then you'd add "store" to the source's dissect property, as you are "removing it".
Disk
RootPath: The local disk root path, where the key is used to store the correlation Id (required).
OperationDissect: Any dissections marked here will not be used when attempting the store, load, or query operations (optional: none).
S3
Bucket: The name of the bucket in S3, where the key is used to store the correlation Id (required).
Prefix: A prefix to prepend to the Storage Key (optional: empty).
PurgeObjectVersions: When true, removes all existing object versions after uploading a new one (optional: false).
OperationDissect: Any dissections marked here will not be used when attempting the store, load, or query operations (optional: none).
DynamoDb
TableName: The name of the DynamoDB table (required).
PartitionKeyName: The name of the Partition Key in the DynamoDB table (where the key is stored) (required).
SortKeyName: The name of the Sort Key in the DynamoDB table (where the key's action is split out, and stored) (optional: depending on the table setup).
TimeToLiveName: The name of TTL attribute for the table (optional).
TimeToLiveHours: The minimum time this item should be kept for (optional).
OperationDissect: Any dissections marked here will not be used when attempting the store, load, or query operations (optional: none).
Key
Application: A name to represent the program / service storing the correlation Id (i.e. customer_service) (required).
Environment: The stage the request is running in (optional).
Namespace: Pulls the action from the top level calling method defined in said namespace (optional: must provide action in Storage Key parameter instead).
Offset: Offset may be used to go "down" a function call, when using the namespace (i.e. to skip middleware etc) (optional: must provide action in Storage Key parameter instead).
