Elasticsearch logger provider for Microsoft.Extensions.Logging. Writes direct to Elasticsearch using the Elastic Common Schema (ECS), with semantic logging of structured data from message and scope values, for use with the Elasticsearch-Logstash-Kibana (ELK) stack. The results can be viewed and queried in the Kibana console.
Elasticsearch-Logstash-Kibana (ELK) stack logger provider for Microsoft.Extensions.Logging.
Writes direct to Elasticsearch using the Elastic Common Schema (ECS),
with semantic logging of structured data from message and scope values. The results can be viewed and queried in the Kibana console.
NOTE: To use version 9.0.0 of this package you need atleast version 8.15.0 of the Elastic Stack.
Usage
Add a reference to the Elastic.Extensions.Logging package:
dotnet add package Elastic.Extensions.Logging
Then, add the provider to the loggingBuilder during host construction, using the provided extension method.
using Elastic.Extensions.Logging;
// ...
.ConfigureLogging((hostContext, loggingBuilder) =>
{
loggingBuilder.AddElasticsearch();
})
The default configuration will write to a local Elasticsearch running at http://localhost:9200/.
Once you have sent some log event, open Kibana (e.g. http://localhost:5601/) and define an
index pattern for "dotnet-*" with the time filter "@timestamp".
You can then discover the log events for the index. Some useful columns to add
are log.level, log.logger, event.code, message, tags, and process.thread.id.
If you are running multiple applications or on multiple servers, you might want to
include service.type, service.version, and host.hostname.
Additional fields are defined below, and all individual message and scope values are logged
as custom key/value pairs, e.g. .
labels.*
labels.CustomerId
Basic configuration
For deployment you will usually want to override the configuration with your actual server location(s).
The other useful value to configure is a tag for the environment, e.g. Development/Staging/Production.
DateTimeOffset when the message was logged, including local offset.
message
string
The formatted log message and arguments.
tags
array
Custom tags from configuration, e.g. [ "Staging", "Priority" ]. Can have multiple values.
event.action
string
The name of the logged EventId, e.g. ErrorProcessingCustomer.
event.code
string
The numeric value (as a string) of the EventId, e.g. 5000.
event.severity
long
The syslog severity corresponding to the log level, 2 = critical, 3 = error, 4 = warning, 6 = information, 7 = debug and trace. (Also used in the Systemd format of ConsoleLoggerProvider)
log.level
string
The log level: Critical, Error, Warning, Information, Debug, or Trace.
log.logger
string
The category name (namespace and class) of the logger, e.g. HelloElasticsearch.Worker.
The event.severity field is numeric and can be used to order events by level, e.g. Kibana query event.severity <= 4 will get all messages that have log level Warning or worse.
Error fields
If the log message includes an exception, the details are reported in the error fields.
Field
Type
Description
error.message
string
The Message property of any exception.
error.stack_trace
string
Full details of the exception, Exception.ToString(), including the stack trace and the stack trace of any inner exceptions.
error.type
string
The type of the error message, e.g. System.DivideByZeroException
Custom fields
Following the ECS conventions, these use alternative Title casing, to identify them as non-standard fields.
Field
Type
Description
MessageTemplate
string
The original message template, e.g. "Unexpected error processing customer {CustomerId}."
Scopes
array
Array of string formatted scope values, in the order added.
Label values
Field
Type
Description
labels.*
string
Custom key/value pairs of all named parameter values and named scope values. All values are strings (no nested objects).
Label values can be accessed by their key, for example if the message, or scope, includes a parameter CustomerId, then the value will be logged as labels.CustomerId can be searched in Kibana using "labels.CustomerId: 12345".
Example:
The following will generate two labels, labels.EndTime from the message and labels.CustomerId from the scope:
using (_logger.BeginScope("{CustomerId}", customerId))
{
_logger.LogWarning("End of processing reached at {EndTime}.", end);
}
Labels are taken from the message (state) and any scope values (may be disabled via the configuration options). In Microsoft.Extensions.Logging an
internal FormattedLogValues is used in the ILogger overloads for log levels and scopes; it implements the IEnumerable<KeyValuePair<string,object>>
interface that is used to extract the individual parameter values.
The labels property in ECS should not contain nested objects, so values are converted to keyword strings. For most objects this is just calling ToString(),
with specific formats for some types, e.g. calling string on a list is usually not very useful, so the contents of the list is logged instead.
Labels value formatting
Type
Formatting
byte
Hex, e.g. "9A"
byte[]
Prefixed hex, e.g. "0x12789AF0"
DateTimeOffset
ISO format, e.g. "2020-01-02T03:04:05.000000+06:00"
DateTime
In most cases DateTimeOffset should be used instead (1). Where DateTime is used for date only (with no time component), it is formatted as a date, e.g. "2020-01-02". If it has a time component, the roundtrip ("o") format is used.
IEnumerable
Values separated by ", " (configurable)
IDictionary<string, object>
A string containing key value pairs, e.g. token="0x12789AF0" count="5"
other values
The result of ToString(), including scalar values, e.g. the number 5.3 is logged as the string "5.3"
These identify the version of the logger provider being used.
Field
Type
Description
agent.type
string
Name of the logger provider assembly, Elastic.Extensions.Logging.LoggerProvider.
agent.version
string
Informational version number of the logger assembly, e.g. 1.1.1+bd3ad63.
ecs.version
string
Version of ECS standard used, currently 1.5.
Service fields
This identifies the application/service that is running and generating the logs.
The values are pulled from the entry assemb, Assembly.GetEntryAssembly(), using the Name
and AssemblyInformationalVersionAttribute values (if informational version is not set
it falls back to assembly Version).
Field
Type
Description
service.type
string
Name of the entry assembly, HelloElasticsearch.
service.version
string
Informational version number of the entry assembly, e.g. 1.2.0-beta.1+79d095a.
Note: You should be using a build process that sets the assembly informational version correctly.
e.g. If you have a dotnet project using git you can install the local tool GitVersion.Tool,
and use it to automatically generate semantic version numbers from the git branch information.
To install the tool:
dotnet new tool-manifest
dotnet tool install GitVersion.Tool
Then use the tool to create a semantic version number that can be used in your build process:
dotnet tool restore
dotnet gitversion
You are welcome to use the build.ps1 script in this repository as an example.
Tracing fields
Field
Type
Description
trace.id
string
Cross-service trace correlation identifier. From Activity.Current.RootId from System.Diagnostics, with a fallback to CorrelationManager.ActivityId. Can be overridden by a message or scope value trace.id.
transaction.id
string
Transaction for this service, e.g. individual request identifier. If in W3C format, parse out the SpanId from Activity.Current.Id from System.Diagnostics, otherwise just use the full Activity.Current.Id (e.g. if hierarchical). Can be overridden by message or scope value transaction.id.
ASP.NET will automatically pass correlation identifiers between tiers; from 3.0 it also supports the W3C Trace Context standard (https://www.w3.org/TR/trace-context/).
The value of Activity.Current.RootId is used as the cross-service identifier (in W3C format this is the Trace ID),
if in W3C format the Span ID portion of Activity.Current.Id is used for the transaction, otherwise the full value is used (this is consistent with the way ASP.NET works).
It is recommended to turn on W3C format, for compatibility with other systems:
Activity.DefaultIdFormat = ActivityIdFormat.W3C;
Host fields
NOTE: Can be disabled via configuration.
Field
Type
Description
host.architecture
string
The processor architecture, e.g. X64. Value of RuntimeInformation.OSArchitecture.
host.hostname
string
The computer name. Value of Environment.MachineName.
host.os.full
string
Full description of the operation system. Value of RuntimeInformation.OSDescription.
host.os.platform
string
Operating system platform. Value of Environment.OSVersion.Platform.
host.os.version
string
Operating system version. Value of Environment.OSVersion.Version.
Process fields
NOTE: Can be disabled via configuration.
Field
Type
Description
process.name
string
The current process name. From Process.GetCurrentProcess().
process.pid
long
The current process ID. From Process.GetCurrentProcess().
process.thread.id
long
Current thread ID. Value of Thread.CurrentThread.ManagedThreadId.
process.thread.name
string
Name of the thread. From Thread.CurrentThread.Name.
User fields
NOTE: Can be disabled via configuration.
Field
Type
Description
user.domain
string
The current domain, either the machine name or a Windows domain. Value of Environment.UserDomainName.
user.id
string
Current user principal name, if set. Value of Thread.CurrentPrincipal.Identity.Name.
