AWSWatchman creates and maintains AWS CloudWatch alerts.
Dynamic monitoring. This program creates and maintains CloudWatch alerts for infrastructure in AWS. It covers DynamoDB tables, SQS queues and more.
The details of who to alert and what tables to alert on must be stored in configuration files.
:warning: Be careful. This code, when used correctly, will modify your AWS account by adding CloudWatch alerts to multiple resources. By default it will do a dry run which will tell you what alarms would be added. You must add to enable writes.
--RunMode GenerateAlarms
Why
Unmonitored infrastructure is a problem waiting to happen, so all infrastructure in AWS should have appropriate alerts via CloudWatch. It is usually possible to declare all these alarms with the resource upfront in CloudFormation. But this is not always the best way to do it. AWSWatchman is good for cases where the CloudFormation definitions are harder to use. For example:
Dynamically scaled resources such as DynamoDB tables where the table can be scaled up
Dynamically created resources such as SQS queues, where the queue can be subscribed to a topic at runtime by code.
Verifying existing resources by scanning all resources in the system and programmatically identifying ones that do not have alarms, rather than inspecting all the CloudFormation text that created them.
If you are using the new resource types and have a high number of alarms you will need to specify and S3 bucket/path that the CloudFormation template can be deployed to. The AWS credentials will need permissions to put objects into that location.
RunMode: One of TestConfig, DryRun, or GenerateAlarms. Optional, default is DryRun. Mode behaviours are:
TestConfig: Configuration files are loaded and validated. Used to test syntax of changes to configuration. AWS credentials are not needed.
DryRun: All actions short of writing alarms are performed. Used to test what the effects of the configuration will be on the AWS account.
GenerateAlarms: A full run of the program. You must specify GenerateAlarms in order to actually write alarms.
AwsAccessKey and AwsSecretKey. Optional. Supply both of these parameters in order to specify AWS credentials on the command line.
AwsProfile Specify a named AWS profile to use for credentials. Optional.
AwsRegion The AWS region to use. Optional, default is eu-west-1.
ConfigFolder: The ppath to the configuration files. Required.
Verbose: One of true or false. Give more detailed output. Optional, default is false.
WriteCloudFormationTemplatesToDirectory. If set, alarms deployed via CloudFormation will be written to this folder instead of deployed. Note that this does not affect SQS and DynamoDB alarms which currently use a different deployment method.
AwsLogging. Enable AWS SDK logging. Default is false. If true, AWS metrics and error responses are logged to the console.
AWS connection credentials will be found in the following order:
If AwsAccessKey and AwsSecretKey are specified, these will be used.
If AwsProfile is specified, the named profile will be used.
Fallback to configuration file, default profile credentials, federated credentials and environment variables.
Test configuration run
Test that the configuration files can be read and pass validation. AWS credentials are not required for this.
When run, Watchman does things in approximately this order:
Validate command-line parameters, and stop if they are invalid.
Read the configuration folder and load all alerting groups.
Validate the alerting groups in the configuration. Stop if the configuration is invalid. Stop if the run mode is TestConfig.
For each resource type, across all alerting groups:
Populate resource names. Expand regular expression patterns into full resource names that match the pattern.
For the set of alarm definitions for each service (e.g. DynamoDB, SQS, RDS), apply either the default threshold from those definitions, or from the alerting group, or from the specific resource definition, depending on what is defined.
Create alarm models and create CloudFormation from them.
Commit the changes. Skipped if the run mode is DryRun.
Report on "orphans", i.e. resources that are not covered by any alarms in alerting group, excluding "CatchAll" groups.
Quartermaster
Quartermaster is a reporting tool to examine your DynamoDB usage. It will send a weekly "Dynamo provisioning report" email to the reporting targets. This lists the read and write capacity, provisioned and peak of actual usage, across all all the DynamoDB tables and their indexes listed in the alerting group for the last week, and the usage as a percentage of provisioned capacity.
The percentage use can be used to track which tables and indexes are approaching the threshold, and which are over-provisioned. The intention is to allow you to change capacities up or down without first needing an alert.
