CloudWatch Log Mailer

Feeding logs from various AWS services to CloudWatch Logs is a common pattern. From Lambda logs, through ECS Task logs, to Linux /var/log/* messages forwarded by awslogs daemon, they can end up in CloudWatch. How can we monitor them – and get alerted – when something of interest gets logged?

People often use Elastic Search, Splunk, Graylog or similar heavy weight solutions. But do we really need that? Very often all we want is to know when a certain keyword, e.g. “Error”, appears in the messages. Luckily that’s very easy to do.

CloudWatch Logs support notification Filters – when a pattern of interest (e.g. the keyword Error) appears in one of the log group’s Log Streams the Filter can trigger an action, e.g. send the matching record to a SNS topic. Such an alert is usually quite terse – just one matching line from the logs, no context and all that wrapped in a JSON format. It’s not something you’d want to send to your support team.

However we can leverage the built-in Filtering functionality and with a bit of Lambda coding generate a lot more user friendly alert emails.

CloudWatch LogMailer is a simple Lambda function that gets called when a message in the monitored Log Group matches the specified Filter. The Lambda then downloads the last few messages from the matched Log Stream to provide some context and emails it to the Operator.

It’s completely serverless, low-maintenance and very low cost solution for simple log monitoring.

At the high level we’ve got:

• Log Group with Log Streams from your logging source (e.g. Lambda function).
• Subscription Filter that looks for specified keywords, e.g. for the word Error
• LogMailer Lambda function that when receives the info about the match and downloads a few minutes of the previous logs to provide some context
• The Lambda then builds an email message and sends it to SNS for delivery

I like to deploy all my stuff using CloudFormation because it facilitates reusable, reproducible, all-or-nothing, easy-to-delete deployments.

The complete code is in my aws-cloudwatch-logmailer GitHub repository where you’ll find several files:

logmailer-lambda.py

This is the actual Lambda function written in Python.

logmailer-cloudformation.yml

CloudFormation template that creates all the infrastructure (Lambda, SNS Topic, IAM Roles, etc).

The Lambda code is embedded in the template for convenience.

Note that the actual LogGroup that we want to monitor is not part of this template. The idea is that you’ll have one central Log Mailer lambda function and use it to monitor many different LogGroups in your AWS environment.

To demonstrate how to use the central Log Mailer created from the above template have a look at the sample tester-cloudformation.yml. It creates the usual Lambda-related resources but also a these three resources needed for LogMailer integration:

  ## Lambda can create its own LogGroup but we explicitly pre-create it here
  ## - for easier integration with LogMailer
  ## - to make it CloudFormation controlled
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${TesterLambda}"
      RetentionInDays: 7

  ## Subscription Filter specifying the actual keyword pattern to look for
  ## - Update FilterPattern as required for each LogGroup
  LogMailerFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      FilterPattern: "?ERROR ?Error ?error ?FAIL ?Fail ?fail ?FATAL ?Fatal ?fatal ?Traceback"
      LogGroupName: !Ref LogGroup
      DestinationArn: !Ref LogMailerLambdaArn       # <== The LogMailer Lambda ARN

  ## Give CloudWatch Logs the permission to call LogMailer Lambda
  LogMailerTriggerPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref LogMailerLambdaArn         # <== The LogMailer Lambda ARN
      Action: lambda:InvokeFunction
      Principal: logs.amazonaws.com
      SourceArn: !GetAtt LogGroup.Arn

Put these three resources in each CloudFormation template and you'll have get a low-cost, low-maintenance log monitoring solution.

If you deployed both the above templates you will soon receive an email from Log Mailer notifying you that there was an Error keyword found in the Tester Log Stream...

Enjoy 🙂