'One of the key features of Amazon CloudWatch Logs is the ability to store and monitor logs securely. Users can configure log data to be encrypted at rest using AWS Key Management Service (KMS) to ensure that sensitive data is protected.

As an experienced Cloud and DevOps Engineer, I have observed that sensitive data can sometimes inadvertently be written to logs. During the development or testing phase, developers may include incoming or outgoing data in their logs for troubleshooting purposes. However, when deploying to a production environment, debugging code that contains sensitive data can be overlooked and left in the logs, posing a significant risk to data security.

Fortunately, AWS has recently introduced a game-changing feature that enables the detection and protection of sensitive information in CloudWatch logs. In the following sections, I'll walk you through how to use this powerful tool to secure your log data and protect your organization's sensitive information.

Create a lambda that prints sensitive data

Created a Python 3.9 lambda with this simple piece of code that write email address to the logs. You may even write credit card information, date of birth, etc., to the logs.

import json

def lambda_handler(event, context):

    print('hello@gmail.com')

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Upon running the lambda a few times, the sensitive information is printed in the logs:

Detect and Protect Sensitive Data

Head over to the lambda log group and find the "Data protection" tab.

Add a new policy with 'Email Address'. There are plenty of options like address, date of birth, credit card and so on, to choose from.

Currently, there are several audit destinations available, including S3, CloudWatch Log Group, and Kinesis. To store the audit information securely, you can create a new CloudWatch Log Group specifically for this purpose. This will allow you to monitor and analyze audit data effectively, providing valuable insights into the security of your environment.

Once data protection is enabled, we can see the sensitive information count. It doesn't seem to scan the existing logs. Fair enough.

Let's head over to the lambda and execute it a couple of times.

A few seconds later, the count of sensitive data increased to 2:

In the "sensitive-data-log-group", one can see what kind of sensitive information is emitted by the resource.

In the log group for lambda, Cloudwatch cleverly masks the sensitive information: