How to Differentiate Manual Changes from Terraform Changes in S3 Bucket

Objective

In this article we will learn how to track manual changes to S3 bucket which is managed by Terraform

• SNS notifications are sent only for manual changes (like uploads or deletions) made to an S3 bucket through the AWS Console.
• Terraform changes to the S3 bucket should not trigger SNS notifications.

Components and Workflow

1. S3 -> CloudTrail -> EventBridge -> Lambda -> SNS

• CloudTrail logs events for S3 actions (uploads, deletions, bucket policy changes).
• EventBridge listens for these events from CloudTrail.
• Lambda function processes these events, filtering out those made by Terraform.
• The Lambda sends SNS notifications only for manual actions.

1. Create a CloudTrail to Log Events

CloudTrail is used to record all API activities, including manual and Terraform changes.

Step-by-Step:

1. Open CloudTrail Console:
2. Go to the AWS CloudTrail console.
3. Create a New Trail. Click on Create trail. Name: Provide a name for your trail (e.g., S3ManualChangeTrail).
4. Configure S3 Bucket to Store Logs:

• Specify an S3 bucket where CloudTrail will store its logs. You can use an existing bucket or create a new one. Ensure permissions are set for CloudTrail to write to the bucket.

5. Enable Event Logging:

• Ensure Management Events are enabled to track S3 bucket API activities.
• Under Read/Write events, choose Read/Write to capture both.

6. Finish Creating the Trail : Click Create trail to save.

This will capture all actions taken on your S3 buckets.

2. Create EventBridge Rule for Specific S3 Events

Use EventBridge to filter S3 events and trigger Lambda for further processing.

Step-by-Step:

1. Open EventBridge Console:

• Go to the AWS EventBridge console.

1. Create a Rule:

• Click on Create rule.
• Name: Provide a name (e.g., S3ManualEventRule).

2. Define Event Source:

• Under Event source, select AWS events.
• Event pattern: Choose S3 as the source and filter specific events:
• PutObject, DeleteObject, and other relevant S3 events.

3. Add User Identity Filter (Optional):

• You can further filter by userIdentity to ignore automated actions like Terraform by excluding actions from the role used by Terraform.

4. Set Target:

• Select Lambda function as the target.

5. Finish and Create Rule.

Event pattern which has all S3 events including bucket level and object level

Event target as Lambda function

3. Create Lambda Function to Filter Events and Send SNS

Lambda will process the EventBridge events and send notifications to SNS for manual changes only.

Step-by-Step:

1. Open Lambda Console:

• Go to the AWS Lambda console.

2. Create a Function:

• Click on Create function.
• Choose Author from scratch.
• Name the function (e.g., s3notify-terraform).
• Runtime: Choose Python 3.8 (or any preferred runtime).
• Execution role: Create a new role with basic Lambda permissions.

3. Add Code:

• Place the following Lambda function code that filters for manual changes:

import boto3
import json

sns = boto3.client('sns')
def lambda_handler(event, context):
    # Extract userIdentity from event to check if it's Terraform
    user_identity = event['detail']['userIdentity']['arn']
    
    # If the userIdentity ARN is related to Terraform, ignore the event
    if 'terraform-iam' in user_identity:
        return {
            'statusCode': 200,
            'body': 'Terraform action, no notification sent.'
        }
    
    # If the change is manual, publish an SNS notification
    sns.publish(
        TopicArn='<SNS_ARN>',
        Message=f"Manual change detected in S3 by {user_identity}"
    )
    
    return {
        'statusCode': 200,
        'body': 'SNS notification sent for manual change.'
    }

1. Save the Function.
2. Permissions:

• Ensure that the Lambda function’s IAM role has the necessary permissions to publish to SNS and read from CloudTrail.

4. Create SNS Topic to Receive Notifications

SNS will send you notifications when Lambda detects manual changes in S3.

Step-by-Step:

1. Open SNS Console:

• Go to the AWS SNS console.

2. Create Topic:

• Click on Create topic.
• Type: Choose Standard.
• Name: Name your topic (e.g., sns_s3_notify).

3. Create Subscription:

• Once the topic is created, click on Create subscription.
• Protocol: Choose Email.
• Endpoint: Provide your email address to receive notifications.

4. Confirm Subscription:

• Check your email and confirm the subscription.

Now apart from terraform changes, if there is any manual changes in S3 bucket, it will be notified.

Happy Cloud Computing !!!