Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration
- Setup EC2 instance Steps to Set Up EC2 Instance in a Private Subnet To set up an EC2 instance in a VPC with two private subnets (as AWS Lambda requires at least two private subnets) and one public subnet (for NAT Gateway)
Step 1: Create a VPC
Navigate to VPC Dashboard: Open the VPC Console from the AWS Management Console.
Create a New VPC: Click on Create VPC. Provide a name for the VPC (e.g., log-automation-vpc). Select an IPv4 CIDR block (e.g., 10.0.0.0/16 ). Click Create.
Step 2: Create Subnets
Create Private Subnets (for EC2 instances): Go to the Subnets section in the VPC console. Click Create Subnet. Select the VPC you created. Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 1Create two private subnets in a Availability Zone ( for high availability use different AZs): Subnet 1 (Private): 10.0.16.0/20 Subnet 2 (Private): 10.0.128.0/20
Create Public Subnet (for NAT Gateway): Create another subnet for the public-facing resources (NAT Gateway). Subnet 3 (Public): 10.0.0.0/20 (this can be in a third AZ or one of the first two). Step 3: Set Up NAT Gateway
Create the NAT Gateway: Go to NAT Gateways in the VPC dashboard. Click Create NAT Gateway. Select the Public Subnet (Subnet 3) Click Create.
Step 4: Configure Route Tables
Create a Route Table for Public Subnet: In the Route Tables section of the VPC console, click Create route table. Provide a name (e.g., public-route-table ). Select your VPC and click Create. Associate this route table with the Public Subnet (Subnet 3).
Create a Route Table for Private Subnets: Create another route table for the private subnets (e.g., private-route- table ). In the Route Tables section, click Create route table and select your VPC. Edit the route table to route outbound traffic from the private subnets to the NAT Gateway: Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 2Destination: 0.0.0.0/0 (for all outbound traffic). Target: Select the NAT Gateway. Associate this route table with the Private Subnets (Subnets 1 and 2). Diagram of VPC Setup Create IAM Role for EC2 instance 1: Create IAM Role
Go to IAM Console: IAM Console.
Create Role: Select AWS service > EC2 (for trusted entity). Attach the policies: AmazonEC2RoleforSSM (for Systems Manager access). AmazonS3ReadOnlyAccess (for read-only access to S3). Name the role (e.g., EC2-SSM-S3-Role ) and create it. 2: Attach Role to EC2 Instance
Go to EC2 Console: EC2 Console. Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration
Modify Instance Role: Select your EC2 instance. Choose Actions > Security > Modify IAM Role. Select the role you created ( EC2-SSM-S3-Role ) and save. 3: Use this Role for EC2 instance while setup Step 5: Launch EC2 Instances in Private Subnets
Navigate to EC2 Dashboard: Open the EC2 Console. Click on Launch Instance
Select an AMI and Instance Type: Choose an Amazon Machine Image (AMI) (Ubuntu). Select the instance type (e.g., t3.micro ).
Configure Instance Settings: In Network Settings, select your VPC. Select one of the Private Subnets for the Subnet (e.g., Subnet 1 or Subnet 2 ). Set Auto-assign Public IP to Disable (since it's in a private subnet). Choose an existing security group or create a new one with appropriate inbound/outbound rules (e.g., SSH access from your IP). Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration
Configure Storage and Tags as needed.
Launch the Instance. Step 6: Security Groups and Key Pair
Security Groups: Ensure the security group associated with the EC2 instance allows inbound SSH (port 22) Create inbound rule for port 443 (allow ssm)
Key Pair: If not already done, create or select an existing SSH key pair to access the EC2 instance. Step 7: Test the Setup
Verify EC2: Go to System Manager > Fleet Manager In the dashboard connection of ssm agent should be listed with ec2 instance ID.
IAM Role for Lambda Create a Lambda function that will access EC2, S3, and CloudWatch, you need to create an IAM role for Lambda with the following permissions:
Go to the IAM Console.
Create a new IAM Role and choose Lambda as the trusted entity.
Attach the following policies: a. AWSLambdaBasicExecutionRole (for Lambda to log to CloudWatch). b. AmazonEC2ReadOnlyAccess (for Lambda to interact with Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 5EC2). c. AmazonS3FullAccess (for Lambda to interact with S3).
Save the role, and ensure it’s assigned to the Lambda
Create the Lambda Function Navigate to AWS Lambda in the AWS Console. Click on Create Function, then select Author from Scratch.
Provide a name for the Lambda function (e.g., LogRetrievalLambda).
Choose Python 3.8 or later as the runtime.
Under Permissions, choose the IAM Role that was created earlier for the Lambda function.
Save Instance ID in the environment variable
In the Lambda function editor, add the following code to interact with EC2, retrieve logs, and upload them to S3: import boto3 import time import json import os
Initialize AWS SDK clients
ssm_client = boto3.client('ssm') s3_client = boto3.client('s3')
S3 Bucket details
S3_BUCKET_NAME = 'task-logs-bucket' S3_LOG_PREFIX = 'logs/' def lambda_handler(event, context):
EC2 instance ID
instance_id = os.environ.get('instanceID') Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 6# Command to fetch logs log_file_path = '/var/log/syslog'
Command to fetch log file from EC2 instance
command = f"cat {log_file_path}" try:
Send command to EC2 via SSM to get the log file
response = ssm_client.send_command( InstanceIds=[instance_id], DocumentName="AWS-RunShellScript", Parameters={'commands': [command]} )
Get the Command ID from the response
command_id = response['Command']['CommandId']
Wait for the command to complete and retrieve the outp
time.sleep(60) # Wait for command to execute (adjust if
Fetch the command invocation result
result = ssm_client.get_command_invocation( CommandId=command_id, InstanceId=instance_id )
Get the output of the command (log data)
log_data = result['StandardOutputContent']
Create a unique filename for the log file in S3
timestamp = time.strftime("%Y%m%d-%H%M%S") log_file_name = f"{S3_LOG_PREFIX}my_app_log_{timestamp}
Upload the log to S3
s3_client.put_object( Bucket=S3_BUCKET_NAME, Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 7Key=log_file_name, Body=log_data ) print(f"Log file uploaded to S3: {log_file_name}") except Exception as e: print(f"Error: {str(e)}") raise e 4. Test and Monitor • To test the Lambda function, click on Test in the Lambda console and create a test event (you can use a basic test event if not specific). • Execute the Lambda function and check the CloudWatch Logs for the output. Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 8Automating Log Retrieval from EC2 to S3 using Lambda and CloudWatch integration 9