Image Pixelator: Event-Driven Real-Time S3-to-Lambda Image Processing Workflow using Terraform

In this blog, we’ll explore how to build an event-driven image processing pipeline using AWS Lambda, S3, and Terraform. When an image is uploaded to an S3 bucket, a Lambda function is triggered to pixelate the image in real-time.

We’ll use Terraform to automate the entire infrastructure setup, making the process efficient, scalable, and repeatable.

Table of Contents-
· Project Overview
· Stage 1: Setting Up the S3 Buckets
· Stage 2: Set Up the IAM Role for Lambda
· Stage 3 (pre/optional): Prepare the Lambda Function Deployment Package
· Stage 3 — Create the Lambda Function
· Stage 4 — Configure the Lambda Function & Trigger
· Terraform Infrastructure Deployment
· Stage 5— Test and Monitor
· Stage 6— Cleanup
· Conclusion

Prerequisites

Before we get started, ensure you have:

• An AWS account
• AWS CLI configured on your local machine
• Terraform installed (if not, follow Terraform installation)
• Basic knowledge of AWS services like Lambda, S3, and IAM
• s3 bucket for state management (versioning and encryption enabled)
• DynamoDB Table for state lock

Project Overview

This project automates the real-time pixelation of images using AWS Lambda. When an image is uploaded to the source S3 bucket, it triggers a Lambda function that processes the image in five levels of pixelation (8x8, 16x16, 32x32, 48x48, and 64x64). The processed images are then saved in a separate, designated processed S3 bucket.

For this project, I’ve included both manual steps and Terraform code for each stage. You can choose either approach, but I recommend trying both for a better understanding of the process.

Terraform will automate the deployment of the necessary AWS infrastructure, including S3 buckets, Lambda functions, IAM roles, and permissions, ensuring a smooth and streamlined setup.

• Github Repository Link

Stage 1: Setting Up the S3 Buckets

To store the original and processed images, we’ll create two S3 buckets: one for the source images and one for the processed images.

1. Source Bucket: This bucket will hold the original images that trigger the processing pipeline.
2. Processed Bucket: This bucket will store the pixelated versions of the images.

Create the S3 Buckets using the S3 Console

- Bucket 1 (Source): <unique-name>-source
- Bucket 2 (Processed): <unique-name>-processed

Example:
- Source Bucket: pixelator-project-source
- Processed Bucket: pixelator-project-processed

1. Navigate to S3 Console and click on Create Bucket -

2. Enter the unique name for the bucket -

3. Now, Click on Create bucket —

• Repeat the same steps to create the processed bucket. You should now have both the buckets.

Terraform code to set-up these buckets —

Step 1 — Configure the AWS Provider & Remote State Backend

Follow Step-1 in this blog to set-up the s3 backend for State Management and DynamoDB for State Locking.

Create a file named backend.tf and paste below code in that file.

Note — All the variables are mentioned at the end in variables.tf and dev.tfvars file

Step 2: Create the AWS S3 Bucket

Create a file named main.tf and paste the below code in that file -

Stage 2: Set Up the IAM Role for Lambda

Next, we need to create an IAM role that will allow our Lambda function to access the S3 buckets and write logs to CloudWatch.

1. Navigate to IAM Console.

• First click on Roles and then click on Create Role.

2. Create a New Role:
- Select Lambda as the trusted entity. Click on Next.

• Skip the Permissions screen and Click on Next.
• Name the role PixelatorRole and click on Create Role.

3. Attach Inline Policy:

• Navigate to the newly created Role and click on Create Inline Policy

• The policy should grant the Lambda function the necessary permissions to interact with both the source and processed buckets. Below is the required JSON policy.

{
 "Version": "2012–10–17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": "s3:*",
 "Resource": [
 "arn:aws:s3:::<source-bucket-name>",
 "arn:aws:s3:::<source-bucket-name>/*",
 "arn:aws:s3:::<processed-bucket-name>",
 "arn:aws:s3:::<processed-bucket-name>/*"
 ]
 },
 {
 "Effect": "Allow",
 "Action": "logs:CreateLogGroup",
 "Resource": "arn:aws:logs:us-east-1:<aws-account-id>:*"
 },
 {
 "Effect": "Allow",
 "Action": [
 "logs:CreateLogStream",
 "logs:PutLogEvents"
 ],
 "Resource": [
 "arn:aws:logs:us-east-1:<aws-account-id>:log-group:/aws/lambda/pixelator:*"
 ]
 }
 ]
}

Replace the placeholders (<source-bucket-name>, <processed-bucket-name>, and <aws-account-id>) with actual values.

• Select the JSON editor in the right side menu and paste the above policy. Click on Next.

• Give the policy name — pixelator-inline-policy and Click on Create policy.

Terraform Code to create the role and inline policy —

First, we need to create a data block to fetch the AWS account ID.

Now, we’ll create the role.

This code will create the inline policy.

Stage 3 (pre/optional): Prepare the Lambda Function Deployment Package

If you’re using macOS or Linux, follow these steps to create a deployment package for your Lambda function. (Windows users can skip this section and use the pre-built zip file).

1. Create a new folder and navigate into it:

mkdir my_lambda_deployment
cd my_lambda_deployment

2. Create the Lambda code:

• Inside the folder, create another folder called lambda:

mkdir lambda
cd lambda

• Inside the lambda folder, create a file called lambda_function.py and paste in the following code:

import os
import json
import uuid
import boto3

from PIL import Image

# Get bucket name for processed images
processed_bucket = os.environ['processed_bucket']

s3_client = boto3.client('s3')

def lambda_handler(event, context):
 print(event)

# Get bucket and object key from event
 source_bucket = event['Records'][0]['s3']['bucket']['name']
 key = event['Records'][0]['s3']['object']['key']

# Generate a temp name and location for the image
 object_key = str(uuid.uuid4()) + '-' + key
 img_download_path = '/tmp/{}'.format(object_key)

# Download the image from the source bucket
 with open(img_download_path, 'wb') as img_file:
 s3_client.download_fileobj(source_bucket, key, img_file)

# Pixelate the image in different variations
 pixelate((8, 8), img_download_path, '/tmp/pixelated-8x8-{}'.format(object_key))
 pixelate((16, 16), img_download_path, '/tmp/pixelated-16x16-{}'.format(object_key))
 pixelate((32, 32), img_download_path, '/tmp/pixelated-32x32-{}'.format(object_key))
 pixelate((48, 48), img_download_path, '/tmp/pixelated-48x48-{}'.format(object_key))
 pixelate((64, 64), img_download_path, '/tmp/pixelated-64x64-{}'.format(object_key))

# Upload pixelated images to processed bucket
 s3_client.upload_file('/tmp/pixelated-8x8-{}'.format(object_key), processed_bucket, 'pixelated-8x8-{}'.format(object_key))
 s3_client.upload_file('/tmp/pixelated-16x16-{}'.format(object_key), processed_bucket, 'pixelated-16x16-{}'.format(object_key))
 s3_client.upload_file('/tmp/pixelated-32x32-{}'.format(object_key), processed_bucket, 'pixelated-32x32-{}'.format(object_key))
 s3_client.upload_file('/tmp/pixelated-48x48-{}'.format(object_key), processed_bucket, 'pixelated-48x48-{}'.format(object_key))
 s3_client.upload_file('/tmp/pixelated-64x64-{}'.format(object_key), processed_bucket, 'pixelated-64x64-{}'.format(object_key))

def pixelate(pixelsize, image_path, pixelated_img_path):
 img = Image.open(image_path)
 temp_img = img.resize(pixelsize, Image.BILINEAR)
 new_img = temp_img.resize(img.size, Image.NEAREST)
 new_img.save(pixelated_img_path)

3. Package the Lambda function:

• Install the Pillow library (required for image processing) and package the function:

pip install Pillow -t .
zip -r ../lambda-deployment-package.zip .

4. Upload the Lambda package to the Lambda console (skip if using pre-created zip).

Stage 3 — Create the Lambda Function

We’ll now create the Lambda function, which will be triggered by S3 uploads to perform real-time image pixelation

1. Access the Lambda Console:

• Open the Lambda Console.

2. Create the Lambda Function:

• Click the Create Function button.

• Author from Scratch (instead of choosing a blueprint).
• For the Function name, enter pixelator.
• For Runtime, select Python 3.9.
• For Architecture, select x86_64.

3. Set Lambda Permissions:

• For Permissions, expand the Change default execution role dropdown.
• Choose Use an existing role.
• In the Existing role dropdown, pick PixelatorRole (the role you created earlier).
• Click Create Function.

4. Upload the Lambda Code:

• Close any notification dialogues or pop-ups that appear.
• In the function page, click Upload from and select .zip file.

• Option 1: Download the pre-created Lambda ZIP file to your local machine from this link and click Download.
• Option 2: If you’ve created your own zip file in Stage 3(pre), locate and upload that .zip file.

• After uploading, click Save.

Note: The upload will take a few minutes. Once complete, you might see a notification saying:
“The deployment package of your Lambda function ‘pixelator’ is too large to enable inline code editing. However, you can still invoke your function.”
This is completely normal and not an issue.

Stage 4 — Configure the Lambda Function & Trigger

1. Set Environment Variables:

• Go to the Configuration tab in the Lambda function.
• Select Environment variables from the left-hand menu.
• Click Edit, then Add environment variable.

• Set the Key as processed_bucket.
• Set the Value as the bucket name for your processed images. For example, if your processed bucket is named pixelator-project-processed, enter that here.
• Important: Ensure you are using the processed bucket, not the source bucket. If you use the source bucket here, the Lambda function will continuously trigger itself.
• Click Save.

2. Adjust Timeout Settings:

• In the General configuration section, click Edit.

• Set the Timeout to 1 minute and 0 seconds.
• Click Save.

3. Add S3 Trigger to Lambda:

• In the same Configuration tab, click Add trigger.

• In the dropdown, select S3.
• Under Bucket, select your source bucket (this is where images will be uploaded).
• Ensure you are picking the correct bucket and not the processed bucket or any other bucket.
• Check the Recursive invocation acknowledgment box. This means the Lambda function will be triggered every time a new object is uploaded to the source bucket.
• Once checked, click Add.

Terraform Code for Lambda function and it’s configurations —

• Terraform code to set up permissions for S3 triggers -

• Terraform code to configure an S3 trigger -

• Terraform variables files —

Terraform Infrastructure Deployment

To get started with the Terraform Infrastructure Deployment, you can either compile all the code in a single main.tf file or break it down into multiple files for better modularity.

Alternatively, you can find the complete code here.

Steps to deploy:

1. Clone the repository in your local machine.
2. Navigate to the terraform-files directory.
3. Set-up local machine to interact with your AWS account. Follow these instructions [Section — Getting Started] in this blog.

Once there, follow the standard Terraform commands to initialize, plan, and apply the deployment.

• First, we need to initialize our terraform providers by running -

terraform init

• Now, Let’s format and validate our code.

terraform fmt
terraform validate

• Let’s check what resources we’re going to add in AWS by running —

terraform plan -var-file dev.tfvars

• Finally, the moment has arrived to apply our changes and witness the magic of Terraform in action.

terraform apply -var-file dev.tfvars --auto-approve

• Upon successfully executing this command, we’ll get the following output.

Stage 5— Test and Monitor

We’ll now test the Lambda function by uploading images to the source S3 bucket and verify that the pixelation process works as expected by checking the CloudWatch logs for any errors or issues during the execution.

1. Open Multiple Browser Tabs:

• Open one tab for the CloudWatch Logs console.
• Open two tabs for the S3 Console.

2. Upload Test Images:

• In one S3 tab, navigate to your source bucket.
• Go to the Objects tab and click Upload.
• Upload some image files. You can use your own images or sample images that you might have.

• Once the upload is complete, the images will trigger the Lambda function.

3. View Lambda Logs:

• Switch to the CloudWatch Logs tab.
• Click the Refresh icon to locate the log stream named /aws/lambda/pixelator.

• Click the most recent log stream. If it doesn’t appear immediately, keep refreshing until it does.

• Expand the log stream starting with {'Records': [{'eventVersion':... to see details about the Lambda invocation, including the uploaded object name in 'object': {'key'}.

4. View Processed Images in the Processed Bucket:

• Switch to the second S3 Console tab, which should be open to your processed bucket.
• Click the Refresh icon to update the view.
• You should see the processed pixelated images (8x8, 16x16, 32x32, 48x48, and 64x64).

• Open each image to verify that the pixelation is applied correctly. You should see the original image progressively pixelated in the five different sizes.

Stage 6— Cleanup

To avoid incurring any unwanted charges on AWS, please ensure to delete all the resources created during this project.

1. Delete the Lambda Function:

• Open the Pixelator Lambda function in the Lambda console.
• Click Delete to remove the function.

2. Delete IAM Role:

• Go to the IAM Roles console.
• Click on the PixelatorRole.
• Click Delete and confirm the deletion of the role.

3. Empty and Delete the S3 Buckets:

• Go to the S3 Console.
• For each bucket (source and processed):

4. Select the bucket.

5. Click Empty.

6. Type permanently delete and click Empty again to confirm.

7. Once emptied, close the dialog and return to the S3 console.

8. Ensure the bucket is selected and click Delete.

9. Type the name of the bucket to confirm the deletion.

Terraform Clean-up —

Run below command to delete all the resources created by Terraform -

terraform destroy -var-file dev.tfvars -auto-approve

Completion:

• You’ve successfully cleaned up the resources created in this guide.

Conclusion

In this tutorial, we have walked through setting up an event-driven image processing pipeline using AWS Lambda and S3, with Terraform for provisioning. We have configured the pipeline to pixelate images uploaded to an S3 bucket, storing the processed images in another bucket.

Finally, we tested the pipeline, monitored the logs, and ensured the images were pixelated and stored correctly. We also covered the cleanup process to remove resources and avoid unnecessary costs.

With this foundational knowledge, you can apply similar patterns to create more complex serverless workflows using AWS services and Terraform. Happy coding!

If you’ve discovered any value or gained insights from this blog, I would greatly appreciate it if you could show your appreciation by giving this story a clap.

Please feel free to reach out to me on LinkedIn with any questions, or you can drop them into the comment section below. I would be more than happy to assist you.