AWS with Terraform (Day 18)

Day 18: Image Processing Serverless Project using AWS Lambda and Terraform

As part of my ongoing hands-on AWS Terraform learning, I completed a serverless image processing project that demonstrates how to build an event-driven, scalable workflow using AWS Lambda. The entire infrastructure is provisioned using Terraform, keeping the setup reproducible, version-controlled, and production-aligned.

The goal of this project was simple but practical: upload a single image and automatically generate multiple optimized variants without managing any servers.

Project Overview

This project implements an automated image processing pipeline using AWS serverless services.

When an image is uploaded to a source S3 bucket, an S3 event triggers a Lambda function. The Lambda function processes the image using the Pillow library and stores multiple optimized versions in a destination S3 bucket. Logging and execution metrics are captured in CloudWatch.

The entire workflow is deployed using Terraform.

Architecture Summary

The solution consists of the following components:

A source S3 bucket where original images are uploaded
A destination S3 bucket where processed images are stored
An AWS Lambda function written in Python for image processing
A Lambda layer containing the Pillow dependency
An S3 event notification to invoke Lambda on object creation
An IAM role and policy following least-privilege access
CloudWatch logs for monitoring and debugging

Each uploaded image is converted into five variants:

JPEG with 85 percent quality
JPEG with 60 percent quality
WebP format
PNG format
A 200x200 thumbnail

Terraform Implementation Highlights

All resources are provisioned using Terraform with a focus on security and maintainability.

S3 buckets are created with unique names, versioning enabled, server-side encryption, and public access fully blocked.

The Lambda execution role grants only the required permissions:
read access to the source bucket
write access to the destination bucket
CloudWatch logging permissions

The Pillow library is packaged as a Lambda layer to keep the function lightweight and reusable. The Lambda function configuration includes runtime, handler, timeout, memory settings, and environment variables such as the destination bucket name.

An S3 bucket notification is configured to trigger the Lambda function on object creation events, along with an explicit Lambda permission allowing invocation from S3.

Dependency Packaging and Reliability

One important lesson from this project is avoiding dependency mismatches.

Native libraries like Pillow must be compiled for the same environment that Lambda runs on. To prevent runtime import errors, the Lambda layer is built inside a Docker container that matches AWS Lambda’s Amazon Linux runtime. This ensures the layer works reliably in production rather than only on the local machine.

Testing and Verification

After deployment, testing was straightforward.

Uploading an image to the source S3 bucket immediately triggered the Lambda function. CloudWatch logs confirmed successful execution, and the destination bucket contained all five processed image variants.

This validated the end-to-end flow from event trigger to processing and storage.

Key Learnings

Built a fully serverless, event-driven image processing pipeline
Used Terraform to provision and wire AWS services cleanly
Packaged native Python dependencies using Lambda layers
Avoided runtime issues by building layers with Docker
Applied least-privilege IAM policies for security
Understood Lambda execution behavior, cold starts, and logging
Designed a scalable solution without managing servers

Diagram




Final Thoughts

This project reinforced how powerful serverless architectures can be when combined with Infrastructure as Code. With Terraform and AWS Lambda, it is possible to build scalable, cost-efficient systems that respond automatically to events while remaining easy to manage and extend.

Next improvements could include exposing processed images through API Gateway, integrating CloudFront for CDN delivery, or adding metadata extraction and tagging.

Day 18 focused on building something practical, production-oriented, and fully automated using AWS and Terraform.

Here is the session link: 


Comments

Post a Comment

Popular posts from this blog

AWS with Terraform (Day 01)

Image
  Terraform Fundamentals – Day 01 (My Learning Notes as a DevOps Engineer) Today I wrapped up Day-01 of my Terraform with AWS series, and even with my DevOps background, this session helped me tighten my fundamentals and rethink how I approach cloud provisioning. Terraform is something I’ve used before, but today was more about understanding why IaC matters at scale and why Terraform continues to dominate in real-world DevOps ecosystems. Understanding Infrastructure as Code (IaC) As a DevOps engineer, IaC is not new to me. But today’s session broke it down in a way that clarified its real value. IaC = Infrastructure represented as code. Instead of building resources manually through AWS/Azure/GCP consoles, we define everything in .tf files. This ensures our setup is predictable, repeatable, and version-controlled. What I liked most is how clearly the contrast was shown: Cloud-Agnostic Tools (Terraform / Pulumi) Work across AWS, Azure, GCP, Kubernetes, GitHub, etc. ...
Read more

AWS with Terraform (Day 27)

Image
Automating AWS Infrastructure Using Terraform and GitHub Actions Infrastructure automation is where DevOps truly becomes real. Writing Terraform code is only half the story—the real value comes when infrastructure changes are version-controlled, reviewed, scanned, approved, and applied automatically . On Day 27 of my DevOps journey, I implemented a production-grade CI/CD pipeline to automate AWS infrastructure using Terraform and GitHub Actions , following best practices used in real-world teams. This blog walks through the architecture, workflow design, safety controls, and lessons learned . Why Automate Terraform with CI/CD? Running Terraform manually from a laptop works for learning, but it quickly breaks down in team and production environments: Local state files are risky and hard to share Credentials scattered across machines No standardized reviews or approvals No security or policy checks before apply No clear audit trail By integrating Terraform with G...
Read more

AWS with Terraform (Day 02)

Image
  AWS with Terraform – Day 02 Deep Dive into Providers, Versioning & the Real “Bridge” Between Code and Cloud Today’s session took me from simply writing Terraform code to actually understanding the engine behind it — the Provider layer . And honestly, once this clicks, Terraform stops feeling like just a tool and starts feeling like a real engineering system. Here’s what I learned and how I’ve started applying it. Providers: The Plugin That Makes Terraform Real The simplest way I now understand it: Terraform writes the story. Providers translate it into cloud actions. Before today, I treated the provider block as just two lines of mandatory code. Now I see it as the translator that turns HCL into actual AWS API calls . Example: I write: resource "aws_s3_bucket" "demo" { bucket = "my-app-demo" } The AWS provider turns that into the exact S3 API request. This layer saves us from manually hitting APIs, building payloads, or juggling ...
Read more