AWS with Terraform (Day 23)

End-to-End Observability on AWS Using Terraform

A Real-World Serverless Project (Day 23 Completed)

As a DevOps engineer, I’ve learned that building applications is only half the job. The real challenge begins when those applications are running in production. If you can’t observe your system—logs, metrics, alerts, failures—you’re flying blind.

On Day 23 of my hands-on DevOps journey, I completed an end-to-end observability stack on AWS using Terraform for a real-world serverless image-processing application.
This project focuses on building production-grade monitoring, logging, dashboards, alarms, and notifications, all fully automated and reproducible.

This blog walks through what I built, why I built it, and how it works in practice.

Project Overview: Serverless Image Processing Pipeline

At the core of this project is a simple but realistic serverless workflow:

  1. A user uploads an image to an S3 upload bucket

  2. An AWS Lambda function is triggered

  3. The function processes the image into multiple formats:

    • WEBP

    • JPG

    • PNG

    • Thumbnail

    • Additional resized variants

  4. Processed images are stored in a separate S3 processed bucket

  5. Observability is layered on top using CloudWatch and SNS

This mirrors real-world serverless workloads where visibility, performance, and error detection are critical.

Why Build Observability the Terraform Way

I deliberately implemented observability using Terraform (Infrastructure as Code) instead of manual console configuration.

Why?

  • Reproducibility – The entire monitoring stack can be recreated in minutes

  • Auditability – Dashboards, alarms, and thresholds are version-controlled

  • No configuration drift – No hidden console changes

  • Environment parity – Same observability for dev, staging, and prod

  • Production realism – This is how mature teams manage observability

Observability should not be an afterthought—it should be part of the infrastructure itself.

Terraform Project Structure (Production-Style)

The project is organized using reusable Terraform modules, closely resembling real production repositories.

Key Modules

1. SNS Notifications

  • Three SNS topics:

    • Critical alerts

    • Performance alerts

    • Log-based alerts

  • Email subscriptions (SMS optional)

  • SNS topic policies allowing CloudWatch to publish events

2. S3 Buckets

  • Separate upload and processed buckets

  • Versioning enabled

  • Server-side encryption (AES-256)

  • Public access completely blocked

Security and durability are non-negotiable in production.

3. Lambda Function (Image Processor)

  • IAM role with least-privilege permissions:

    • CloudWatch logs

    • S3 GetObject, PutObject, and version access

    • CloudWatch custom metric publishing

  • S3 event notifications wired to Lambda

  • Explicit Lambda permissions for S3 invocation

4. Lambda Layer (Pillow + Dependencies)

  • Python dependencies packaged as a Lambda Layer

  • Built using Docker to ensure Linux runtime compatibility

  • Avoids local OS and architecture issues

5. CloudWatch Metric Filters

  • Converts log events into custom metrics

  • Enables application-level observability beyond default Lambda metrics

6. Dashboards and Alarms

  • CloudWatch dashboard with multiple widgets

  • Alarms on both standard and custom metrics

  • SNS integration for automated alerting

Building the Lambda Layer (Docker-Based)

Native Python dependencies like Pillow can break if built on the wrong OS or architecture. To avoid this, the project builds the layer inside Docker.

Flow:

  1. Docker runs a Python 3.12 Linux container

  2. Dependencies are installed inside the container

  3. Output is a pillow-layer.zip

  4. Terraform uploads the zip as a Lambda Layer

This approach ensures 100% compatibility with the Lambda runtime and avoids painful production failures.

Custom Metrics Using Log Metric Filters

Default Lambda metrics are useful—but not enough.

This project adds application-aware observability using custom metrics derived from logs.

Key Custom Metrics

  • Image processing errors

    • Triggered when log level is ERROR

  • Processing time

    • Extracted from structured logs

    • Enables average and max latency tracking

  • Successful image processing count

  • Original image size

    • Helps detect unusually large uploads

  • S3 access denied events

    • Surfaced via CloudTrail → CloudWatch Logs

Each metric filter transforms log patterns into numeric CloudWatch metrics, enabling alarms and dashboards.

CloudWatch Dashboards (Operational Visibility)

The dashboard provides a single-pane-of-glass view of the system:

Included Widgets

  • Lambda invocations, errors, throttles

  • Duration metrics (average, max, P99)

  • Concurrent executions

  • Custom image processing time

  • Custom image size trends

  • Recent error logs (CloudWatch Logs Insights)

This allows operators to understand health, performance, and failures at a glance.

Alarms, SNS Topics, and Notification Flow

Alerts are categorized by severity:

🔴 Critical Alerts

  • Repeated image processing failures

  • Unauthorized access attempts

  • Sent to critical SNS topic (email/SMS)

🟠 Performance Alerts

  • High latency (P99)

  • Excessive concurrent executions

🟡 Log-Based Alerts

  • Access denied events

  • Anomalous log patterns

Each SNS topic is explicitly permitted to receive CloudWatch alarm events, and subscriptions must be confirmed via email.

Practical Deployment Steps

  1. Install prerequisites:

    • Terraform

    • AWS CLI (configured)

    • Docker

  2. Build Lambda layer:

    scripts/build_layer_docker.sh

  3. Configure variables:

    • Region

    • Environment

    • Project name

    • Alert email addresses

  4. Initialize Terraform:

    terraform init

  5. Plan and apply:

    terraform plan
terraform apply -auto-approve

  6. Confirm SNS subscriptions via email

  7. Test by uploading images

  8. Destroy resources when done:

    terraform destroy

Testing Common Alarm Scenarios

I validated the observability stack by intentionally triggering failures:

  • Uploading valid images → success metrics only

  • Uploading non-image files → error metrics and alarms

  • Concurrent uploads → concurrency alarms

  • Oversized files → size-based alerts

  • Unauthorized access simulations → security alerts

This confirms the system fails loudly and visibly, which is exactly what production observability should do.

Troubleshooting Lessons Learned

  • Backend errors usually mean S3 permissions or naming issues

  • Docker-based layer builds are far more reliable than local builds

  • CloudWatch metrics can take a few minutes to appear

  • Metric filter patterns must exactly match log structure

  • Unconfirmed SNS subscriptions silently drop alerts

These are real-world lessons, not textbook theory.

Diagram




Conclusion

This project reinforced a critical DevOps principle:

If you can’t observe it, you can’t operate it.

By implementing observability programmatically with Terraform, monitoring becomes a first-class citizen of infrastructure—not an afterthought.

Logs, metrics, dashboards, alarms, and notifications together create a proactive safety net for serverless applications.

Here is the session link:



Comments

Post a Comment

Popular posts from this blog

AWS with Terraform (Day 01)

Image
  Terraform Fundamentals – Day 01 (My Learning Notes as a DevOps Engineer) Today I wrapped up Day-01 of my Terraform with AWS series, and even with my DevOps background, this session helped me tighten my fundamentals and rethink how I approach cloud provisioning. Terraform is something I’ve used before, but today was more about understanding why IaC matters at scale and why Terraform continues to dominate in real-world DevOps ecosystems. Understanding Infrastructure as Code (IaC) As a DevOps engineer, IaC is not new to me. But today’s session broke it down in a way that clarified its real value. IaC = Infrastructure represented as code. Instead of building resources manually through AWS/Azure/GCP consoles, we define everything in .tf files. This ensures our setup is predictable, repeatable, and version-controlled. What I liked most is how clearly the contrast was shown: Cloud-Agnostic Tools (Terraform / Pulumi) Work across AWS, Azure, GCP, Kubernetes, GitHub, etc. ...
Read more

AWS with Terraform (Day 27)

Image
Automating AWS Infrastructure Using Terraform and GitHub Actions Infrastructure automation is where DevOps truly becomes real. Writing Terraform code is only half the story—the real value comes when infrastructure changes are version-controlled, reviewed, scanned, approved, and applied automatically . On Day 27 of my DevOps journey, I implemented a production-grade CI/CD pipeline to automate AWS infrastructure using Terraform and GitHub Actions , following best practices used in real-world teams. This blog walks through the architecture, workflow design, safety controls, and lessons learned . Why Automate Terraform with CI/CD? Running Terraform manually from a laptop works for learning, but it quickly breaks down in team and production environments: Local state files are risky and hard to share Credentials scattered across machines No standardized reviews or approvals No security or policy checks before apply No clear audit trail By integrating Terraform with G...
Read more

AWS with Terraform (Day 02)

Image
  AWS with Terraform – Day 02 Deep Dive into Providers, Versioning & the Real “Bridge” Between Code and Cloud Today’s session took me from simply writing Terraform code to actually understanding the engine behind it — the Provider layer . And honestly, once this clicks, Terraform stops feeling like just a tool and starts feeling like a real engineering system. Here’s what I learned and how I’ve started applying it. Providers: The Plugin That Makes Terraform Real The simplest way I now understand it: Terraform writes the story. Providers translate it into cloud actions. Before today, I treated the provider block as just two lines of mandatory code. Now I see it as the translator that turns HCL into actual AWS API calls . Example: I write: resource "aws_s3_bucket" "demo" { bucket = "my-app-demo" } The AWS provider turns that into the exact S3 API request. This layer saves us from manually hitting APIs, building payloads, or juggling ...
Read more