Real-Time Serverless Image Processing & Media Pipeline
Asynchronously transcode, watermark, and generate responsive image thumbnails using S3 Event Notifications, AWS Lambda, and Amazon EventBridge.
1. Business Problem & Context
An e-commerce marketplace receives over 200,000 product image uploads each day from sellers. The legacy system processed uploads synchronously on web servers, which caused HTTP request timeouts (504 Gateway Timeout) whenever sellers uploaded 20MB RAW or PNG files, leading to abandoned listings.
2. Requirements & Constraints
- Asynchronous Decoupling: User upload completes in <500ms; background processing runs asynchronously.
- Generate Multiple Tiers: WebP thumbnails (150x150), medium cards (600x600), and zoom views (1600x1600).
- Zero Server Management: Scale from 0 at night to 5,000 concurrent transformations during flash sales.
3. Architecture Overview & Data Flow
Interactive Architecture Diagram (Use controls to zoom & pan)
- Direct Pre-Signed URL Upload: The client requests a temporary S3 pre-signed PUT URL from the backend API, bypassing backend application servers entirely for upload bandwidth.
- EventBridge Notification: S3 emits an
ObjectCreatedevent directly to Amazon EventBridge. - SQS Buffering: EventBridge pushes the event into an SQS queue to protect downstream database writes and rate-limit third-party APIs.
- Lambda Transformation: Lambda functions read SQS batches, download the source image, generate WebP variants using libvips/Sharp, and save them to the public CDN bucket.
4. AWS Services Used & Rationales
AWS Services Architecture Rationale
Concrete reasons why these specific services were chosen over alternatives
| Service | Category | Architectural Rationale ("Why this service?") |
|---|---|---|
| AWS Lambda | Compute | Executes media resizing in parallel micro-containers, paying only for the 650ms execution duration. |
| Amazon S3 Pre-Signed URLs | Storage | Enables direct browser-to-S3 uploads, keeping massive image payloads off application compute instances. |
| Amazon EventBridge | Serverless | Provides declarative event routing rules filtering by S3 file extensions and bucket prefixes. |
| Amazon SQS | Serverless | Acts as a buffer with a Dead-Letter Queue (DLQ) to capture corrupt or unparseable image files. |
5. Key Design Trade-offs
Architecture Decision & Trade-Off Matrix
Evaluating alternative approaches under real-world constraints
Synchronous Node.js Web Server Resizing
- + Immediate response in same HTTP call
- − Ties up HTTP threads
- − Crashes under large files
- − High CPU billing 24/7
Asynchronous S3 + SQS + Lambda (Chosen)
✓ Chosen Design- + Zero idle cost
- + Instant client response
- + Automatic horizontal scale to 10k concurrency
- + Corrupt files isolated in DLQ
- − Client must poll or use WebSockets for completion notification
6. Implementation Highlights
Performance Tuning Optimizing Lambda Memory for CPU-Intensive Sharp Resizing
AWS Lambda allocates vCPU proportionally to configured memory. While image resizing requires ~150MB of RAM, configuring Lambda with 1769MB (1 full dedicated vCPU) drops execution time from 2,400ms down to 650ms.
Because Lambda charges Gigabyte-Seconds, running a faster 1769MB function actually costs less money than running a slow 512MB function.
7. Results & Key Metrics
- Client Upload Time: Reduced from 8.2s to 450ms via direct S3 pre-signed URLs.
- Cost: 200,000 daily transformations cost just $1.15 per day on AWS Lambda.
8. Key Architectural Takeaways
Media Ingestion Rule: Never route multi-megabyte file uploads through your API servers. Always issue S3 Pre-Signed URLs and process events asynchronously.
