Resilient Webhook Ingestion Engine: Buffering 10,000 Webhooks/Sec
Ingest third-party webhooks (Stripe, GitHub, Shopify) directly into Amazon SQS via API Gateway without intermediate compute to prevent traffic dropouts.
1. Business Problem & Context
A payments platform integrates with external gateways (Stripe, Adyen). During major shopping events (Black Friday), payment gateways emit massive webhook bursts. The previous system placed a Lambda function directly behind API Gateway to write into a relational database; during spikes, Lambda connection exhaustion caused HTTP 500 errors, leading payment providers to disable webhook endpoints.
2. Requirements & Constraints
- Zero Ingestion Failures: Never drop a webhook under any traffic spike.
- Sub-50ms Acknowledgment: Return
HTTP 200 OKimmediately to the caller. - Strict Idempotency: Payment events must only be processed exactly once.
3. Architecture Overview & Data Flow
Interactive Architecture Diagram (Use controls to zoom & pan)
4. AWS Services Used & Rationales
AWS Services Architecture Rationale
Concrete reasons why these specific services were chosen over alternatives
| Service | Category | Architectural Rationale ("Why this service?") |
|---|---|---|
| Amazon API Gateway Service Proxy | Serverless | Pushes payloads directly to Amazon SQS using IAM role credentials, eliminating cold starts and intermediate Lambda execution costs. |
| Amazon SQS | Serverless | Buffers up to billions of in-flight messages with configurable retention (up to 14 days). |
| Amazon DynamoDB | Database | Uses conditional put-item (attribute_not_exists) to guarantee idempotent processing. |
5. Key Design Trade-offs
Architecture Decision & Trade-Off Matrix
Evaluating alternative approaches under real-world constraints
API Gateway -> Lambda -> Database
- + Simple synchronous debugging
- − High concurrency exhausts DB connections
- − Double billing (APIGW + Ingress Lambda + Worker Lambda)
API Gateway Direct SQS Proxy -> Batch Lambda (Chosen)
✓ Chosen Design- + Zero dropped webhooks
- + Fastest response time (22ms)
- + 45% lower compute costs via batch processing
- − Requires API Gateway VTL mapping template setup
6. Implementation Highlights
IaC Recipe API Gateway SQS Integration Request Template
Action=SendMessage&MessageBody=$util.urlEncode($input.body)&MessageAttribute.1.Name=webhook_source&MessageAttribute.1.Value.DataType=String&MessageAttribute.1.Value.StringValue=$input.params('x-webhook-provider')7. Results & Key Metrics
- Zero Ingestion Loss during 12,000 webhooks/sec Black Friday surge.
- P99 Ingress Latency dropped from 420ms to 22ms.
8. Key Architectural Takeaways
Serverless Design Pattern: If your Lambda function only exists to read a JSON payload and push it into an SQS queue or DynamoDB table, replace the Lambda with an API Gateway direct service integration.
