Polyglot Microservices on Amazon EKS with Karpenter Autoscaling
Running 80+ microservices on Amazon Elastic Kubernetes Service (EKS) with Karpenter just-in-time node provisioning and AWS Load Balancer Controller.
1. Business Problem & Context
A global retail conglomerate runs 80 distinct microservices written in Go, Java, Python, and Node.js. Their legacy Kubernetes setup used standard Cluster Autoscaler with fixed Node Groups. When traffic surged, provisioning new EC2 nodes took over 6 minutes, causing pending pods to stall and requests to drop.
2. Requirements & Constraints
- Sub-60s Autoscaling: Scale nodes from 20 to 150 instances dynamically in under 1 minute.
- Right-Sizing & Mixed Fleets: Intelligently select the cheapest available EC2 instance family (Graviton, x86, Spot, On-Demand) matching exact pod CPU/memory requirements.
- Granular Pod Security: Zero shared node-level AWS IAM credentials (enforce IRSA).
3. Architecture Overview & Data Flow
Interactive Architecture Diagram (Use controls to zoom & pan)
4. AWS Services Used & Rationales
AWS Services Architecture Rationale
Concrete reasons why these specific services were chosen over alternatives
| Service | Category | Architectural Rationale ("Why this service?") |
|---|---|---|
| Amazon EKS | Containers | Provides standard Kubernetes API compatibility, multi-AZ control planes, and automated master node upgrades. |
| Karpenter | Compute | Evaluates pending pod requirements directly and calls EC2 Fleet APIs without the latency of Auto Scaling Groups. |
| IAM Roles for Service Accounts (IRSA) | Security | Uses OIDC identity providers to issue temporary STS credentials scoped strictly to specific Kubernetes namespaces. |
5. Key Design Trade-offs
Architecture Decision & Trade-Off Matrix
Evaluating alternative approaches under real-world constraints
Kubernetes Cluster Autoscaler + Fixed Managed Node Groups
- + Standard upstream Kubernetes component
- − Slow scale-up (5-8 mins)
- − Inflexible rigid instance sizes
- − Wasted capacity on underutilized nodes
Amazon EKS + Karpenter (Chosen)
✓ Chosen Design- + Sub-45s node boot time
- + Dynamic right-sizing of instances
- + Automatic consolidation & graceful spot termination handling
- − Requires Karpenter NodePool CRD management
6. Implementation Highlights
Kubernetes Manifest Karpenter NodePool CRD Configuration
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
requirements:
- key: "karpenter.sh/capacity-type"
operator: In
values: ["spot", "on-demand"]
- key: "kubernetes.io/arch"
operator: In
values: ["arm64", "amd64"]
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["c", "m", "r"]
nodeClassRef:
name: default
disruption:
consolidationPolicy: WhenUnderutilized
expireAfter: 720h7. Results & Key Metrics
- Scale-Up Speed: Dropped from 6.5 minutes down to 42 seconds.
- Cost Optimization: 38% cluster compute bill reduction through automated Spot instance adoption and bin-packing consolidation.
8. Key Architectural Takeaways
Kubernetes Autoscaling Law: For large-scale EKS clusters, replace legacy Auto Scaling Groups with Karpenter to eliminate node group sprawl and achieve sub-minute just-in-time provisioning.
