EKS at Scale: Karpenter Autoscaling & JVM SRE
AWS EKS · Karpenter · Prometheus · Grafana · JVM Off-Heap Profiling
The Challenge
The Approach
768MB container limit for the 512MB heap. Adjusted startup checks for warmup, introduced Karpenter scheduling, and kept payment and authentication workloads on On-Demand capacity.System Architecture
EKS Platform Architecture
EKS Platform Architecture
Multi-AZ Kubernetes with Karpenter autoscaling and full SRE observability. Click a node to view details.
Interactive Demo
Interact with the simulation below. It replicates the actual engineering scenario.
Business Impact
Technical Highlights
- 12 microservices across 3 AZs with PodAntiAffinity preventing same-node co-location of critical replicas
- Karpenter v1 NodePool provisioning replacing static node groups, mixing On-Demand and Spot instances for a reported 74% reduction in listed compute charges
- HPA scaling on CPU at 80% threshold and custom Prometheus metrics including request latency p99 and queue depth
- Pod Disruption Budgets enforcing minAvailable during rolling updates and voluntary node drains
- Liveness probes checking /health every 10s with 3-failure threshold, readiness gates on database connectivity
- Prometheus scraping 847 metric series across all services with 15s intervals and 30-day retention
- Grafana dashboards with SLI panels: request rate, error rate, latency p50/p95/p99, saturation, and pod restart count
- Alertmanager routing critical alerts to PagerDuty, warnings to Slack, with 5-minute deduplication
- Sentry integration capturing full stack traces, source maps, and release tracking per Helm deployment
- Helm charts with GitHub Actions GitOps: lint, diff, dry-run on PR, auto-deploy on merge, rollback on probe failure
- Container memory right-sizing: JVM heap at 512MB with container limit at 768MB for off-heap headroom, eliminating OOMKilled events
- Network policies restricting pod-to-pod traffic to only declared dependencies, default-deny ingress
Frequently Asked Questions
Q: What does Karpenter optimize for in this cluster?
Karpenter optimizes for both compute cost and provisioning speed. It observes unscheduled pods, determines the exact CPU and memory requirements, and launches right-sized EC2 instances directly from diversified Spot and On-Demand pools in under 82 seconds.
Q: Does using Spot instances with Karpenter hurt production reliability?
No. Mission-critical, stateful, and authentication services are pinned to On-Demand nodes via nodeSelectors and tolerations. Spot instances are used exclusively for stateless API replicas, queue consumers, and batch workloads protected by PodDisruptionBudgets and multi-AZ diversification.
Q: How fast does Karpenter provision nodes compared to Cluster Autoscaler?
Legacy AWS Cluster Autoscaler requires scaling ASGs and waiting for EC2 initialization, typically taking 4 to 5 minutes. Karpenter talks directly to the EC2 fleet API, bypassing Auto Scaling Groups to provision ready nodes in under 82 seconds.
Want results like this for your infrastructure?
I specialize in taking complex AI pipelines and cloud setups from concept to high-availability production. Let's discuss how to optimize your workloads, secure your environment, and reduce cloud costs.