Platform & SRE
Freelance client
EKS at Scale: Karpenter Autoscaling & JVM SRE
AWS EKS · Karpenter · Prometheus · Grafana · JVM Off-Heap Profiling
<82sScale-Up Latency (Down from 4.5 Minutes)
01
The Challenge
Cluster Autoscaler took 4–5 minutes to scale node groups during traffic spikes, causing pending pod queues. Java services crashed from silent off-heap OOMKilled errors inside 512MB container limits.
What Made This Genuinely Hard
- Ungoverned at every layer simultaneously: Wrong-region EC2 instances, dead NAT gateways still billing transit, single RDS instance sized for peak-year traffic, and orphaned EBS volumes + Elastic IPs.
- The OOM crashes were invisible without off-heap analysis: A
512MBJVM heap inside a512MBcontainer memory limit left zero room for thread stacks, NIO buffers, and GC overhead. Raised container limit to768MBwith heap at512MB, eliminating OOMKilled events. - Zombie pods required liveness-probe design: Profiled startup traces to set
initialDelaySeconds: 60andfailureThreshold: 3to stop killing warming pods.
02
The Approach
Profiled JVM memory to set 768MB container limit with 512MB heap (giving off-heap room). Tuned startup probes (
Here is the provisioner config I ran — annotated for the decisions that moved cost:
initialDelaySeconds: 60) to stop killing warming pods. Deployed weighted Karpenter provisioners mixing Spot for batch and On-Demand for payment/auth with ttlSecondsAfterEmpty: 30.Here is the provisioner config I ran — annotated for the decisions that moved cost:
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: node.kubernetes.io/instance-type
operator: In
values: ["m5.xlarge", "m6i.large", "c6i.xlarge"]
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a", "us-east-1b"]
weight: 100
limits:
resources:
cpu: "64"
memory: 256Gi
providerRef:
name: default
ttlSecondsAfterEmpty: 30Deep-Dive Architecture & Interactive Workbench:
Read the complete Karpenter EKS Cost Optimization Guide with Production YAML →
Read the complete Karpenter EKS Cost Optimization Guide with Production YAML →
03
System Architecture
Loading architecture diagram...
04
EKS Platform Architecture
EKS Platform Architecture
Multi-AZ Kubernetes with Karpenter autoscaling and full SRE observability. Click a node to view details.
Select a node to view infrastructure details
Primary data flow
Error tracking (async)
Interactive Demo
Interact with the simulation below. It replicates the actual engineering scenario.
Loading simulator...
06
Overview
Rebuilt cluster autoscaling with Karpenter spot-first provisioning — cut compute spend 70% while keeping 99.98% uptime. Cluster Autoscaler took 4–5 minutes to scale node groups during traffic spikes, causing pending pod queues. Java services crashed from silent off-heap OOMKilled errors inside 512MB container limits. Profiled JVM memory to set 768MB container limit with 512MB heap (giving off-heap room). Tuned startup probes (
initialDelaySeconds: 60) to stop killing warming pods. Deployed weighted Karpenter provisioners mixing Spot for batch and On-Demand for payment/auth with ttlSecondsAfterEmpty: 30. Scale-up latency dropped from 4.5m to <82s. Compute bill dropped 70% ($3.6K → $950/mo). Zero OOM outages across 6 months.07
Business Impact
Scale-up latency dropped from 4.5m to <82s. Compute bill dropped 70% ($3.6K → $950/mo). Zero OOM outages across 6 months. Platform uptime reached 99.98% over a 6-month measurement window with zero OOMKilled events.
| Metric | Before Optimization | After Optimization |
|---|---|---|
| Monthly EKS Compute | $3,600/mo (On-Demand static) | $950/mo (70% net reduction) |
| Node Provisioning Time | 4–5 minutes (Cluster Autoscaler) | < 82 seconds (Karpenter) |
| Spot Fleet Adoption | 0% (100% On-Demand) | 68% of cluster compute |
| Mean Time to Recovery | 10+ minutes (manual restarts) | < 2 minutes (automated probes) |
| Measured SLA | 99.2% (frequent OOM downtime) | 99.98% over 6 months |
08
Technical Highlights
- 12 microservices across 3 AZs with PodAntiAffinity preventing same-node co-location of critical replicas
- Karpenter v1 NodePool provisioning replacing static node groups, mixing On-Demand and Spot instances for 70% cost reduction
- HPA scaling on CPU at 80% threshold and custom Prometheus metrics including request latency p99 and queue depth
- Pod Disruption Budgets enforcing minAvailable during rolling updates and voluntary node drains
- Liveness probes checking /health every 10s with 3-failure threshold, readiness gates on database connectivity
- Prometheus scraping 847 metric series across all services with 15s intervals and 30-day retention
- Grafana dashboards with SLI panels: request rate, error rate, latency p50/p95/p99, saturation, and pod restart count
- Alertmanager routing critical alerts to PagerDuty, warnings to Slack, with 5-minute deduplication
- Sentry integration capturing full stack traces, source maps, and release tracking per Helm deployment
- Helm charts with GitHub Actions GitOps: lint, diff, dry-run on PR, auto-deploy on merge, rollback on probe failure
- Container memory right-sizing: JVM heap at 512MB with container limit at 768MB for off-heap headroom, eliminating OOMKilled events
- Network policies restricting pod-to-pod traffic to only declared dependencies, default-deny ingress
09
Frequently Asked Questions
Q: What does Karpenter optimize for in this cluster?
Karpenter optimizes for both compute cost and provisioning speed. It observes unscheduled pods, determines the exact CPU and memory requirements, and launches right-sized EC2 instances directly from diversified Spot and On-Demand pools in under 82 seconds.
Q: Does using Spot instances with Karpenter hurt production reliability?
No. Mission-critical, stateful, and authentication services are pinned to On-Demand nodes via nodeSelectors and tolerations. Spot instances are used exclusively for stateless API replicas, queue consumers, and batch workloads protected by PodDisruptionBudgets and multi-AZ diversification.
Q: How fast does Karpenter provision nodes compared to Cluster Autoscaler?
Legacy AWS Cluster Autoscaler requires scaling ASGs and waiting for EC2 initialization, typically taking 4 to 5 minutes. Karpenter talks directly to the EC2 fleet API, bypassing Auto Scaling Groups to provision ready nodes in under 82 seconds.
Want results like this for your infrastructure?
I specialize in taking complex AI pipelines and cloud setups from concept to high-availability production. Let's discuss how to optimize your workloads, secure your environment, and reduce cloud costs.