Production EKS Platform with SRE Observability

The platform ran on a single managed node group of three m5.xlarge On-Demand instances with no autoscaling. During traffic spikes, pods would queue in Pending state because the node group had no capacity headroom and took four to five minutes to scale. The payment service JVM was configured with a 512MB heap inside a container with a 512MB memory limit, leaving zero room for off-heap allocations like thread stacks, NIO buffers, and garbage collection overhead. Every checkout spike triggered OOMKilled events. When pods crashed, Kubernetes rescheduled them but startup took forty-five seconds due to JVM warmup, causing visible outages. No liveness probes existed, so zombie pods that were technically running but not responding to requests stayed in the ALB target group rotation. The only monitoring was CloudWatch container insights with five-minute aggregation granularity, meaning the team discovered outages from customer complaints, not alerts. There were no SLO definitions, no error budgets, and no structured incident response process.

The first change was replacing the static managed node group with Karpenter. Karpenter provisions nodes on demand when pods cannot be scheduled, selecting instance types from a weighted pool that prioritizes Spot instances for non-critical workloads and reserves On-Demand for the payment and auth services. This eliminated the four-minute node scaling delay because Karpenter provisions in under ninety seconds. For the OOMKilled problem, I right-sized every container: the JVM heap stays at 512MB but the container limit is 768MB, giving 256MB of headroom for off-heap memory. Liveness probes hit the /health endpoint every ten seconds with a three-failure threshold before Kubernetes kills the pod. Readiness probes gate on database connectivity so pods only enter the ALB target group after confirming they can serve traffic. PodAntiAffinity rules prevent two replicas of the same service from landing on the same node, so a single node failure never takes out all replicas. Pod Disruption Budgets enforce minAvailable during rolling updates and voluntary node drains. The observability stack replaced CloudWatch entirely. Prometheus scrapes all twelve services at fifteen-second intervals, collecting 847 metric series. Grafana dashboards show the four golden signals: request rate, error rate, latency percentiles, and saturation. Alertmanager routes critical alerts like error rate above five percent for two minutes to PagerDuty, and warnings like pod restart count above three to Slack. Sentry captures application errors with full stack traces, source maps, and release tags tied to the Helm chart version. The CI/CD pipeline uses GitHub Actions with Helm: pull requests trigger lint, diff, and dry-run against the staging cluster. Merges to main auto-deploy to production with a five-minute health check window. If readiness probes fail during the rollout, Helm automatically rolls back to the previous release.

Designed and operated a production Kubernetes platform on AWS EKS running twelve microservices across three availability zones. The platform uses Karpenter for node provisioning with a mix of On-Demand and Spot instances to cut compute costs by roughly forty percent. Horizontal Pod Autoscaler scales on both CPU utilization and custom Prometheus metrics like request latency p99. The full SRE observability stack runs Prometheus for metrics collection, Grafana for dashboards, Alertmanager for incident routing to PagerDuty and Slack, and Sentry for application-level error tracking with source maps. Helm charts with GitHub Actions GitOps handle deployments through dev, staging, and production environments with automated rollback on failed health checks.

Platform uptime reached ninety nine point nine percent over a six month measurement window. Mean time to detect dropped from roughly fifteen minutes of manual log tailing to under ninety seconds via Alertmanager firing to PagerDuty. Mean time to recover dropped from ten plus minutes of manual pod restarts to under two minutes through automated liveness probe recovery and HPA pre-scaling. Compute spend dropped by approximately forty percent after switching from fixed On-Demand node groups to Karpenter provisioning with Spot instance pools. Pod crash loops from OOMKilled events dropped to zero after right-sizing container memory limits with headroom for off-heap allocations.