Hasan Iqbal Butt

The production payment pod hit an out-of-memory crash in a twelve-node EKS cluster. Liveness probes were failing, the Horizontal Pod Autoscaler could not scale because the node pool was already at capacity, and alert fatigue from over two hundred non-critical alerts was drowning out the actual P0 signal. The on-call engineer had to manually SSH into nodes, figure out which pod was failing, restart it by hand, and then verify the fix. That whole cycle averaged about forty five minutes per incident.

Client's AWS bill was $6,400/mo for a mid-size SaaS platform. Over-provisioned EC2 instances running in the wrong region (ap-southeast-1 instead of us-east-1), 3 NAT Gateways routing S3 and DynamoDB traffic through public internet at $0.045/GB, no CDN so every API response served directly from ALB, RDS running a db.r5.xlarge at full capacity 24/7 even at 3 AM, and orphaned resources accumulating — 3 unattached 500GB EBS volumes, 2 floating Elastic IPs, and a dead NAT Gateway from a deleted VPC.

An 18GB chatbot model deployed on GCP Cloud Run took over 2 minutes to cold-start. The container had to download the full model weights from GCS on every scale-from-zero event. Max memory allocation of 32GB was barely enough — the model plus the Python runtime consumed 28GB at peak. During business hours, cold starts caused 504 Gateway Timeouts for the first users after an idle period.

Onboarding new DevOps engineers required giving them access to production Kubernetes clusters for hands-on training. This posed security risks — one accidental kubectl delete could take down a service. Sandbox environments were expensive ($800/mo per engineer) and constantly drifted from production configs, making training unrealistic.