High-Availability Booking Platform — AWS ECS on EC2
ECS (EC2) · Terraform Modules · Blue-Green Deploy · WAF · RDS Proxy
Client details anonymized. Metrics from the production system.
- Freelance project. I was the sole infrastructure engineer. The client had six services on ECS with manual Console deploys, no IaC, no WAF, and a database that crashed weekly from connection exhaustion.
- I moved everything to Terraform modules with isolated state, added RDS Proxy, implemented blue-green deployments via GitHub Actions, and put WAF in front of the ALB. EC2 launch type instead of Fargate saved $4,200/month.
- Platform has run at 99.95% uptime for nine months. Deploys went from 3-5 minutes of downtime to zero. The database has not crashed since RDS Proxy went live.
The Challenge
The Terraform was one 800-line
main.tf with a single state file. Changing a security group rule risked the entire infrastructure. The RDS PostgreSQL instance had no connection pooling. During peak hours, the database hit its 200-connection limit and new requests got connection refused errors. This happened weekly.There was no WAF. A security audit found SQL injection attempts in the access logs. Deployments required three to five minutes of downtime. Rollbacks were manual. Database backups were
pg_dump commands run whenever an engineer remembered.Constraints
- Solo infrastructure engineer. No DevOps team, no SRE, no platform team. Just me and the client's six backend developers.
- Budget constraint: Fargate was rejected at $8k/month. The infrastructure had to cost less than $4k/month including all AWS services.
- Zero-downtime requirement: the booking platform could not go offline for releases. Previous 3-5 minute deploy windows were losing bookings.
- MENA region compliance: traffic had to be geo-restricted. WAF rules had to block non-MENA requests without affecting legitimate users.
- Existing services had no health check endpoints. I had to add /health routes to all six services before blue-green deploys could work.
Key Decisions
ECS launch typeEC2 with Reserved Instances
Considered: Fargate (serverless, simpler) / EC2 with Reserved Instances (cheaper, more control)
Why: Six services running 24/7 with predictable baseline load. Fargate per-second billing costs $8k/month. EC2 Reserved Instances cost $3.8k. The 55% savings justified the added complexity of managing the ASG and capacity provider.
Trade-off: I now manage EC2 instance lifecycle, AMI updates, and capacity planning. Fargate eliminates all of that. For services that run intermittently, I still use Fargate.
Terraform module isolationPer-module state files with S3 backend
Considered: Single state file (simpler) / Per-module state files (safer, more complexity)
Why: The 800-line single-state Terraform was dangerous. One typo in a WAF rule could destroy the VPC. Isolated state files mean blast radius is contained to one module.
Trade-off: Six state files means six terraform apply commands for a full infra change. Cross-module references use data sources and remote state, which adds indirection.
Connection pooling approachRDS Proxy
Considered: Application-level pooling (HikariCP/pgBouncer) / RDS Proxy (managed, AWS-native)
Why: Six services means six connection pools to tune independently. RDS Proxy centralizes pooling, handles failover transparently during RDS Multi-AZ switchover, and requires no application code changes.
Trade-off: RDS Proxy adds ~$50/month cost and a small latency overhead (single-digit ms). Application-level pooling is free but requires per-service configuration and does not handle failover.
Deployment strategyBlue-green via ECS deployment configuration
Considered: Rolling update (simpler) / Blue-green with ALB target group swap (zero-downtime)
Why: Rolling updates still cause brief connection drops during task rotation. Blue-green with a separate target group means the old tasks serve traffic until the new ones are verified healthy.
Trade-off: Blue-green requires roughly double the EC2 capacity during deployment (both task sets running simultaneously). Deploy windows use more compute for about five minutes.
The Approach
I split the 800-line
main.tf into six Terraform modules, each with its own state file: networking, ecs-cluster, ecs-service, rds, s3-cloudfront, and waf. All state is on S3 with DynamoDB locking and KMS encryption. Changing WAF rules never risks the networking layer.RDS Proxy sits between the application and the database, multiplexing two thousand peak connections onto two hundred pooled ones. That fixed the weekly crashes. Blue-green deployments work through ECS deployment configuration. GitHub Actions builds the image, pushes to ECR, launches green tasks in a separate target group, waits for ALB health checks, then shifts traffic. If health checks fail, the pipeline rolls back automatically. AWS WAF attaches to the ALB with OWASP Top 10 rules, rate limiting, and MENA geo-restriction.
System Architecture
Interactive Demo
Walk through the system design and data flow for this project step by step.
Interactive Platform Demo
Blue environment is serving all traffic. Click Deploy to start a blue-green release.
Overview
Business Impact
What Went Wrong
RDS Proxy had a cold start problem I did not anticipate. After quiet overnight periods, the first burst of morning traffic hit the proxy before it had warmed its connection pool. The first fifty to one hundred requests saw connection wait times of three to five seconds. The fix was a Lambda warmup function on a CloudWatch Events schedule that sends a lightweight query every five minutes.
The Terraform module split created a dependency ordering issue during the initial migration. The
ecs-service module referenced the ALB target group ARN from the networking module via a data source. During the first apply, the networking module had not been applied yet, so the data source returned nothing. I caught it in the plan output. The fix was adding explicit depends_on relationships and documenting the apply order.Lessons Learned
Managed services have cold starts too. RDS Proxy is marketed as always-ready, but the connection pool scales down during quiet periods. If the application has a sharp traffic ramp in the morning, the proxy needs to be kept warm. This applies to any connection pooler.
Terraform module isolation is worth the complexity, but the dependency graph must be documented. The apply order for a fresh environment is: networking → rds → ecs-cluster → ecs-service → s3-cloudfront → waf. Getting this wrong does not break anything if you read the plan, but it wastes time.
Technical Highlights
- EC2 launch type over Fargate: 55% cost reduction, Reserved Instances saving $4,200/month
- 6 microservices on ECS: auth, booking, payment, notification + 2 frontends via ECR private registry
- 6 Terraform modules (networking, ecs-cluster, ecs-service, rds, s3-cloudfront, waf) with isolated state files
- RDS Proxy connection pooling: 2,000+ peak connections reduced to 200 pooled max
- Blue-green ECS deployment with ALB health check gating and automatic rollback
- AWS WAF with OWASP Top 10 rules, rate limiting at 1,000 req/IP/5min, MENA geo-restriction
- CloudFront CDN with S3 origin, cache invalidation on deploy, HTTPS-only
- GitHub Actions CI/CD: build, test, push to ECR, deploy, health check, auto-rollback
- CloudWatch: container logs (30-day retention), custom metrics, composite alarms to SNS/Slack/PagerDuty
- Automated RDS snapshots every 12 hours with 30-day retention and cross-region copy
- Route53 health checks, DNS failover, ALB across 3 AZs, AZ-balanced ECS task placement
Want results like this for your infrastructure?
I specialize in taking complex AI pipelines and cloud setups from concept to high-availability production. Let's discuss how to optimize your workloads, secure your environment, and reduce cloud costs.