Serving an 18GB ML Model on Cloud Run

The 18GB transformer model exceeded Cloud Run default memory limits. Even with maximum memory allocation at 32GB, the Python runtime plus model consumed 28GB at peak, leaving almost no headroom. The original Docker image included the model weights baked in, producing a 22GB container image. Every cold start required pulling this entire image from Container Registry and then loading all weights into memory, which took over two minutes. During low traffic periods, Cloud Run scaled to zero instances. The next incoming request would trigger the full cold start sequence, causing 504 Gateway Timeout errors for the first user. Deploying updates was also painful because pushing a 22GB image to the registry took over twenty minutes and consumed significant egress bandwidth.

The first change was separating the model weights from the application code. I built a multi-stage Dockerfile where the final stage contains only the Python runtime, dependencies, and application code, producing an image under 50MB. The 18GB model weights live in a dedicated Cloud Storage bucket in the same region as the Cloud Run service. At container startup, gcsfuse mounts the GCS bucket as a read-only FUSE filesystem at a local path. The model framework loads weights directly from this mounted path using memory-mapped file access, so there is no download step. I split the model initialization into chunked stages. The tokenizer and configuration load first, which takes about one second. The startup probe passes at this point. Then the heavy transformer layers stream in progressively. By the time the first real request arrives, the most commonly used layers are already warm in memory. For the scale-to-zero problem, I configured min-instances to 1 during business hours using Cloud Scheduler. A cron job sends an HTTP GET to the Cloud Run health endpoint every ten minutes between 8AM and 8PM. This keeps at least one warm instance alive without paying for idle compute time outside business hours. The Cloud Run service is configured with 2 vCPU, 32GB memory, startup CPU boost enabled, and concurrency set to 80 requests per container. Cloud Monitoring collects custom metrics for inference latency at p50, p95, and p99 percentiles, cold start event count, and container memory utilization. Alerts fire when p95 latency exceeds two seconds or when cold start rate exceeds one event per hour.

Deployed and served an 18GB transformer-based chatbot model on GCP Cloud Run with production-grade cold start optimization. The system uses gcsfuse to mount model weights directly from Cloud Storage as a FUSE filesystem, eliminating the need to download weights on every container startup. A multi-stage Docker build keeps the application image under 50MB while the 18GB model lives in a separate GCS bucket. Cloud Scheduler sends keepalive pings during business hours to prevent scale-to-zero, and Cloud Monitoring tracks inference latency, memory usage, and cold start events.

Cold start latency dropped from over two minutes to eight seconds, a fifteen times improvement. The keepalive strategy eliminated all 504 Gateway Timeout errors during business hours. Infrastructure cost for the keepalive mechanism was approximately twelve dollars per month. The multi-stage Docker build reduced the container image from 22GB to under 50MB, cutting Artifact Registry storage costs and deploy times by over ninety five percent. The system handles sustained inference traffic at p95 latency under 1.2 seconds.