MLOps & GPU Serving

Freelance client

Self-Hosted LLM Inference on Bare-Metal GPUs

NVIDIA A100 · vLLM · llama.cpp · AWQ Quantization · CUDA 12.2

By Hasan Butt·Published: May 2025

82%Inference Cost Reduction vs. Closed APIs

The Challenge

US legal tech client was burning $18,000/month on proprietary document summarization token bills with 1.45s p95 latency and strict data-privacy compliance needs. The firm had 200 active attorneys. The initial attempt used cloud-hosted A100 instances at $4.50 per GPU-hour. They purchased two refurbished Dell PowerEdge R750xa servers, each with two A100 80GB GPUs at a colocation facility. The primary hurdle was driver compatibility: Ubuntu 22.04 came with driver 535, but llama.cpp required CUDA 12.2 (driver 545+). Additionally, unquantized Llama 3.1 70B required 140GB of VRAM, leaving insufficient memory for KV caching beyond 8 concurrent users.

The Approach

Rejected cloud-managed endpoints (AWS SageMaker/GCP Vertex at $38K+/yr reserved baseline). Deployed bare-metal dedicated A100s running custom vLLM and llama.cpp runtimes with AWQ/GGUF quantization, continuous batching, and KV cache paging. For NVIDIA driver management, wrote Ansible playbooks with DKMS snapshots for automated rollbacks. Q4_K_M quantization shrank model memory from 140GB to 40GB, unlocking 48 concurrent sequences with 97% MMLU benchmark retention. FastAPI gateway provides dynamic queue-depth routing with Prometheus + DCGM telemetry.

System Architecture

Loading architecture diagram...

Interactive Demo

Walk through the system design and data flow for this project step by step.

Interactive Topology HUD

Click any component node in the topology cluster to inspect SRE specifications and telemetry.

Node Telemetry HUD

gateway status

FastAPI GatewayAPI INGRESS ROUTER

Metric 01p99 Latency180ms

Metric 02Concurrency48 active streams

Metric 03Load BalancingRound Robin

Telemetry Graph FeedLive

CONFIG SPECIFICATION DETAILS

A custom Python FastAPI gateway coordinates incoming client requests, manages model version hot-swaps, and balances loads across GPU workers.

BenefitEnsures low-latency streaming via Server-Sent Events (SSE) and handles up to 48 concurrent requests before queueing.

Risk Without ThisWithout queue scheduling, concurrent load spikes would cause out-of-memory errors on individual GPU workers.

Overview

Moved 200 attorneys off $18K/month in API token bills onto self-hosted A100s at 82% lower cost with 420ms p95 latency. Built a self-hosted LLM inference platform running Llama 3.1 70B on four NVIDIA A100 80GB GPUs across two bare metal servers. The system uses llama.cpp compiled with CUDA 12.2 for GPU-accelerated inference, serving quantized GGUF models through a custom FastAPI gateway with load balancing across GPU workers. GGUF Q4_K_M quantization reduced the model from 140GB to 40GB while maintaining 97% of the full-precision benchmark score on MMLU. The platform handles 2,400 tokens per second aggregate throughput and serves a p99 latency of 180 milliseconds for the first token. Total infrastructure cost is $3,240 per month, compared to $18,000 per month on closed APIs.

Business Impact

Monthly bill dropped from $18,000/mo to $3,240/mo (82% cut). Latency dropped from 1,450ms to 420ms. Zero third-party data egress. Aggregate throughput across the four GPUs is 2,400 tokens per second with concurrent request handling via continuous batching. First-token latency sits at 180 milliseconds p99. The GGUF Q4_K_M quantization retained 97% of the original model quality on MMLU benchmarks and 95% on HumanEval coding tasks. The system handles up to 48 concurrent inference requests before queueing.

Technical Highlights

llama.cpp compiled from source with CUDA 12.2, cuBLAS, and flash attention enabled for maximum GPU throughput
GGUF Q4_K_M quantization reducing Llama 3.1 70B from 140GB to 40GB with 97% quality retention on MMLU
Tensor parallelism across 4 NVIDIA A100 80GB GPUs using llama.cpp native multi-GPU support with NVLink
Continuous batching handling 48 concurrent requests with dynamic batch scheduling based on sequence length
Custom FastAPI gateway with health-check-based routing across 2 bare metal servers, graceful draining on GPU worker restart
NVIDIA driver and CUDA toolkit upgrade automation with Ansible playbooks, pre-upgrade GPU health checks, and automatic rollback on failure
Prometheus metrics exporter for GPU temperature, memory utilization, inference latency histogram, and tokens per second
Nginx reverse proxy with rate limiting (60 requests per minute per API key), request buffering, and SSL termination
Model hot-swap capability: load a new GGUF model file without restarting the inference server or dropping active connections
Automated benchmarking pipeline that runs MMLU and HumanEval after every quantization to catch quality regressions before deployment

Frequently Asked Questions

Q: What was the biggest challenge in this self-hosted GPU setup?

Managing NVIDIA driver upgrades (CUDA 12.2 / driver 545+) on bare-metal Ubuntu servers without risking system downtime. I solved this with Ansible automation providing DKMS kernel module snapshots and automated post-install validation.

Q: Why choose GGUF Q4_K_M quantization?

Benchmarking 5 quantization profiles proved Q4_K_M reduced model footprint from 140GB to 40GB, freeing enough VRAM for KV caching across 48 concurrent users while maintaining 97% benchmark retention on MMLU.

Q: What was the concrete financial ROI?

The monthly bill dropped from $18,000/mo on proprietary cloud APIs to $3,240/mo for colocation and power (an 82% cut), while dropping p95 latency to 420ms with zero third-party data egress.

Want results like this for your infrastructure?

I specialize in taking complex AI pipelines and cloud setups from concept to high-availability production. Let's discuss how to optimize your workloads, secure your environment, and reduce cloud costs.

Book Free 20-Min Infra Audit →Hire on Upwork

Related Projects

Platform & SRE