MLOps & Model Serving

Freelance client

Custom LLM Serving on Bare Metal GPUs with llama.cpp

llama.cpp · NVIDIA A100 · CUDA 12.2 · Nginx · Prometheus · GGUF Quantization

By Hasan Iqbal Butt·Published: May 2025

82%Cost Reduction vs Cloud GPUs

The Challenge

The client, a mid-size law firm with 200 attorneys, was spending $18,000 per month on managed LLM API calls. They wanted to bring inference in-house for cost control and data privacy. The initial attempt used vLLM on rented cloud A100 instances at $4.50 per GPU-hour. They purchased two refurbished Dell PowerEdge R750xa servers, each with two A100 80GB GPUs, at a colocation facility. The problem started with NVIDIA drivers: Ubuntu 22.04 shipped with driver 535, but llama.cpp required CUDA 12.2 (driver 545+). The Llama 3.1 70B model in full precision needs 140GB of GPU memory, leaving insufficient KV cache for more than 8 concurrent requests. The firm needed at least 40.

The Approach

The infrastructure has three layers: GPU driver management, inference server, and API gateway. For NVIDIA driver management, I wrote Ansible playbooks that handle the full upgrade lifecycle. The playbook runs nvidia-smi, creates a DKMS snapshot as rollback, installs the new driver, and runs a validation suite. If any check fails, it rolls back automatically. The inference layer runs llama.cpp compiled from source with CUDA 12.2, cuBLAS, and flash attention. I tested five GGUF quantization formats. Q4_K_M was the best trade-off: 140GB to 40GB, freeing enough memory for 48 concurrent sequences at 4096 context length. llama.cpp runs as a systemd service with tensor parallelism across GPUs via NVLink. The API gateway is a FastAPI application routing requests by queue depth. Nginx handles SSL and rate limiting. Prometheus scrapes GPU metrics via NVIDIA DCGM exporter. Grafana dashboards show real-time utilization and cost per token.

System Architecture

Loading architecture diagram...

Interactive Demo

Walk through the system design and data flow for this project step by step.

Interactive Topology HUD

Click any component node in the topology cluster to inspect SRE specifications and telemetry.

Node Telemetry HUD

gateway status

FastAPI GatewayAPI INGRESS ROUTER

Metric 01p99 Latency180ms

Metric 02Concurrency48 active streams

Metric 03Load BalancingRound Robin

Telemetry Graph FeedLive

CONFIG SPECIFICATION DETAILS

A custom Python FastAPI gateway coordinates incoming client requests, manages model version hot-swaps, and balances loads across GPU workers.

BenefitEnsures seamless streaming via Server-Sent Events (SSE) and handles up to 48 concurrent requests before queueing.

Risk Without ThisWithout queue scheduling, concurrent load spikes would cause out-of-memory errors on individual GPU workers.

Overview

Built a self-hosted LLM inference platform running Llama 3.1 70B on four NVIDIA A100 80GB GPUs across two bare metal servers. The system uses llama.cpp compiled with CUDA 12.2 for GPU-accelerated inference, serving quantized GGUF models through a custom FastAPI gateway with load balancing across GPU workers. GGUF Q4_K_M quantization reduced the model from 140GB to 40GB while maintaining ninety seven percent of the full-precision benchmark score on MMLU. The platform handles 2,400 tokens per second aggregate throughput and serves a p99 latency of 180 milliseconds for the first token. Total infrastructure cost is $3,200 per month, compared to roughly $18,000 per month for equivalent capacity on cloud GPU instances.

Business Impact

The self-hosted setup reduced monthly inference costs from $18,000 (using cloud A100 instances) to $3,200 for colocation and power, an eighty two percent reduction. Aggregate throughput across the four GPUs is 2,400 tokens per second with concurrent request handling via continuous batching. First-token latency sits at 180 milliseconds p99. The GGUF Q4_K_M quantization retained ninety seven percent of the original model quality on MMLU benchmarks and ninety five percent on HumanEval coding tasks. The system handles up to 48 concurrent inference requests before queueing. GPU utilization averages seventy eight percent during peak hours.

Technical Highlights

llama.cpp compiled from source with CUDA 12.2, cuBLAS, and flash attention enabled for maximum GPU throughput
GGUF Q4_K_M quantization reducing Llama 3.1 70B from 140GB to 40GB with 97% quality retention on MMLU
Tensor parallelism across 4 NVIDIA A100 80GB GPUs using llama.cpp native multi-GPU support with NVLink
Continuous batching handling 48 concurrent requests with dynamic batch scheduling based on sequence length
Custom FastAPI gateway with health-check-based routing across 2 bare metal servers, graceful draining on GPU worker restart
NVIDIA driver and CUDA toolkit upgrade automation with Ansible playbooks, pre-upgrade GPU health checks, and automatic rollback on failure
Prometheus metrics exporter for GPU temperature, memory utilization, inference latency histogram, and tokens per second
Nginx reverse proxy with rate limiting (60 requests per minute per API key), request buffering, and SSL termination
Model hot-swap capability: load a new GGUF model file without restarting the inference server or dropping active connections
Automated benchmarking pipeline that runs MMLU and HumanEval after every quantization to catch quality regressions before deployment

Frequently Asked Questions

Q: What was the biggest challenge in this self-hosted GPU setup?

The biggest challenge was managing the NVIDIA driver upgrades (CUDA 12.2 / driver 545+) on bare-metal Ubuntu servers without risking system bricking. I resolved this by writing Ansible playbooks that automated DKMS backups, driver installation, and multi-step post-install validation checks, with automated rollbacks on any failure.

Q: Why did you choose GGUF Q4_K_M quantization over other formats?

I benchmarked five formats (Q8_0, Q6_K, Q5_K_M, Q4_K_M, and Q4_0) on MMLU and HumanEval benchmarks. GGUF Q4_K_M quantization reduced the model memory requirement from 140GB to 40GB, which freed up critical GPU memory for KV caching to handle up to 48 concurrent requests, while maintaining 97% of the original model performance on benchmarks.

Q: What was the concrete cost saving of this deployment?

We reduced the client's monthly LLM inference costs from $18,000 using cloud A100 instances to approximately $3,200 for colocation hosting and power, achieving an 82% cost reduction while improving first-token latency to 180ms p99.

Want results like this for your infrastructure?

I specialize in taking complex AI pipelines and cloud setups from concept to high-availability production. Let's discuss how to optimize your workloads, secure your environment, and reduce cloud costs.

Book a Free 30-Min Call →Hire on Upwork

Related Projects