Serving Engine Internals

Running LLMs in production requires choosing the right runtime engine. Across 10 model deployments, we evaluated the three dominant engines: vLLM, SGLang, and llama.cpp. We discovered deep architectural differences, memory leaks, and engine-specific quirks that impact scalability.

What is the difference between vLLM, SGLang, and llama.cpp?

vLLM excels at multi-GPU continuous batching, SGLang wins on structured JSON and RadixAttention, and llama.cpp is the only practical choice for GGUF on heterogeneous hardware.

When should you use vLLM in production?

Choose vLLM for high-throughput FP8/BF16 serving on multi-GPU EKS with KubeRay, but disable CUDA graph capture (enforce_eager=True) when using GPU snapshots. vLLM is the most mature engine for high-throughput batching. However, it exhibits critical gotchas when combined with serverless GPU checkpointing (CRIU).

The Snapshot Hang Bug

During deployment of GLM-4.7-Flash-FP8, the container would hang indefinitely upon snapshot restore. The root cause was vLLM's default **CUDA Graph capture**. At startup, vLLM captures CUDA graphs for shapes up to max_num_seqs=16.

When Modal captures a GPU snapshot, the captured CUDA graphs refer to memory addresses that become invalid upon container restoration. The fix was forcing eager execution or running eager warmups:

# vLLM initialization with eager mode execution guards
from vllm import LLM, SamplingParams

llm = LLM(
    model="THUDM/glm-4-9b-chat",
    quantization="fp8",
    enforce_eager=True, # Disable graph capture to allow safe CRIU restores
    max_model_len=8192
)

When should you use SGLang instead of vLLM?

SGLang wins when snapshot-based scale-to-zero and structured JSON routing matter. Watch for FlashInfer VRAM creep under concurrent streaming on large MoE models. SGLang excels at complex, multi-turn prompts and structured JSON outputs due to **RadixAttention** (which dynamically caches KV caches in an LRU radix tree).

The Flash-Attention Memory Leak

During stress testing of the MiniMax-M2.7 (456B MoE) model on a 4× B200 cluster, we detected a slow but steady VRAM creep under concurrent streaming requests. Running a deep memory profiler revealed that SGLang's flash-attention buffer allocation was not freeing intermediate PyTorch tensor handles during model sharding across tensor parallel ranks.

We mitigated this by forcing garbage collection and flushing the PyTorch caching allocator after initialization, and utilizing SGLang's --chunked-prefill-size 4096 to cap the maximum context allocation block.

When is llama.cpp the right production serving engine?

Use llama.cpp for GGUF on L4, MacBook, or edge hardware, not for high-concurrency production where it lacks dynamic continuous batching. When serving GGUF models (like Gemma-4-26B-A4B-it), llama.cpp is the only engine capable of running across heterogeneous hardware (like Macbooks or cheap L4 GPUs).

Source-Compilation Pipeline

To support the latest Gemma-4 architectures before they landed in stable package releases, we built a custom multi-stage Docker build that compiles llama.cpp directly from source:

# Multi-stage CUDA builder
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builder
RUN apt-get update && apt-get install -y git cmake build-essential
RUN git clone --recursive https://github.com/ggerganov/llama.cpp.git && \
    cd llama.cpp && mkdir build && cd build && \
    cmake -DGGML_CUDA=ON .. && make -j$(nproc) llama-server

# Runtime image copy
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04
COPY --from=builder /llama.cpp/build/bin/llama-server /usr/local/bin/llama-server

We then wrap the llama-server binary in a Python subprocess, performing HTTP-level health checks before exposing it to the gateway.

How do quantization formats compare for production inference?

BF16 is the quality baseline; FP8 halves disk size with <0.05% perplexity loss on SGLang; GGUF Q4_K_M trades ~0.85% quality for 21 GB footprints on llama.cpp.

What is the cheapest way to serve a 70B+ LLM on AWS?

Self-hosted EKS + KubeRay + vLLM on Spot GPUs with Karpenter/KEDA. DocumentAI achieved $0.025/document vs $0.466 on Bedrock (18.6× reduction) at 50K documents/day using Qwen3.6-35B FP8.

What are the key lessons for choosing a serving engine?

Match the engine to workload shape: SGLang for structured JSON and snapshot hooks, vLLM for multi-node throughput, llama.cpp only for GGUF on edge or heterogeneous hardware.

1. SGLang wins for structured routing: SGLang compiles the output JSON schema directly into a regex guide, running up to 5× faster than vLLM's Out-of-Line JSON parsers.
2. vLLM is superior for multi-node setups: When scaling beyond a single node, vLLM's Ray integration makes Tensor Parallelism across machines seamless.
3. Never use llama.cpp for high concurrency: llama.cpp lacks dynamic continuous batching; it queues concurrent requests synchronously, resulting in high Time-to-First-Token (TTFT) when loaded.

Source: §2 (GLM-4.7-Flash-FP8), §5 (Gemma-4-26B-it), §6 (MiniMax-M2.7).