Hardening a $50/hr GPU Cluster

Deploying massive Mixture-of-Experts (MoE) models like GLM-5.1 (754B total, 67B active) on an 8× B200 cluster or Qwen3.5-397B (17B active) on 4× B200 GPUs represents the absolute limits of serverless MLOps. At $50/hr and $25/hr respectively, startup delays, silent container crashes, and compilation bottlenecks are extremely costly.

What hardware topology is needed for 754B MoE on B200 clusters?

GLM-5.1-Open FP8 needs 8× B200 at TP=8 (~$50/hr); Qwen3.5-397B FP8 fits on 4× B200 at TP=4 (~$25/hr). The table below is the production topology we hardened on Modal.

What causes 15-minute cold starts on B200 GPU clusters?

MoE architectures compile custom kernels (DeepGEMM, FlashInfer) JIT on first boot. At $50/hr, each 15-minute warmup wastes $12.50 in idle compute. Mixture-of-Experts architectures use custom kernel engines like DeepGEMM (for GLM-5.1) or FlashInfer (for Qwen3.5) to run FP8 matrix multiplications dynamically. On first boot, these kernels compile Just-in-Time (JIT) to optimize execution shapes.

This JIT phase took 12 to 15 minutes. At $50/hr, every container warm-up cost $12.50 in idle compute.

The Fix: Caching JIT Kernels on Persistent Volumes

Instead of compiling kernels on every cold start, we redirected all dynamic kernel compilers to dump their output onto a persistent modal.Volume. This required mapping the environment variables of Triton, PyTorch Inductor, DeepGEMM, and FlashInfer to a mounted cache folder:

# Redirect JIT caches to persistent Modal volume
ENV TRITON_CACHE_DIR="/model-cache/.triton"
ENV TORCHINDUCTOR_CACHE_DIR="/model-cache/.inductor"
ENV FLASHINFER_WORKSPACE_DIR="/model-cache/.flashinfer"
ENV DEEPGEMM_CACHE_DIR="/model-cache/.deepgemm"

By mounting this volume with volume.reload() on container startup, subsequent container boots load the precompiled `.so` binary caches instantly. Cold start JIT overhead dropped from 15 minutes to 0 seconds.

How do you fix TP=8 DeepGEMM startup race conditions?

Use a crash-on-first-boot, succeed-on-second pattern: seed caches on TP=1, then let TP=8 containers read pre-written JIT artifacts from a persistent volume. During the compilation phase on GLM-5.1, launching SGLang or vLLM in a multi-GPU Tensor Parallel setting (TP=8) causes a race condition. Since all 8 GPUs attempt to write to the Triton/DeepGEMM JIT cache folder at the same millisecond, file locks collide, throwing a fatal cudaErrorIllegalAddress or NCCL timeout error.

The Solution: "Crash-on-First-Boot, Succeed-on-Second" Pattern

Since compiling TP=8 concurrently was unstable, we implemented a robust two-phase fallback:

1. Phase 1 (Offline Compilation): Run a single-GPU compilation script (TP=1) on a cheaper L4/L40S GPU to seed the matrix shape cache.
2. Phase 2 (Asymmetric Setup): On the main TP=8 cluster, the volume is loaded. If a file-lock crash occurs, the container terminates immediately. Modal automatically respawns the container, which reads the now-written cache files from the volume without trying to compile them, succeeding in 2-7 seconds.

How do you prevent silent subprocess death in GPU inference containers?

Run an active Python watchdog that streams child logs and calls os._exit(1) on the parent when the serving subprocess dies, forcing Modal to recycle zombie containers. Modal orchestrates containers via a Python entry point. Because SGLang or vLLM must be started as a child process (to manage C++ libraries and multi-GPU IPC), standard Python crash handlers cannot catch their errors.

If SGLang crashed mid-request due to a CUDA Out-of-Memory (OOM) or NCCL sync failure, the child process died, but the Modal parent container remained running. Client requests continued routing to the container, receiving endless 502 Connection Refused errors.

The Fix: Active Heartbeat & Log Streaming Daemon

We built an active watchdog wrapper in Python that monitors child process health, pipes stdout/stderr to the main Modal logger, and terminates the container on any failure:

import subprocess
import threading
import sys
import os
import time

def stream_logs(pipe):
    for line in iter(pipe.readline, b''):
        sys.stdout.write(line.decode())
        sys.stdout.flush()

# Start the serving engine process
cmd = ["python", "-m", "sglang.launch_server", "--model-path", "/model-cache/GLM-5.1-FP8", "--tp", "8"]
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# Thread to stream logs in real-time to Modal dashboard
threading.Thread(target=stream_logs, args=(process.stdout,), daemon=True).start()

# Monitor thread
def monitor_engine():
    while True:
        if process.poll() is not None:
            print(f"CRITICAL: Serving process exited with code {process.returncode}")
            # Force exit the parent container immediately so Modal recycles the node
            os._exit(1) 
        time.sleep(10)

threading.Thread(target=monitor_engine, daemon=True).start()

Why do weight-loading watchdogs kill large MoE deployments?

Default 60-second SGLang watchdogs terminate workers during 5–7 minute weight loads. Set --watchdog-timeout 1200 for 754B FP8 models. Loading 754B parameters (even in FP8, it is ~400 GB of weights) from a persistent volume takes between 5 and 7 minutes. By default, SGLang's internal watchdog kills the worker processes if they do not communicate with the master node within 60 seconds.

We configured --watchdog-timeout 1200 (20 minutes) to prevent the coordinator from prematurely killing workers during the long weights-loading phase.

What scale-to-zero savings are achievable on B200 clusters?

With kernel caching and reliable recycling, scale-to-zero on 4× B200 cut idle costs from ~$18,000/month to ~$4,500/month while preserving sub-minute recovery. Running 4× B200 GPUs continuously costs $18,000/month. By implementing GPU snapshots and caching, we reduced the warm-up latency to the point where we could use Modal's auto-scaling with min_containers=0.

What are the key lessons for hardening production GPU clusters?

Stream subprocess logs, use os._exit(1) on child failure, and isolate JIT caches on persistent volumes. Never redirect serving engine output to /dev/null.

1. Subprocess logs are vital: Never redirect child process outputs to /dev/null. Pipe them to a stream logger to identify GPU errors immediately.
2. Watch your exit codes: Use os._exit(1) to instantly kill parent processes on subprocess failures. Standard sys.exit() executes cleanup handlers that can hang indefinitely if GPUs are in an error state.
3. Isolate caching: Always direct compiler cache folders to a persistent directory outside the ephemeral container root to guarantee warm-starts.

What errors appear most often on hardened B200 clusters?

TP=8 JIT file-lock races, NCCL timeouts during init, and watchdog kills during long weight loads are the top production failure modes. Each has a documented mitigation below.

Source: §6 (GLM-5.1-FP8), §9 (Qwen3.5-397B-A17B).