Hardening a $50/hr GPU Cluster

Deploying massive Mixture-of-Experts (MoE) models like GLM-5.1 (754B total, 67B active) on an 8× B200 cluster or Qwen3.5-397B (17B active) on 4× B200 GPUs represents the absolute limits of serverless MLOps. At $50/hr and $25/hr respectively, startup delays, silent container crashes, and compilation bottlenecks are extremely costly.

The Hardware & Model Topology

Bottleneck 1: The 15-Minute JIT compilation tax

Mixture-of-Experts architectures use custom kernel engines like DeepGEMM (for GLM-5.1) or FlashInfer (for Qwen3.5) to run FP8 matrix multiplications dynamically. On first boot, these kernels compile Just-in-Time (JIT) to optimize execution shapes.

This JIT phase took 12 to 15 minutes. At $50/hr, every container warm-up cost $12.50 in idle compute.

The Fix: Caching JIT Kernels on Persistent Volumes

Instead of compiling kernels on every cold start, we redirected all dynamic kernel compilers to dump their output onto a persistent modal.Volume. This required mapping the environment variables of Triton, PyTorch Inductor, DeepGEMM, and FlashInfer to a mounted cache folder:

# Redirect JIT caches to persistent Modal volume
ENV TRITON_CACHE_DIR="/model-cache/.triton"
ENV TORCHINDUCTOR_CACHE_DIR="/model-cache/.inductor"
ENV FLASHINFER_WORKSPACE_DIR="/model-cache/.flashinfer"
ENV DEEPGEMM_CACHE_DIR="/model-cache/.deepgemm"

By mounting this volume with volume.reload() on container startup, subsequent container boots load the precompiled `.so` binary caches instantly. Cold start JIT overhead dropped from 15 minutes to 0 seconds.

Bottleneck 2: The DeepGEMM Startup Race Condition

During the compilation phase on GLM-5.1, launching SGLang or vLLM in a multi-GPU Tensor Parallel setting (TP=8) causes a race condition. Since all 8 GPUs attempt to write to the Triton/DeepGEMM JIT cache folder at the same millisecond, file locks collide, throwing a fatal cudaErrorIllegalAddress or NCCL timeout error.

The Solution: "Crash-on-First-Boot, Succeed-on-Second" Pattern

Since compiling TP=8 concurrently was unstable, we implemented a robust two-phase fallback:

1. Phase 1 (Offline Compilation): Run a single-GPU compilation script (TP=1) on a cheaper L4/L40S GPU to seed the matrix shape cache.
2. Phase 2 (Asymmetric Setup): On the main TP=8 cluster, the volume is loaded. If a file-lock crash occurs, the container terminates immediately. Modal automatically respawns the container, which reads the now-written cache files from the volume without trying to compile them, succeeding in 2-7 seconds.

Bottleneck 3: Silent Subprocess Death (Zombie Containers)

Modal orchestrates containers via a Python entry point. Because SGLang or vLLM must be started as a child process (to manage C++ libraries and multi-GPU IPC), standard Python crash handlers cannot catch their errors.

If SGLang crashed mid-request due to a CUDA Out-of-Memory (OOM) or NCCL sync failure, the child process died, but the Modal parent container remained running. Client requests continued routing to the container, receiving endless 502 Connection Refused errors.

The Fix: Active Heartbeat & Log Streaming Daemon

We built an active watchdog wrapper in Python that monitors child process health, pipes stdout/stderr to the main Modal logger, and terminates the container on any failure:

import subprocess
import threading
import sys
import os
import time

def stream_logs(pipe):
    for line in iter(pipe.readline, b''):
        sys.stdout.write(line.decode())
        sys.stdout.flush()

# Start the serving engine process
cmd = ["python", "-m", "sglang.launch_server", "--model-path", "/model-cache/GLM-5.1-FP8", "--tp", "8"]
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# Thread to stream logs in real-time to Modal dashboard
threading.Thread(target=stream_logs, args=(process.stdout,), daemon=True).start()

# Monitor thread
def monitor_engine():
    while True:
        if process.poll() is not None:
            print(f"CRITICAL: Serving process exited with code {process.returncode}")
            # Force exit the parent container immediately so Modal recycles the node
            os._exit(1) 
        time.sleep(10)

threading.Thread(target=monitor_engine, daemon=True).start()

Watchdog Timeouts in Weight Loading

Loading 754B parameters (even in FP8, it is ~400 GB of weights) from a persistent volume takes between 5 and 7 minutes. By default, SGLang's internal watchdog kills the worker processes if they do not communicate with the master node within 60 seconds.

We configured --watchdog-timeout 1200 (20 minutes) to prevent the coordinator from prematurely killing workers during the long weights-loading phase.

Scale-to-Zero Economics

Running 4× B200 GPUs continuously costs $18,000/month. By implementing GPU snapshots and caching, we reduced the warm-up latency to the point where we could use Modal's auto-scaling with min_containers=0.

Key Learnings

1. Subprocess logs are vital: Never redirect child process outputs to /dev/null. Pipe them to a stream logger to identify GPU errors immediately.
2. Watch your exit codes: Use os._exit(1) to instantly kill parent processes on subprocess failures. Standard sys.exit() executes cleanup handlers that can hang indefinitely if GPUs are in an error state.
3. Isolate caching: Always direct compiler cache folders to a persistent directory outside the ephemeral container root to guarantee warm-starts.

Cluster Error Log Catalog

Source: §6 (GLM-5.1-FP8), §9 (Qwen3.5-397B-A17B).