Fixing CUDA Out of Memory (OOM) When Serving DeepSeek-R1 with vLLM

When deploying DeepSeek-R1 (such as DeepSeek-R1-Distill-Qwen-14B or 32B) using vLLM, encountering a sudden torch.cuda.OutOfMemoryError: CUDA out of memory during startup or high concurrency is one of the most common issues developers face.

By default, vLLM attempts to pre-allocate up to 90% of available GPU memory for its Key-Value (KV) cache. If your model weights consume most of the VRAM, or if the context length is set too high, the initialization or first inference batch crashes immediately.

Here is the quick, production-tested solution to eliminate CUDA OOM errors when serving DeepSeek-R1 with vLLM.

The Core Solution: 4 Critical vLLM Flags

Adjust your startup command or Docker configuration to include these four memory-tuning flags:

python3 -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --gpu-memory-utilization 0.95 \
  --max-model-len 8192 \
  --kv-cache-dtype fp8 \
  --enforce-eager \
  --port 8000

Why Each Flag Works:

1. --gpu-memory-utilization 0.95: Tells vLLM to utilize up to 95% of available VRAM instead of leaving excessive overhead, maximizing space for KV-cache blocks.
2. --max-model-len 8192 (Crucial): By default, modern models claim 32k to 128k context windows. If left unconstrained, vLLM pre-allocates massive KV cache tables that immediately exceed physical VRAM. Clamping this to 4,096 or 8,192 tokens frees up several gigabytes.
3. --kv-cache-dtype fp8: Enables FP8 (8-bit floating point) quantization for the KV cache. This cuts the memory footprint of active conversation context by 50% with virtually zero degradation in reasoning accuracy.
4. --enforce-eager: Disables CUDA graph capture. While CUDA graphs improve throughput slightly, they consume an extra 1–2 GB of temporary scratchpad memory during initialization. Disabling it prevents startup OOM crashes on tight GPUs (like RTX 3090 / 4090).

Python Client Verification

Once the vLLM server is running with the flags above, verify connectivity in Python:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none"
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
    messages=[
        {"role": "user", "content": "Explain PagedAttention in 2 sentences."}
    ],
    temperature=0.6,
    max_tokens=256
)

print(response.choices[0].message.content)

Sizing Your VRAM Before Deploying

Before downloading model weights, calculate whether your GPU can handle the target model and context window using our free Local LLM & DeepSeek-R1 VRAM Hardware Sizer.

For full multi-GPU and container orchestration, read our complete guide on Deploying DeepSeek-R1 and Llama 3.3 Locally with vLLM and Docker.

Related posts:

Fixing Common LangChain Memory Deprecation and Migration to LangGraph Fixing Ollama Connection Refused (HTTP 11434) in Python and Docker Deploying DeepSeek-R1 and Llama 3.3 Locally with vLLM and Docker Human-in-the-Loop AI Agents: Implementing Approval Breakpoints in LangGraph