Deploying DeepSeek-R1 and Llama 3.3 Locally with vLLM and Docker

The release of open-weights frontier reasoning models like DeepSeek-R1 and high-parameter foundational models like Llama 3.3 70B has triggered a massive shift in AI infrastructure. While hosted proprietary APIs charge per token and introduce strict data privacy concerns, running models on your own hardware guarantees total data residency, zero rate limits, and predictable infrastructure costs.

However, running high-parameter reasoning models in production requires far more than basic llama.cpp or desktop GUIs. To serve hundreds of concurrent user requests with sub-second time-to-first-token (TTFT), engineering teams rely on vLLM—the high-throughput distributed LLM serving engine powered by PagedAttention and continuous request batching.

If you have already configured local models using our guide to Local LLM [Tool Calling](https://www.techjunkgigs.com/tool-calling-in-ai-agents-how-llms-execute-real-world-actions-in-2026/) with Ollama and Python, upgrading to a dedicated vLLM cluster is the next logical step for production workloads. In this comprehensive engineering guide, we walk through how to deploy deepseek r1 locally vllm docker containers, configure GPU tensor parallelism, and consume the OpenAI-compatible REST API from Python.

1. High-Level Architecture: vLLM PagedAttention vs Standard Inference

Traditional Python inference engines allocate a contiguous block of GPU VRAM for the maximum possible context window of every active request. This results in severe memory fragmentation (often 60% to 80% wasted VRAM) and caps concurrent requests to single digits.

vLLM resolves this with PagedAttention, inspired by virtual memory paging in operating systems:

+-------------------------------------------------------------------------+
|                  TRADITIONAL GPU INFERENCE (FRAGMENTED VRAM)            |
|                                                                         |
|  Request 1 (Prompt: 200 tokens)  ---> [ Allocated 8,192 Tokens VRAM ]   |
|  Request 2 (Prompt: 500 tokens)  ---> [ Allocated 8,192 Tokens VRAM ]   |
|                                                                         |
|  Result: 70%+ of GPU VRAM wasted as reserved but empty buffers.         |
+-------------------------------------------------------------------------+

                                    VS

+-------------------------------------------------------------------------+
|                 vLLM PAGEDATTENTION & CONTINUOUS BATCHING               |
|                                                                         |
|  Physical GPU Memory: Dynamic Non-Contiguous Page Blocks                |
|  [Page 1: Req 1] [Page 2: Req 2] [Page 3: Req 1] [Page 4: Req 2]        |
|                                                                         |
|  Continuous Batching: New requests join mid-flight without waiting.     |
|  Throughput: 3x - 5x higher tokens/sec compared to standard Hugging Face|
+-------------------------------------------------------------------------+

By storing Key-Value (KV) cache tensors in non-contiguous memory blocks, vLLM achieves near-zero memory waste and scales seamlessly to high-concurrency enterprise workloads.

2. Hardware & VRAM Sizing Guide for DeepSeek-R1 and Llama 3.3

Before deploying, select the distilled or full variant that matches your available GPU budget:

3. Host Prerequisites: Installing the NVIDIA Container Toolkit

To allow Docker containers to access your physical GPU hardware, install the NVIDIA Container Toolkit on Ubuntu or Debian Linux:

# 1. Configure the production NVIDIA package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |   sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# 2. Install toolkit packages
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# 3. Configure Docker daemon and restart
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 4. Verify GPU access inside Docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

4. Step-by-Step Deployment: Docker Compose Setup

Step 1: Writing docker-compose.yml

Create a production-ready docker-compose.yml file to orchestrate the vLLM server:

version: '3.8'

services:
  vllm-deepseek:
    image: vllm/vllm-openai:latest
    container_name: vllm_reasoning_node
    runtime: nvidia
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
      - VLLM_ATTENTION_BACKEND=FLASH_ATTN
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
      --trust-remote-code
      --dtype auto
      --gpu-memory-utilization 0.92
      --max-model-len 8192
      --enforce-eager
      --port 8000

Step 2: Launching the vLLM Container

Start the service in detached mode:

docker compose up -d

Monitor model weights downloading and engine initialization:

docker compose logs -f

When you see the message: INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit), your local server is ready for high-speed inference.

5. Multi-GPU Tensor Parallelism for 70B Models

When serving Llama-3.3-70B-Instruct or DeepSeek-R1-Distill-Llama-70B across dual GPUs (e.g., 2x RTX 3090 24GB), pass --tensor-parallel-size 2:

docker run --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model casperhansen/deepseek-r1-distill-llama-70b-awq \
  --quantization awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 8192

6. Consuming the Local Model in Python (OpenAI SDK & Streaming)

Because vLLM natively implements the OpenAI REST API specifications, any modern client library can connect by pointing base_url to http://localhost:8000/v1:

import os
from openai import OpenAI

# Connect to local vLLM container
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"  # vLLM requires no API key by default
)

# Stream DeepSeek-R1 reasoning tokens
stream = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
    messages=[
        {"role": "system", "content": "You are an expert distributed systems engineer."},
        {"role": "user", "content": "Explain how PagedAttention solves KV cache fragmentation in vLLM. Show a concrete numerical example."}
    ],
    temperature=0.6,
    stream=True
)

print("\n--- Streaming Local DeepSeek-R1 Response ---\n")
for chunk in stream:
    token = chunk.choices[0].delta.content or ""
    print(token, end="", flush=True)
print("\n")

To expose this capability in custom web backends, check out our guide on FastAPI Streaming LLM Responses with Server-Sent Events.

7. Connecting to Retrieval-Augmented Generation (RAG) Pipelines

To integrate your local vLLM engine into enterprise document search, connect it with LangChain or LlamaIndex:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

local_llm = ChatOpenAI(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
    openai_api_base="http://localhost:8000/v1",
    openai_api_key="dummy_key",
    temperature=0.2
)

prompt = ChatPromptTemplate.from_template("Answer the question based on the context:\n{context}\nQuestion: {question}")
chain = prompt | local_llm

response = chain.invoke({
    "context": "Our internal microservice auth uses Ed25519 asymmetric keys with 15-minute JWT rotation.",
    "question": "What key type and rotation interval does our auth service use?"
})
print(response.content)

For setting up high-performance vector retrieval, explore our tutorials on Top RAG Frameworks for LLMs and Production RAG with PostgreSQL and pgvector.

8. Common Errors & Troubleshooting

Error 1: RuntimeError: CUDA out of memory

Root Cause: The model weights plus KV cache allocation exceeded physical VRAM.
Fix: Reduce --gpu-memory-utilization from 0.95 to 0.85, or lower --max-model-len from 8192 to 4096. Alternatively, switch to an AWQ 4-bit quantized checkpoint.

Error 2: docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

Root Cause: The NVIDIA Container Toolkit was not installed or the Docker daemon was not restarted.
Fix: Run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker.

Error 3: ImportError: FlashAttention-2 is not installed

Root Cause: Passing flash attention flags on an unsupported hardware architecture (Turing or older).
Fix: Add the flag --enforce-eager to your vLLM command line.

9. Frequently Asked Questions (FAQ)

What is the advantage of vLLM over Ollama?

Ollama is designed primarily for single-user desktop convenience. vLLM is built for multi-user production servers: it features PagedAttention, continuous batching, tensor parallel multi-GPU scaling, and delivers up to 4x higher token throughput under concurrent load.

Can I run DeepSeek-R1 671B on consumer GPUs?

No. The full 671B model requires at least 400 GB of VRAM across an 8x H100 or 8x A100 GPU cluster. However, the distilled variants (14B and 32B) retain remarkable mathematical and coding reasoning and run comfortably on a single RTX 3090 or RTX 4090.

How do I enable API key authentication in vLLM?

Pass the --api-key your_secret_key flag in the command section of your Docker Compose file. Clients must then supply that token in the Authorization: Bearer <key> header.

10. Summary & Production Checklist

1. Verify GPU requirements: Select distilled models for single GPUs; use tensor parallelism (--tensor-parallel-size) for 70B models.
2. Install NVIDIA Container Toolkit: Verify GPU visibility with docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi.
3. Launch with Docker Compose: Set --gpu-memory-utilization 0.90 and map port 8000:8000.
4. Use OpenAI SDK: Set base_url="http://localhost:8000/v1" with streaming enabled.
5. Monitor system metrics: Use nvidia-smi and vLLM /metrics endpoint for Prometheus telemetry.

Related posts:

Fixing Common LangChain Memory Deprecation and Migration to LangGraph Quantizing and Running DeepSeek-R1 with Ollama, llama.cpp, and GGUF in Python Fixing Ollama Connection Refused (HTTP 11434) in Python and Docker Human-in-the-Loop AI Agents: Implementing Approval Breakpoints in LangGraph