vLLM vs TGI vs Ollama Production Cost

Inference Serving Engine Benchmark: PagedAttention, Continuous Batching & Memory Efficiency

Serving open-weight frontier models (DeepSeek-V3, Llama 3.3 70B, Qwen 2.5 Coder) in production requires selecting the right inference runtime. Ollama is built for single-user local development using `llama.cpp` CPU/GPU bindings; in multi-user concurrent production, Ollama serializes requests, leading to request queue pileups and catastrophic latency degradation.

For high-concurrency production deployments, the real battle is between vLLM (UC Berkeley) and Text Generation Inference (TGI - Hugging Face). vLLM pioneered PagedAttention, which divides the Key-Value (KV) cache into virtual memory pages, eliminating internal and external fragmentation. This unlocks up to 4x higher request concurrency on the same NVIDIA H100 hardware compared to standard static memory allocation.

Engine Performance & Throughput Benchmark (Llama 3.3 70B on 1x H100 SXM5 80GB FP8)

Production vLLM Docker Launch Command with Automatic Prefix Caching

docker run --gpus all -d -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model neuralmagic/Llama-3.3-70B-Instruct-FP8 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.94 \
  --max-model-len 8192 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-seqs 128

Inference Gotchas: Chunked Prefill & Multi-GPU Topology

• Inter-GPU Interconnect (NVLink vs PCIe): When running 70B models across multiple GPUs with Tensor Parallelism (`--tensor-parallel-size 2` or `4`), PCIe-only interconnects incur severe communication bottlenecks during AllReduce operations. Always ensure your host supports NVLink (NVIDIA SXM5 or HGX) to prevent 40% throughput degradation.
• Chunked Prefill for Latency Predictability: Without chunked prefill, a user sending a massive 16k-token prompt blocks the GPU for 800ms during the prefill phase, stalling token generation for all other 63 active users. Enabling `--enable-chunked-prefill` splits large prompt computations across iterations, protecting p99 Time to First Token (TTFT).

Prometheus Scraping Config & Grafana Alert Rules for vLLM

# Prometheus alert rules for vLLM GPU saturation
groups:
  - name: vllm_alerts
    rules:
      - alert: VllmHighGpuCacheUsage
        expr: vllm:gpu_cache_usage_factor > 0.92
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "vLLM KV Cache exhaustion on {{ $labels.instance }}"
          description: "GPU KV cache usage exceeds 92%. New requests will trigger prompt queueing."

Advanced KV Cache Sizing Math & Memory Allocation

Calculating exact KV cache memory capacity is vital to prevent mid-generation CUDA Out-Of-Memory (OOM) faults. For a model with $L$ layers, hidden dimension $H$, and multi-query / grouped-query attention heads $N_{kv}$, the memory consumed per token is given by $2 \times 2 \times L \times H \times \text{precision bytes}$. For Llama 3.3 70B (Grouped-Query Attention with 8 KV heads, 80 layers, head dim 128) in FP8 precision, each token requires 160 KB per sequence. On an 80GB H100 with 36GB reserved for model weights and 4GB for activations, the remaining 40GB allocates enough space for over 250,000 active KV cache tokens—comfortably handling 64 concurrent users generating 4k context sequences simultaneously without CPU swapping.