Model 8x H100 SXM5 node rental, continuous batching throughput (vLLM / TensorRT-LLM), KV-cache saturation, diurnal idle drop, and MLOps maintenance to find your exact profitability threshold.
At 500M tokens/month, self-hosting loses money due to idle cluster capacity.
| Expense Category | Self-Hosted Dedicated | Commercial API |
|---|---|---|
| Raw Compute Rent (730h) | $15,709.60 | $0.00 |
| MLOps DevOps Engineering | $2,000.00 | $0.00 |
| NVMe Scratch & Egress Transfer | $25.00 | Included |
| API Inference Tokens | $0.00 | $3,300.00 |
| Total Monthly Expense | $17,734.60 | $3,300.00 |
| Effective Cost / 1M Tokens | $35.47 / M | $6.60 / M |
Unless your sustained token volume reaches 2.68 Billion tokens per month, paying commercial API rates is far more cost-effective. Dedicated hardware incurs fixed costs regardless of user volume, exposing early startups to severe margin collapse.
docker run --gpus all -d -p 8000:8000 --ipc=host \ -v /data/models:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model deepseek-ai/DeepSeek-R1-Distill-Llama-70B \ --tensor-parallel-size 8 \ --max-model-len 16384 \ --gpu-memory-utilization 0.92 \ --quantization fp8
Developers frequently calculate cost assuming 100% saturation: "An H100 generates 300 tps, meaning 780M tokens/month per GPU." In practice, human users sleep. Nighttime traffic drops to <10%, while weekend traffic plunges by 75%. Because you pay $21.52/hr every second the server is online, your effective real-world utilization rarely exceeds 28% to 35%, tripling your effective cost per token.
In transformer generation, the decoding phase is strictly memory-bandwidth bound. An Nvidia H100 SXM5 delivers 3.35 TB/s of HBM3 memory bandwidth. For a 70B parameter model in FP16 (140GB), reading the model weights requires 41.8ms per output token, capping single-stream throughput at ~24 tokens/sec. Only by batching 40+ concurrent user streams does the H100 saturate its compute tensor cores.
DeepSeek-R1 (671B parameters) requires 1,340GB of VRAM in unquantized 16-bit precision. A standard 8x H100 SXM5 node provides only 640GB of VRAM (8 x 80GB). To run unquantized R1, you must lease a 16x H100 cluster across an InfiniBand fabric, doubling your hourly bill to ~$42/hr. On a single 8x node, you are constrained to FP8 or 4-bit AWQ quantized weights.
Every active token in context consumes VRAM for Key-Value activations. When serving large reasoning prompts (16k to 32k context), KV-cache can consume more VRAM than the model weights themselves! If your traffic spikes without strict continuous batching limits, the vLLM worker will throw an Out-Of-Memory (OOM) CUDA crash, causing complete service downtime.
| Open-Weights Model | Parameter Count | FP16 Weights VRAM | FP8 Weights VRAM | Recommended Cluster | Max 8k Concurrency |
|---|---|---|---|---|---|
| DeepSeek-R1 Full | 671B MoE (37B active) | 1,342 GB (OOM on 8x) | 671 GB | 16x H100 SXM5 (Dual Node) | ~96 streams (FP8) |
| DeepSeek-R1-Distill-70B | 70.6B Dense | 141 GB | 71 GB | 2x to 4x H100 SXM5 | ~340 streams |
| Llama 3.3 70B Instruct | 70.6B Dense | 141 GB | 71 GB | 2x to 4x H100 SXM5 | ~340 streams |
| Qwen 2.5 72B Instruct | 72.7B Dense | 145 GB | 73 GB | 2x to 4x H100 SXM5 | ~310 streams |
| Mistral Large 2 (123B) | 123B Dense | 246 GB | 123 GB | 4x to 8x H100 SXM5 | ~220 streams |
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-deepseek-70b-service
namespace: ml-inference
spec:
replicas: 1
selector:
matchLabels:
app: vllm-deepseek-70b
template:
metadata:
labels:
app: vllm-deepseek-70b
spec:
containers:
- name: vllm-engine
image: vllm/vllm-openai:latest
args: [
"--model", "deepseek-ai/DeepSeek-R1-Distill-Llama-70B",
"--tensor-parallel-size", "8",
"--max-model-len", "16384",
"--gpu-memory-utilization", "0.92",
"--kv-cache-dtype", "fp8",
"--enable-chunked-prefill", "true",
"--max-num-seqs", "128"
]
resources:
limits:
nvidia.com/gpu: 8
memory: "480Gi"
cpu: "64"
requests:
nvidia.com/gpu: 8
memory: "256Gi"
cpu: "32"
ports:
- containerPort: 8000
volumeMounts:
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cache/huggingface
name: model-storage
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 128Gi
- name: model-storage
persistentVolumeClaim:
claimName: nfs-model-weights-pvc
Model PagedAttention VRAM pools, max concurrent streams, and tokens/sec throughput across H100 clusters.
Serverless GPUPinpoint the exact duty cycle tipping point between per-second serverless execution and 24/7 dedicated VMs.
Cloud GPUsCompare RunPod micro-billing against 24/7 secure instances with FlashBoot container startup caching.