Model per-second execution costs, container cold starts, and idle keep-alive windows against 24/7 dedicated cloud instances to find your exact deployment tipping point.
At 1,500 requests/day, serverless scale-to-zero saves significant money over 24/7 dedicated hardware.
| Billing Dimension | Serverless Deployment | Dedicated 24/7 Instance |
|---|---|---|
| Active Inference Compute | $43.11 | Included in 730h |
| Cold Starts & Keep-Alive Wait | $1.09 | $0.00 (Zero latency) |
| Persistent Network Volume (PVC) | $10.00 | $10.00 |
| Baseline 24/7 Hardware Rent | $0.00 (Scale-to-zero) | $321.20 |
| Total Monthly Spend | $54.20 | $331.20 |
| Cost per 1,000 Requests | $1.20 / 1k | $7.36 / 1k |
With your current daily traffic, RunPod Serverless is dramatically cheaper. You save $277.00/mo because you aren't paying for the 92% of the day when your GPU would otherwise sit completely idle.
import runpod
import torch
from transformers import pipeline
# Global model initialization (persists across warm invocations)
generator = None
def init_worker():
global generator
generator = pipeline('text-generation', model='/runpod-volume/models/llama-70b-fp8', device_map='auto')
def handler(event):
prompt = event['input'].get('prompt', '')
max_tokens = event['input'].get('max_tokens', 100)
result = generator(prompt, max_new_tokens=max_tokens)
return {"generated_text": result[0]['generated_text']}
if __name__ == "__main__":
init_worker()
runpod.serverless.start({"handler": handler})
When an invocation lands on an idle serverless endpoint, latency is split into: (1) Container Provisioning: Docker image pull & host allocation (2-5s with FlashBoot); (2) Model Checkpoint Loading: Transferring 15GB-40GB of safetensors from shared volume into system RAM (4-12s); (3) VRAM Allocation & CUDA Warmup: Transferring tensor buffers across PCIe into GPU HBM (2-4s).
Never bake 30GB model weights directly into your Docker image. Large images experience slow registry download bottlenecks (over 60s). Storing model weights on a persistent RunPod Network Volume ($0.10/GB/mo) located in the same datacenter cluster allows workers to mount weights over a 100Gbps local network fabric in under 8 seconds.
Setting an idle keep-alive window (e.g. 60 seconds) keeps the container warm for consecutive requests, eliminating cold start latency. However, if your traffic arrives at irregular intervals (e.g. one request every 50 seconds), the worker remains in a perpetual warm state, billing active compute rates 100% of the time without actually serving load!
For mature AI applications, the gold standard is hybrid infrastructure: lease 1 dedicated instance (e.g. 1x RTX 4090 or 1x H100) to serve the continuous 24/7 baseline traffic with zero cold starts, and configure an autoscaling RunPod Serverless endpoint that handles overflow traffic only during peak business hour spikes.
| Storage & Weight Architecture | P50 Container Spin | P95 Cold Start Latency | Initialization Cost | Best Workload Match |
|---|---|---|---|---|
| Pre-Baked Docker Image (Docker Hub) | 28.4 seconds | 64.2 seconds | $0.018 / spin | Small <2GB models (BERT, Whisper Tiny) |
| RunPod Network Volume (NFS Mount) | 6.8 seconds | 14.2 seconds | $0.003 / spin | Standard SDXL, FLUX, Llama 8B |
| RunPod FlashBoot (Local SSD Cache) | 2.1 seconds | 4.8 seconds | $0.001 / spin | High-traffic user-facing interactive APIs |
| Dedicated 24/7 Warm VRAM | 0.0 seconds | 0.0 seconds | $0.000 / spin | Mission-critical enterprise SLAs |
# Stage 1: Fast minimal base image with CUDA runtime (not full devel)
FROM nvidia/cuda:12.1.1-runtime-ubuntu22.04
ENV PYTHONUNBUFFERED=1 \
DEBIAN_FRONTEND=noninteractive \
PIP_PREFER_BINARY=1
RUN apt-get update && apt-get install -y --no-install-recommends \
python3-pip python3-dev git && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Pre-install heavy torch wheels to leverage layer caching
RUN pip3 install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
RUN pip3 install --no-cache-dir runpod transformers accelerate safetensors
COPY handler.py /app/handler.py
# Mount weights at runtime via network volume - NEVER COPY MODEL WEIGHTS INTO DOCKER
CMD ["python3", "-u", "/app/handler.py"]