Fastest LLM APIs by Tokens Per Second
Live 2026 inference speed & latency benchmark: Cerebras Wafer-Scale (1,800 tok/s), Groq LPU (820 tok/s), Together AI, and Fireworks AI. Measure time-to-first-token (TTFT) and real-world generation speed under concurrent load.
Interactive Fastest LLM APIs by Tokens Per Second Simulator
Adjust your production metrics below to model monthly expenditure, unit rates, and margin thresholds.
Itemized Unit Economics & Pricing Model
Detailed mathematical cost model comparing raw component units, provider markups, and optimization strategies.
| Architecture Component | Unit Pricing (2026) | Throughput / Quota Included | Production Notes & Trade-offs |
|---|---|---|---|
| Cerebras Wafer-Scale Engine (WSE-3) | 1,800 tok/s (8B) / 450 tok/s (70B) | $0.10 / $0.60 per 1M tokens | Giant monolithic silicon wafer eliminates memory bandwidth bottleneck |
| Groq Language Processing Unit (LPU) | 820 tok/s (8B) / 350 tok/s (70B) | $0.05 / $0.79 per 1M tokens | Deterministic SRAM architecture delivers sub-90ms time-to-first-token |
| Fireworks AI (FireAttention v3) | 220 tok/s (8B) / 140 tok/s (70B) | $0.20 / $0.90 per 1M tokens | Highly optimized CUDA kernels on NVIDIA H100 GPU clusters |
| Together AI Turbo (FlashAttention-3) | 180 tok/s (8B) / 110 tok/s (70B) | $0.18 / $0.88 per 1M tokens | Reliable high-throughput inference with broad open-weight model catalog |
| Standard Cloud APIs (OpenAI / Anthropic) | 45 - 80 tok/s across models | $2.50 - $15.00 per 1M tokens | Standard GPU virtualization shared across millions of global users |
Production Implementation Snippet
Benchmarking Real-Time TTFT and Generation Speed in Python. Deploy directly into your production pipelines.
# tps_benchmark.py - Measure Time-to-First-Token and sustained Tokens-Per-Second
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key="GROQ_API_KEY")
start_time = time.time()
stream = client.chat.completions.create(
model="llama-3.3-70b-versatile",
messages=[{"role": "user", "content": "Explain quantum computing in 250 words."}],
stream=True
)
first_token_time = None
token_count = 0
for chunk in stream:
if chunk.choices[0].delta.content:
if first_token_time is None:
first_token_time = time.time()
token_count += 1
total_time = time.time() - start_time
ttft_ms = (first_token_time - start_time) * 1000
generation_time = time.time() - first_token_time
tps = token_count / generation_time if generation_time > 0 else 0
print(f"Time to First Token (TTFT): {ttft_ms:.1f}ms")
print(f"Sustained Generation Speed: {tps:.1f} tokens/second")Engineering Gotchas & Pitfalls
Critical architectural edge cases documented from real-world high-traffic deployments.
The Concurrent Stream Throttling Cliff
While LPUs generate 800+ TPS on a single connection, running 200 concurrent streams can cause latency degradation or queue buffering if provider concurrency ceilings are breached.
TTFT vs Generation Speed Divergence
A provider may produce 1,000 tokens/second once generation starts, but take 800ms to process a 10,000-token prefill prompt. For RAG apps with giant prompts, prefill throughput matters as much as decode TPS.
Context Window Restrictions on Specialized Silicon
Cerebras and Groq store model weights and KV-cache directly in ultra-fast SRAM. Consequently, very large context windows (128k+) can carry higher pricing or lower availability than standard GPU clusters.
Frequently Asked Questions
Practical architectural questions answered with precise mathematical modeling.
Groq uses custom LPUs (Language Processing Units) powered by high-bandwidth SRAM instead of traditional HBM or GDDR RAM. This eliminates memory bus latency, allowing model parameters to execute at the processor's native clock frequency.
TTFT (Time-to-First-Token) is the latency between sending a request and receiving the first character of the response (crucial for perceived responsiveness). TPS (Tokens Per Second) is the rate at which subsequent words stream into the UI.
On 8B parameter models, Cerebras achieves up to 1,800 tokens/second (nearly 2x faster than Groq). On 70B models, both deliver roughly 350 to 450 tokens/second—fast enough to read a full page of text in less than one second.
Related Architecture Blueprints & Calculators
Explore related cost models and interactive simulators on CheckAPICost.
Hardware Architecture Showdown: LPUs, Wafer-Scale Engines & Speculative Decoding
Real-time conversational agents, interactive code autocomplete, and speculative search workflows require evaluating inference providers on Time-to-First-Token (TTFT) and sustained Tokens per Second (TPS). While standard hyperscalers running vLLM on Nvidia H100s produce 60 to 120 TPS per user sequence, specialized custom ASIC architectures—including Groq LPUs (Tensor Streaming Processors with SRAM), Cerebras CS-3 (Wafer-Scale Engine), and SambaNova SN40L (Reconfigurable Dataflow Units)—generate between 400 and 1,800 tokens per second at competitive per-token pricing.
High-Throughput Inference Benchmark Matrix (Llama 3.3 70B & 8B)
| Provider & Silicon | Model Tested | Output Speed (tok/s) | TTFT Latency | Input ($/M) | Output ($/M) |
|---|---|---|---|---|---|
| Cerebras (CS-3 WSE) | Llama 3.3 70B | 2,100 t/s | 90ms | $0.60 | $1.20 |
| Groq (LPU SRAM) | Llama 3.3 70B | 280 t/s | 120ms | $0.59 | $0.79 |
| Groq (LPU SRAM) | Llama 3.1 8B | 880 t/s | 65ms | $0.05 | $0.08 |
| SambaNova (SN40L RDU) | Llama 3.3 70B | 460 t/s | 110ms | $0.60 | $1.20 |
| Standard H100 (vLLM FP8) | Llama 3.3 70B | 85 t/s | 320ms | $0.13 (DeepInfra) | $0.40 |
Benchmarking TTFT and Streaming TPS (Python AsyncIO)
import time
import httpx
async def measure_inference_speed(api_url: str, api_key: str, model: str):
headers = {"Authorization": f"Bearer {api_key}"}
payload = {"model": model, "messages": [{"role": "user", "content": "Write a 500-word essay on distributed systems."}], "stream": True}
start_time = time.perf_counter()
first_token_time = None
token_count = 0
async with httpx.AsyncClient() as client:
async with client.stream("POST", api_url, headers=headers, json=payload) as response:
async for chunk in response.aiter_lines():
if chunk.startswith("data: ") and not chunk.endswith("[DONE]"):
if first_token_time is None:
first_token_time = time.perf_counter()
token_count += 1
total_time = time.perf_counter() - start_time
ttft = (first_token_time - start_time) * 1000
tps = token_count / (total_time - (first_token_time - start_time))
print(f"TTFT: {ttft:.1f}ms | Generated {token_count} tokens at {tps:.1f} tok/sec")Production SRE Guardrails
- Memory Bandwidth vs Compute Bound: Large models running on standard GPUs are bound by HBM memory bandwidth during the auto-regressive generation phase. Custom ASICs (Groq, Cerebras) utilize on-chip SRAM with petabytes/second of bandwidth, eliminating the memory wall.
- Client Rendering Bottlenecks: When receiving 1,500+ tokens per second from Cerebras, standard React DOM re-renders will freeze the browser tab. Batch token updates in requestAnimationFrame loops (e.g. 60 FPS) to maintain a smooth UI.
Engineering Q&A: Inference Speed Showdown
How does Cerebras achieve 2,000+ tokens per second on Llama 70B?
Cerebras prints the entire model onto a single monolithic wafer (44,000 square millimeters containing 900,000 AI cores and 44GB of on-chip SRAM). Because all weights remain on a single piece of silicon with zero off-chip interconnect delays, memory bandwidth reaches 21 petabytes per second.
When is a sub-$0.15 GPU endpoint better than a 1,000 TPS ASIC?
For background batch jobs, asynchronous document summarization, or cost-sensitive pipelines where latency is non-critical, DeepInfra or Together AI running FP8 models at $0.13/M tokens provides 5x better cost efficiency per dollar.