LLM inference speed: Blackwell vs M5 Max
Unified results from two benchmark runs, 22–24 Sep 2026. Workload: 1024-token prompt, 256 generated tokens, B concurrent independent sequences. Units are tokens/s, summed across the batch unless marked per user.
1 · Model comparison at a fixed B
2 · M5 Max ÷ Blackwell ratio
Cells show the M5 Max value divided by the Blackwell value, for models run on both machines. Below 1× (blue) means Blackwell is faster; above 1× (red) means M5 Max is faster. The Ratio basis filter switches between llama.cpp on both machines (same binary family and GGUF files) and the fastest engine on each machine.
3 · How throughput scales with concurrency
One panel per model, showing the selected metric against B = 1, 4, 16, 64 on a log₂ axis. Each panel has its own y-scale. With Decode per user selected, the panels show how much each user slows down as load grows.
4 · Total throughput vs speed per user
x = aggregate decode tok/s (total across all users), y = decode tok/s per user. Each line connects B = 1 → 64 for one machine and engine. Points further up and to the right are better. Lines that bend steeply downward show where adding users mostly slows each one down.
5 · Data table
Shows every concurrency level. The series, model, and architecture filters apply, and you can sort by clicking a column header. The source of truth is llm_benchmark_unified.csv.
Notes and caveats
- Like-for-like is llama.cpp vs llama.cpp. Both machines ran
llama-batched-benchwith the same flags and GGUF files, and it reports clean prefill and decode phases. vLLM and LM Studio are serving runtimes that measure differently: vLLM decodes at KV depth 128→384 instead of 1024→1280, and LM Studio interleaves prefill with decode, so its decode figure is a conservative lower bound (the optimistic upper bound is in the CSV asdecode_tok_s_upper). - Quantisation differs by engine. llama.cpp uses Q4_K_M, or UD-IQ3_XXS for the Flash-Next 512 and DeepSeek models. vLLM uses NVFP4/FP8/bf16 (the Qwen3.8 distills are bf16, so their vLLM B=1 decode is slower). MLX uses 4-bit g64.
- The Blackwell card is power-capped (Max-Q, 300 W). Its prefill is compute-bound and therefore understated compared with a 600 W card. Its decode is close to native.
- Flash-Next on Blackwell uses the warm median of 2 runs with the cold first run discarded. An older single run (1454 / 1912 prefill) sits in the M5 archive's copy of the Blackwell data; the newer Blackwell archive supersedes it.
- DeepSeek V4-Flash does not reliably fit on the 97 GB card (IQ2_M: 1 success, 5 out-of-memory failures). On M5 Max, llama.cpp ran UD-IQ3_XXS and mlx-vlm ran a 2-bit DQ build.
- M5 gaps: LM Studio MLX crashes on Llama 3.3 70B at B=64 (int32 slice overflow). mlx-vlm measures B=1 only. Qwen3.6-27B and Qwen3.6-35B-A3B were only run through MLX on the Mac.
- "—" means the point was not measured, usually because the KV cache would not fit at that B.