vLLM
Runtime · Text & code
promising
Established open-source model server; the big speed figures are vendor claims
vLLM 0.34.0 enables extensible KV cache by default, makes Model Runner V2 the sole GPU runner, and reports up to 1.95x faster NVFP4 decoding on GB200
Why we say so
The headline numbers are from vLLM's own team, such as 1.7M tok/s per GPU for DeepSeek V4.1 Flash on Vera Rubin NVL72 and 1.95x faster NVFP4 decoding on GB200. Fahd Mirza's hands-on use matches the basics: two models fit side by side on one 48GB card in under 20GB, and Maxwell V1.0 Flash ran locally with a smaller context cutting memory use. No independent benchmark of the speed claims is on record.
Evidence
- Nov 17DeepSeek-V4.1-Flash on Vera Rubin NVL72 hits 1.7M tok/s per GPU at 387 tok/s per user, 58x GB300 at that point; Mooncake KV pool cut P90 TTFT 0.98s to 0.38s
- Nov 12LibTorch Stable ABI means new PyTorch releases no longer force a vLLM rebuild; vLLM-Omni makes 2048² images in 0.63s on 4×B300
- Sep 7Speculative decoding on AMD Instinct MI300X/MI355X with ROCm: native MTP, Gemma 4 MTP, EAGLE-3, DFlash and DSpark, set via --speculative-config
- Sep 6NVFP4 and AWQ W4A16 quants under vLLM failed to close tool calls and botched a Cisco command; commenters call them low-quality quants
- Sep 1Reported to crash on non-standard multi-GPU setups, e.g. RTX 5090 plus 4090 pipeline parallelism with RAM caching, assuming identical devices per rank
Runs on vLLM
123 models · newest firstShow all 123
LFM VL
Maxwell Flash
Aion-1.0-Instruct-2.3B
Kimi
Qwen3.5-4B-Qwen3.8-27B-Draft
Olala mini 7A1B
Perplexity Decider
Perplexity Decider Flash
DaisyRose-27B
Flint
Ripple 65B Preview
SenseNova-Loop 9B
d3
GLM
ERNIE-Flash-Base
Ling
Muse Glimmer
Nemotron 3.5 Lightning
Hy
Inkling-Small
Agnes
LongCat
Alpie Seed
ThinkingCap-Qwen3.8-27B
GLiDE
KORMo-Web-11B
KORMo-Web-4B
KORMo-Web-9B
Drex Max
DSpine
Qwen3 Embedding
MMFlash
Deletion Tuning
Nex
WVY-Blender-5m
TTF ternary training
Lara
BotPsy-0.8B
Diffbot-Small-XL
OpenThai2.1-OCR-Markdown-20261118
KLM-1.6
Mirage
Hydrogen 1.0
Arete-27B
Marin-Mochi-535B-A23B
Unlimited OCR
Clover 35B
Groot
Nanbeige
Spark-X2.6-1.7B
Raon-1-nano
Sage
Carbon-X
Ling 3.0 Flash Santé
MemT
Liquid d1 2B- penclaw-Kimi-K3.1-pro-unrestricted
Nemotron Super
Step
ALoDLM
AstaBrief
AREX
IQuest-Q1
Aikido Altar
Ovis Embedding
CLM
dots Note
Occamy
Bespoke Nimble
Needle
Laya
DeepSeek Pro
ZGCM
DSpark
MiniCPM
Qwen3.8-27B GSQ-RCO GGUF
DeepSeek Flash Vision
TimesFM
Spark-X2.5-4B
Extropic Z1T
K2 Horizon 7B Uno
Thomson-Small
DiffusionGemma
PhoneLLM
Ornith
Granite
S1-mini
Qwen3.8-4B-Distill
Escha-W2
Ornith 1.5 9B
Qwen3.8-27B Ridge
Antares
Qwen3.8-9B (Empero)
Qwen3.8-27B GGUF (ggml-org)
North Micro Vision
Maple-Preview
Big Bang
Neutrino-8B
Qwen3.6-27B Fable Fusion 711
Shieldstral
Nomic Embed Text
Nemotron Embed
Instella-MoE
GPT-2
Kimi K3 DSpark
Fara
GLM-5.2 Vision NVFP4
Laguna S
Solar Open
Intern-S
catmind
Alternatives
More vLLM