SYLIN

vLLM

Runtime · Text & code

promising

Established open-source model server; the big speed figures are vendor claims

vLLM 0.34.0 enables extensible KV cache by default, makes Model Runner V2 the sole GPU runner, and reports up to 1.95x faster NVFP4 decoding on GB200

AccessOpen codePricefreeRuns onmacOS, LinuxLicenseApache 2.0

Why we say so

The headline numbers are from vLLM's own team, such as 1.7M tok/s per GPU for DeepSeek V4.1 Flash on Vera Rubin NVL72 and 1.95x faster NVFP4 decoding on GB200. Fahd Mirza's hands-on use matches the basics: two models fit side by side on one 48GB card in under 20GB, and Maxwell V1.0 Flash ran locally with a smaller context cutting memory use. No independent benchmark of the speed claims is on record.

Evidence

  1. Nov 17
    DeepSeek-V4.1-Flash on Vera Rubin NVL72 hits 1.7M tok/s per GPU at 387 tok/s per user, 58x GB300 at that point; Mooncake KV pool cut P90 TTFT 0.98s to 0.38s
  2. Nov 12
    LibTorch Stable ABI means new PyTorch releases no longer force a vLLM rebuild; vLLM-Omni makes 2048² images in 0.63s on 4×B300
  3. Sep 7
    Speculative decoding on AMD Instinct MI300X/MI355X with ROCm: native MTP, Gemma 4 MTP, EAGLE-3, DFlash and DSpark, set via --speculative-config
  4. Sep 6
    NVFP4 and AWQ W4A16 quants under vLLM failed to close tool calls and botched a Cisco command; commenters call them low-quality quants
  5. Sep 1
    Reported to crash on non-standard multi-GPU setups, e.g. RTX 5090 plus 4090 pipeline parallelism with RAM caching, assuming identical devices per rank

Runs on vLLM

123 models · newest first
Show all 123

Alternatives

llama.cppPraised for one-binary deployment, heterogeneous hardware and wide quantization support, but called slow for concurrent use
Superlinked Inference EngineRan embedding, re-ranking, entity extraction and generation with four models through one server and client, no separate deployments
Nari Qwen3-TTSAuthors say vLLM-Omni and SGLang-Omni are too slow for production and can break realtime playback at low latency