SYLIN

llama.cpp

by ggml · Runtime · Text & code

promising

Widely used local runtime reaching 1.0 with stable APIs; maintainer load is a concern.

v1.0.0 declares API stability, removes 35 deprecated public-header functions, and adds a hot-expert cache for MoE models larger than GPU memory

AccessOpen codePricefreeRuns onWindows, macOS, Linux, iOS, AndroidLicenseMIT

Why we say so

ggml-org says 1.0 declares API stability, removes 35 deprecated functions and adds a hot-expert cache for large MoE models. Fahd Mirza used it to serve a 35B model locally without trouble. r/LocalLLaMA commenters worry about the pile of open issues and PRs and want more maintainers.

Evidence

  1. Nov 18
    Commenters worry about many open issues and PRs and ask for more maintainers; others ask for upstream MoE streaming techniques
  2. Nov 13
    v1.0.0 released; libllama and libggml adopt semantic versioning, with major bumps for API/ABI breaks
  3. Sep 8
    Running GLM 5.3 Flash needs a special llama.cpp branch with GLM 5.3 support; a stock existing install will not work
  4. Sep 1
    Praised for one-binary deployment, heterogeneous hardware and wide quantization support, but called slow for concurrent use
  5. Aug 15
    Reviewer prefers plain llama.cpp over the Ollama and LM Studio wrappers; lower memory use once context is reduced
Show all 7
  1. Aug 9
    Kimi K3 is not merged into stock llama.cpp; runs from a pull-request branch and CPU-only loading crashes, so a GPU is needed
  2. Jul 18
    Needs the --jinja flag for Gemma 4 tool calls; without it tool definitions are ignored and calls return as plain text

Runs on llama.cpp

18 models · newest first
Show all 18

Alternatives

vLLMReported to crash on non-standard multi-GPU setups, e.g. RTX 5090 plus 4090 pipeline parallelism with RAM caching, assuming identical devices per rank
SGLangReported to crash on mixed-GPU setups and pick wrong kernels, like vLLM, while offering paged attention and high concurrency
Qwen3.6-27B Fable Fusion 7118-bit on an A6000 under llama.cpp: 43.6 tok/s with MTP vs 22 without; draft acceptance 62%. Model card claimed about 20% gain
DFlashWith the DFlash drafter, Muse Glimmer GGUF went from 31.5 to 53.17 tokens/s on an A6000, about 1.7x faster