SYLIN

JetSpec

by Hao AI Lab · Technique · Text & code

Lossless speculative decoding using causal tree drafting, claimed up to 9x faster generation with unchanged output

Evidence

  1. Sep 1
    Shown finishing the same answer faster than DFlash and plain autoregressive decoding in the project's demo
  2. Sep 1
    Each branch of the draft tree attends only to its own earlier guesses, so more of the tree survives verification
  3. Sep 1
    On an H100 with a Qwen 8B model, baseline 28.03 tok/s rose to a 2.82x speedup, below the claimed 9x

Alternatives

Works with