DeepSeek V3 in JAX
by Vaibhaav Tiwari · Framework · Text & code
promising
Tiny readable DeepSeek V3 in JAX for learning; not a usable model.
~190-line pure-JAX DeepSeek V3 with Multi-head Latent Attention, bias-balanced MoE (no auxiliary loss) and multi-token prediction; checkpoint in v1.0 release
AccessOpen codeLicenseMIT
Why we say so
The author describes about 190 lines of JAX with a 36.2M-parameter model trained on 207 detective novels in 2.6 hours on a MacBook Air. An HN commenter ran it end to end on an AMD RX 7600 XT, but needed two workarounds to avoid a GPU hang. It is a teaching-scale project, and the only quality claims come from the author.
Evidence
- Nov 14Ran end-to-end on an RX 7600 XT 16GB under Fedora with ROCm; needed HSA_OVERRIDE_GFX_VERSION=11.0.0 and --xla_gpu_enable_triton_gemm=false to avoid a GPU hang