Colibri
by JustVugg · Runtime · Text & code
Hy3 port adds a GPU tier: hot experts pinned in VRAM, warm in RAM, cold streamed from disk, and promoted by use
Evidence
- Jul 22Expert cache hit rate rose from 21% cold to 71% warm, holding about 113 GB in RAM; speeds improve with use
- Jul 22Hy3 on an H100 with 40 GB VRAM budget ran slowly but faster than CPU-only; GPU use mostly stayed around 1-2 GB
- Jul 21Engine runs GLM 5.2 quantized to int4, keeping the multi-token-prediction head at int8 so speculative decoding keeps working
Alternatives
Works with