TokenWing
Runtime · Text & code
Developer claims it streams a majority of routed MoE experts from SSD; a user reported 45-50 tok/s on a 120B MoE with 12 GB VRAM and 32 GB RAM
Runtime · Text & code
Developer claims it streams a majority of routed MoE experts from SSD; a user reported 45-50 tok/s on a 120B MoE with 12 GB VRAM and 32 GB RAM