Gated MLA
by AMD · Technique · Text & code
Learned gate after multi-head latent attention that turns down weak signals, credited for strong results with few active parameters
by AMD · Technique · Text & code
Learned gate after multi-head latent attention that turns down weak signals, credited for strong results with few active parameters