nine tokens a second

glia runs the local model on honeycrisp. it measured before it optimized: per-dispatch allocation costs 2–5 µs, but every synchronous command buffer costs a fixed 100–135 µs of Metal scheduling, and the GatedDeltaNet layer paid that nine times per token while uploading and reading back the full 3 MB recurrent state on every call. the projections were also re-dequantizing on every token — a 14× mistake, fixed first.

then the whole attention block for one decode token was fused into a single command buffer with the recurrent state resident on the GPU, and the SiLU feed-forward folded into the same buffer. 6.1 tokens a second became 9.0. the benchmark that found it is checked in beside the fix.

sources

• the 14× dequantization fix · fused single-command-buffer GatedDeltaNet block, 6.1 → 9.0 tok/s · SiLU FFN folded into the same buffer
• the measurement: run/examples/dispatch_overhead_bench.rs — per-dispatch 2–5 µs, per command buffer 100–135 µs; the commit body of 5ed62c2 carries the before/after
• machine: Apple M4 Max, honeycrisp Metal backend