#1 ·
Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
Linear AttentionLong ContextModel Architecture
Why it matters
KV-cache growth is the binding constraint on long-context serving, and linear-attention models are the most credible escape route — but they have trailed Transformers on quality. Muir's contribution is diagnostic before it is architectural: it shows the gated delta-rule state is far less sensitive to erasure than to writes, because key energy concentrates on a sparse channel subset while writes scale everything. That analysis justifies a low-rank erase gate and frees parameters for a wider feed-forward net. The payoff is what builders care about: at 1.34B parameters it trains 2.7x faster than Transformer at 32K, extrapolates from 4K training length to 10M-token streams without perplexity drift, and holds near-100% passkey retrieval to 1M tokens.
Muir re-derives the state update for gated linear attention. Empirically, state updates respond far less to the erase gate than to writes, and slow-decaying state rows accumulate large norms that dominate unweighted readout. So Muir fuses all gates into a single sensitivity-allocated low-rank projection, adds a matched-filter readout gate that re-weights state rows, and reinvests the saved parameters into a wider feed-forward network, with a parallel chunkwise update that stays numerically stable at the large chunk sizes wide matrix units need. Across 166M-1.34B parameters it lowers perplexity by 1.8-2.0% over GDN-2 and 2.5-3.6% over Transformer, raises commonsense accuracy by up to 1.9 points, and beats both baselines on multi-hop and multi-needle recall at 1.34B.
arXiv | HuggingFace
#2 ·
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz
Post-trainingDistillationCompute Efficiency
Why it matters
Post-training is where inference-scale teams spend their training budgets, and the default question is whether distillation's dense per-token supervision beats RL's sparser rewards per FLOP. This study gives a clean, systematic answer across task difficulty, model sizes, domains, and optimizers: with an external teacher, on-policy distillation generally loses to GRPO on accuracy per training FLOP and inflates response length — but when the teacher is the student's own GRPO checkpoint, distillation recovers that checkpoint's math accuracy with 50x fewer FLOPs. For a team that has already paid for an RL run, self-distillation is an almost-free way to lock in the accuracy, and the multi-teacher variant is a concrete recipe for merging domain specialists into one model at 5-8% of the specialist compute.
The authors measure held-out accuracy against training FLOPs for vanilla reverse-KL on-policy distillation versus GRPO, sweeping task difficulty, student and teacher size, domain, and optimization choices. With an external post-trained teacher, OPD generally underperforms GRPO on accuracy per FLOP and often suffers response-length inflation, even when the teacher itself was further trained with GRPO. The result flips when the teacher is the student's own GRPO checkpoint: OPD recovers that checkpoint's math accuracy with 50x fewer distillation FLOPs than GRPO needs to reach the same accuracy, and the pattern holds with a larger student, on code, and in a second model family. Multi-teacher OPD consolidates per-domain GRPO experts into one student matching each expert's accuracy, with the distillation stage costing 5.2% of specialist FLOPs on two domains and 8.0% on four; counting specialist training plus distillation, the pipeline reaches joint GRPO performance on a two-domain mix with 42.0% fewer total FLOPs.
arXiv
#3 ·
Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
Xiaobo Chen, Genliang Zhu, Ziyuan Wang, Chu Wang
LLM ReliabilityCode GenerationTesting
Why it matters
If your pipeline lets an LLM repair code after a runtime or API migration, this is the paper to read before the next release. It is preregistered and executes every candidate against the real runtimes, which rules out the usual confounds: an all-assertions gate on typical inputs released 41% of wrong repairs. The fix is cheap — one edge input derived from the release note cuts the wrong-release rate to 6.5% (6.1% on unseen clusters), and a blind frontier-model writer of edge inputs matches it at 7.0%. The sober part: adversarial distractors that match legacy output on typical inputs still pass at 15-17%, so a clean run on normal traffic is still not evidence of correctness.
Across 73 tasks covering 60 documented changes in CPython, Node.js, and SQLite, the authors preregistered a study of LLM-repaired procedures that run cleanly on typical inputs but return wrong results, evaluating every candidate on real runtimes against withheld contracts. The standard all-assertions gate released 0.410 of wrong repairs; a single edge input written from the release note reduced this to 0.065, and 0.061 on 15 unseen clusters. Three edge inputs written blind by GPT-5.6-luna, which never saw the program, reached 0.070. When hard distractors matching legacy output on typical inputs were added, 0.168 and 0.153 of the hard wrong programs still passed their frozen gates. Across four model families, the release-note edge input cut wrong releases per attempt in all three open-weight families on all three menus — for example from 0.233 to 0.057 — without lowering correct completion rates.
arXiv
Outlook
The day's theme is efficiency measured honestly: cheaper long-context state, cheaper post-training, and a sober look at what 'clean run' actually proves about code a model wrote. Expect follow-ups in two directions. On training cost, multi-teacher distillation that consolidates per-domain RL experts into one student at single-digit percentages of specialist FLOPs is a pattern other labs will try to reproduce. On trust, release-note-derived edge inputs are a cheap, high-leverage addition to any pipeline that ships model-generated migrations.