Replacing 25% of heads maintained downstream question-answering accuracy.
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
Positive users praise the research replacing transformer attention heads with readable Python programs as a cool mechanistic interpretability advance, while negative users worry the resulting models have troubling robustness issues.
No Digg Deeper questions have been answered for this story yet.
New paper! We introduce a new automated interpretability technique where attention heads are explained with Python programs. Turns out you can drop-in replace ~40% of attention patterns in Llama-3B with outputs of these programs and barely affect task performance!
More broadly, I’m excited about the actionable implications of this technique: Understanding attention phenomena has historically led to architectural improvements in Transformers (see e.g. attention sinks), and I’m excited about the potential for this technique to uncover more such opportunities.
Make sure to check out @amirihayes_ thread below! ⬇️
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
👉 New preprint! Automated interpretability by approximating / replacing NN components (here attention heads) with programs.
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
@amirihayes_ Very cool work, this paper might be related and interesting for you (very similar themes as yours): https://arxiv.org/abs/2602.08857
Very cool work ⬇️
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
Very cool work!
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
The thing that impresses me here is not that it works (duh of course it's algorithms), but that they suffer a perplexity hit without commensurate downstream collapse. Next you'll say we can do 0th order optimization on arguments of these scripts and…
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
@amirihayes_ this is a really cool idea.
@amirihayes_ https://github.com/AmiriHayes/explaining_attention_heads/blob/main/data%2Fgpt2_programs.py
Explaining Attention with Program Synthesis asks a sharp question: what are attention heads actually doing, and can we describe it precisely?
- The paper uses program synthesis to automatically generate symbolic descriptions of attention head behavior, moving beyond heatmaps toward interpretable rules. - Synthesized programs capture the computational patterns of individual heads in a human-readable form, making mechanistic claims testable. - The approach surfaces structured, repeatable behaviors rather than treating attention as a black box of soft weights.
Lines of inquiry it opens: - Can symbolic mechanisms improve transformer compositional abilities? - Do transformers learn generalizable algorithms or instance-based patterns? - Why do standard transformers fail on problems requiring serial algorithmic reasoning? https://inquiringlines.com/related/2606-19317-explaining-attention-with-program-synthesis/
@amirihayes_ @VictorTaelin
@amirihayes_ wow this seems wild! taking a look!
@amirihayes_ very cool!
@amirihayes_ Everything is pattern recognition.
@amirihayes_ Wow. Neat. Big mech int win
@amirihayes_ kind of counterintuitive that with such structural similarities between the actual and best fit generated attention head, the 100% replacement is still so close to random guessing
@amirihayes_ The RASP paper was a very good one in this domain: https://arxiv.org/abs/2106.06981
@pplank8 @amirihayes_ thanks for pinging me here!!
@amirihayes_ @AbhishekAs34298
@teortaxesTex 这鲁棒性确实有点细思极恐了
