HuggingFace Papers
8/10 signal
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
reasoningmemory
What happened
This paper investigates the 'Knowing-Using Gap' in LLMs, where fine-tuning successfully injects new facts (memorization) but fails to enable the model to use those facts for downstream reasoning. Using a novel 'self-patching' activation-relocation technique, the authors discover a 'knowledge-circuit misalignment': memorized facts exist in the model's weights but fail to route to the computation-effective layers required for reasoning. They propose a simple heuristic strategy that recovers 58-75% of the generalization headroom.
Why it matters
It provides a mechanistic explanation and a practical heuristic for why fine-tuned models fail to reason with newly memorized knowledge.
The take
This is a critical piece of mechanistic interpretability. It explains why naive fine-tuning for knowledge injection often yields models that look smart on paper but fail in agentic workflows. The discovery that the knowledge is there but misrouted suggests we need to shift our focus from 'how to train' to 'how to route' internal activations during fine-tuning.
Do this
Read this paper if you are fine-tuning LLMs to inject custom domain knowledge, and consider applying their activation-patching heuristics to diagnose generalization failures.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.