Medium LLM
★ 9/10 signal
Why Does Your RAG Retrieve Before the Model Thinks?
reasoningcontexttool-use
What happened
Explains the shift from classic pre-retrieval RAG to 'mid-reasoning RAG' and 'thinking-trace retrieval' inside the chain of thought (CoT). Classic RAG fails when reasoning models think for thousands of tokens and face mid-thought uncertainty. Solutions include firing retrieval dynamically when model uncertainty spikes (achieving 71.2% F1 on MuSiQue) and retrieving other models' thinking traces rather than raw documents (boosting Gemini-2.5-Flash on AIME from 53.3 to 83.3). Frontier models like Gemini 3.1 Pro and Kimi K2 are already integrating search and code execution directly inside their thinking steps.
Why it matters
It shifts RAG from a static preprocessing step to an active, dynamic tool-use loop driven by the model's internal reasoning state.
The take
This is a massive paradigm shift. Static RAG is dead for reasoning models. Building systems that allow models to self-trigger retrieval or tool execution mid-thought based on internal entropy/uncertainty metrics is the next frontier of context engineering. Retrieving structured 'thinking traces' rather than raw text chunks is also a brilliant way to transfer reasoning capabilities cheaply.
Do this
Experiment with triggering vector search or tool execution dynamically mid-generation (e.g., when token probability/confidence drops below a threshold) rather than doing it all upfront.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.