HuggingFace Papers Jul 22, 2026 ★ 9/10 signal

ISO: An RLVR-Native Optimization Stack

researchinfrastructurereasoningmodels

Summary

This paper investigates the optimization layer for Reinforcement Learning with Verifiable Rewards (RLVR), identifying a phenomenon called "spectral inheritance": RLVR can reuse a base model's weight spectra while acquiring new behaviors by changing the associated singular frames. Based on this, the authors propose Isospectral Optimization (ISO), a fixed-spectrum optimization framework with offline (ISO-Merger) and online (ISO-Optimizer) versions. On a Qwen3-8B-Base model, the online ISO-AdamW optimizer reaches the same accuracy as AdamW with 2.7x fewer training steps, achieving an aggregate accuracy of 0.495 in 100 steps versus AdamW's 270.

Problem

The optimization methods used for Reinforcement Learning with Verifiable Rewards (RLVR) are largely inherited from pre-training, despite fundamental differences between the two regimes. Pre-training learns from dense, token-level supervision, while RLVR adapts a policy using sparse, outcome-level rewards. This mismatch motivates an investigation into the specific structure of weight-space updates during RLVR to design a more suitable optimization layer.

Method

The authors analyze the singular value decomposition (SVD) of model weights during RLVR, finding that singular value spectra remain close to the base model's (near-isospectrality) while singular frames change. They propose Isospectral Optimization (ISO), a framework that fixes the base model's spectra and optimizes only the singular frames. The framework is instantiated as ISO-Merger for data-free merging of specialist models and ISO-Optimizer, which applies optimizers like AdamW to the frame variables during online training.

Details

Online Optimization (ISO-Optimizer):

On Qwen3-8B-Base, ISO-Optimizer achieves faster convergence and higher final accuracy compared to standard AdamW.

Optimizer	Aggregate Accuracy	Training Steps
AdamW	0.495	270
ISO-AdamW	0.495	100
ISO-AdamW	0.509	210

ISO-AdamW reaches the final accuracy of the AdamW baseline using 2.7x fewer training steps.

Key Finding 1: Spectral Inheritance:

The paper finds that RLVR updates are near-isospectral, meaning they preserve the singular value spectrum of the base model's weights, in sharp contrast to Supervised Fine-Tuning (SFT).

Metric	RLVR	SFT (illustrative)	Description
Relative Spectral Residual (ρ_Σ)	~3%	~35%	Portion of checkpoint change attributable to spectral change.
Spectral Distance (δ_Σ)	~10-2%	Orders of magnitude higher	Distance to the fixed-spectrum family of the base model.
Spectrum-Changing Energy (κ_spec)	~1.0 – 1.4	89 – 1364	Normalized concentration of updates in spectrum-changing coordinates (1.0 is isotropic).

Functionally, restoring the base model's spectrum to an RLVR-trained model while keeping its learned frames preserves most of the performance gains.
Training with a fixed spectrum from the start still allows for strong performance gains, while a control that only updates the spectrum fails to achieve comparable results.

Key Finding 2: Frame Adaptability:

A low-residual reconstruction of the learned checkpoint requires both singular frames (input and output) to be adaptable; fixing either one results in a poor approximation of the final weights.

Fixing the spectrum but adapting both frames (ISO's approach) leaves a median unexplained-update residual of only 1.8%.
Remixing only within the base model's input and output subspaces leaves a median residual of 87%.
Retaining the incoming output subspace or input subspace leaves median residuals of 45% and 42%, respectively.
This pattern holds even when starting a new RL stage from an already RL-trained checkpoint with a different objective.

Offline Merging (ISO-Merger):

ISO-Merger is a data-free method for combining specialist models trained from a shared base.
It operates by combining the singular-frame changes of the specialists while reusing the shared base spectrum.
The method requires no post-merge data, rollouts, gradient updates, or on-policy distillation (OPD).
It is reported to achieve the strongest aggregate performance among the compared data-free merging methods.

What's new

The paper introduces "spectral inheritance" as a core principle of RLVR optimization, showing that models can reuse their base weight spectra while learning new behaviors. It operationalizes this finding into a novel, RLVR-native optimization framework, Isospectral Optimization (ISO), which parameterizes updates in the space of singular frames while keeping the spectrum fixed. This provides a new coordinate system for both online optimization and offline model merging.

Conclusion

The authors conclude that RLVR optimization should be designed around the observed structure of reward-driven adaptation. Instead of inheriting pre-training optimizers, post-training should inherit the base model's weight spectrum and optimize the singular frames. The proposed Isospectral Optimization (ISO) framework, with its offline ISO-Merger and online ISO-Optimizer instantiations, provides a concrete and effective implementation of this principle, leading to improved sample efficiency and performance in RLVR tasks.

Read the source →

Don't read this site daily. Get it in your inbox.

The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.