HuggingFace Papers
8/10 signal
Loop the Loopies!
modelsresearchreasoning
Summary
The paper introduces Loopie, a series of looped Mixture-of-Experts (MoE) Transformer models with 20B (2B active) and 6B (0.6B active) parameters. It addresses the long-standing challenge where, for a fixed compute budget, increasing a model's parameter count has been more effective than looping a smaller model. Loopie is shown to outperform vanilla Transformer baselines trained with the same compute budget. The model achieves gold-medal performance at the 2025 IMO and IPhO without tools, demonstrating strong reasoning abilities from a novel post-training pipeline.
Problem
Looped Transformers have historically faced an efficiency challenge. Given an N-fold increase in pre-training compute, increasing a model's parameter count by a factor of N typically yields better performance than looping a smaller model N times. This work aims to develop a looped Transformer architecture that is more effective than vanilla Transformers under the same compute budget.
Method
The authors developed Loopie, a series of looped Mixture-of-Experts (MoE) models in two sizes: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. The models were evaluated against vanilla Transformer baselines trained with an equivalent compute budget. A novel post-training pipeline was applied to Loopie to enhance its reasoning capabilities.
Details
Performance and Architecture:
- Loopie achieves gold-medal performance at the 2025 International Mathematical Olympiad (IMO) and International Physics Olympiad (IPhO) without the use of external tools.
- Extensive ablation studies show that Loopie models substantially outperform vanilla Transformer baselines that are trained with the same compute budget.
- The evaluation included comparisons with a vanilla 30B-parameter MoE model with 3B active parameters (30B-A3B).
- The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B total parameter model with 2B active parameters, and a 6B total parameter model with 0.6B active parameters.
What's new
This work presents a looped Transformer architecture that, for a given pre-training compute budget, outperforms the standard approach of simply increasing a model's parameter count. This addresses a long-standing performance and efficiency challenge for this class of models.
Conclusion
The authors conclude that the Loopie models successfully address the historical compute-efficiency challenge of looped Transformers. By combining a looped MoE architecture with a novel post-training pipeline, Loopie demonstrates strong reasoning abilities, evidenced by its gold-medal performance on complex scientific reasoning tasks without reliance on external tools.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.