HuggingFace Papers
★ 9/10 signal
xHC: Expanded Hyper-Connections
infrastructuremodelsresearch
Summary
This paper introduces xHC (Expanded Hyper-Connections), a method for scaling the number of parallel residual streams in Transformers beyond the typical limit of N=4. Prior methods like Manifold-Constrained HC (mHC) face diminishing returns and cubic cost scaling. xHC addresses this with two techniques: temporal feature augmentation for richer write-back signals and a sparse architecture that updates only k=4 of N=16 streams, controlling cost. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC. Scaling-law experiments show that baselines require up to 1.5x the compute of xHC to reach the same training loss, establishing large-N residual-stream expansion as a practical scaling axis.
Problem
Hyper-Connections (HC) expand the Transformer residual stream into N parallel streams, offering a promising scaling axis beyond model width and depth. However, existing methods like Manifold-Constrained HC (mHC) show diminishing performance gains and rapidly increasing training costs when scaling N beyond 4. This is attributed to two bottlenecks: an information bottleneck, where a single layer output provides insufficient write-back information for a growing number of streams, and a computational bottleneck, where the cost of generating the residual mixing matrix scales cubically with N.
Method
The proposed method, xHC (Expanded Hyper-Connections), introduces two coordinated designs to enable scaling to N=16 streams. To address the information bottleneck, it uses temporal feature augmentation via multi-scale causal convolutions to enrich the write-back signal. To address the cost bottleneck, it employs a sparse residual-stream architecture that reads from all N streams (dense read) but updates only a subset of k=4 active streams (sparse write), reducing the dominant cost from O(N^3*C) to O(k^3*C). The method was evaluated by pre-training 18B and 28B MoE models and comparing them against vanilla and mHC baselines on a suite of downstream benchmarks.
Details
Main Results on 18B and 28B MoE Models:
xHC (N=16, k=4) was compared against a vanilla baseline and mHC (N=4) with comparable training FLOPs. xHC added 4.1% FLOPs over vanilla at 18B and 3.0% at 28B.
| Model | Method | MMLU | GSM8K | HumanEval | Average Score |
|---|---|---|---|---|---|
| 18B | Vanilla | 48.9 | 37.7 | 25.6 | 40.6 |
| mHC | 54.7 | 44.5 | 23.2 | 44.8 | |
| xHC | 57.2 | 48.4 | 29.3 | 48.8 | |
| 28B | Vanilla | 54.6 | 50.3 | 27.4 | 47.8 |
| mHC | 56.8 | 56.3 | 26.8 | 50.5 | |
| xHC | 60.5 | 59.2 | 31.1 | 53.6 |
Scaling Laws and Compute Efficiency:
- Scaling law experiments were fit on models trained with compute ranging from 1.7e19 to 4.0e20 FLOPs.
- xHC traces a consistently lower loss curve than both mHC and the vanilla baseline across the measured compute range.
- To reach the same loss as xHC, the vanilla baseline requires 1.50x the compute, and mHC requires 1.19x the compute.
Expansion Rate Scaling (N-sweep on 2.5B MoE):
- Increasing N from 4 to 16 in mHC reduces loss by only 0.006 while increasing training FLOPs by 32%.
- Increasing N from 4 to 16 in xHC reduces loss by 0.012 with only a 4% increase in training FLOPs.
Ablation Study (10B MoE model):
| Configuration | Validation Loss | FLOPs Overhead |
|---|---|---|
| Vanilla baseline | 2.029 | - |
| mHC (N=16, dense) | 1.998 | +18.8% |
| mHC (N=16) + Temp Aug | 1.984 | +20.1% |
| xHC (N=16, k=4, full) | 1.983 | +3.3% |
| xHC w/o Dense Read | 1.985 | +3.3% |
| xHC w/o Fixed Streams | 1.986 | +3.3% |
| xHC w/ Softmax Router | 1.988 | +3.3% |
- The ablation shows that temporal augmentation improves performance at large N, and the sparse architecture makes it affordable. Dense read and fixed streams are critical for stability and performance.
Compatibility with Muon Optimizer (18B MoE):
- When paired with the Muon optimizer (instead of AdamW), xHC maintained its advantage.
- The Muon + xHC model achieved an average downstream score of 49.9, a significant improvement over the Muon baseline's score of 43.1.
Efficient Deployment with xHC-Flash:
- To manage memory traffic, the paper introduces xHC-Flash.
- It reduces the per-sublayer memory traffic from 73.5C for full xHC to 40C, which is comparable to the 34C required by mHC at N=4, while retaining most of the performance gains.
What's new
This paper provides a diagnosis of why prior Hyper-Connection (HC) methods fail to scale beyond N=4 residual streams, identifying specific information supply and computational cost bottlenecks. It introduces xHC, the first HC-family method designed to be effective and affordable at large expansion rates (e.g., N=16). The novel contributions are the combination of temporal feature augmentation to enrich write-back signals and a sparse residual-stream architecture (dense read, sparse write) to manage the cubic scaling cost.
Conclusion
The authors conclude that the identified information and cost bottlenecks explain why scaling the number of residual streams in prior work yields diminishing returns. The proposed xHC method addresses both bottlenecks through temporal feature augmentation and a sparse residual architecture, making large-N expansion a practical and compute-efficient scaling axis for LLMs. This is supported by consistent performance gains on 18B and 28B MoE models, favorable scaling laws, and the development of xHC-Flash for practical memory-efficient deployment.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.