A Sovereign, Open-Source Foundation Model for German and English

context

What happened

The authors present SoofiS 30B-A3B, an open-source Mixture-of-Experts (MoE) hybrid Mamba-Transformer foundation model optimized for German and English. It activates 3B parameters per token and maintains a near-constant inference cache as context grows, providing high throughput for long-context, high-concurrency deployments. Pretrained on 2.7 trillion tokens with up-weighted German data, it outperforms existing European sovereign baselines and matches dense 14B-27B models on standard benchmarks, achieving top coding scores among open base models.

Why it matters

It showcases the real-world viability of hybrid Mamba-Transformer MoE models for high-throughput, long-context applications.

The take

The Mamba-Transformer hybrid architecture is highly promising for long-context applications because it bypasses the quadratic KV cache bottleneck of pure Transformers. While SoofiS is targeted at German/English sovereignty, the deployment characteristics of this hybrid MoE model are worth watching for anyone running high-throughput, long-context pipelines.

Do this

Watch the development of Mamba-Transformer hybrids if you are struggling with KV cache scaling costs in production.

Don't read this site daily. Get it in your inbox.

The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.