Aleph Alpha has released Kolibri, an open-weight Combination-of-Specialists (MoE) language mannequin constructed for German and English. Kolibri has 78.1B whole parameters however prompts solely 3.46B, or 4.4%, per token. It accepts as much as 1,048,576 tokens of context, lets customers set reasoning effort per request, and ships below the Apache 2.0 license on Hugging Face. The goal is sovereign deployment in regulated sectors similar to public administration, trade and aerospace.
Is it deployable? Sure. The FP8 checkpoint is about 78GB and runs on a single B200, B300 or H200, or on 2 H100 SXM5 GPUs, served via vLLM with devoted Kolibri reasoning and tool-call parsers.
What’s Kolibri?
Kolibri (Kolibri-1) is a bilingual English-German MoE transformer developed finish to finish by groups in Germany. In line with the technical research report, Aleph Alpha crew managed the complete pipeline: knowledge, structure, coaching infrastructure, post-training and analysis. Coaching ran on infrastructure in Germany and Finland. The design targets the EU General-Purpose AI Code of Practice, the EU AI Act and GDPR. Aleph Alpha is a signatory of that Code, and its knowledge pipeline redacts private knowledge earlier than coaching.
Structure: Sparse Specialists and Hybrid Consideration
Kolibri stacks 50 transformer blocks with a mannequin width of two,560. Each MoE layer scores all 384 routed specialists with a sigmoid router, sends every token to the highest 6, and all the time runs 1 shared skilled. Professional load is balanced with Precise Quantile Balancing and Load-Error Injection.
Consideration makes use of grouped-query consideration with 48 question heads and 4 KV heads. Each fifth block makes use of full consideration with out positional encoding. The opposite 40 blocks use sliding-window consideration over the 512 previous tokens, with RoPE. Sliding-window layers maintain a fixed-size KV cache, so solely 10 layers develop with context size. At matched compute, Aleph Alpha crew stories the hybrid helps sequences 4 occasions longer than a full-attention mannequin.
A Tokenizer Constructed for German
The 128,000-token vocabulary is educated with UniBPE, which builds merges like BPE however scores every merge by Unigram loss. On German textual content it reaches 4.90 bytes per token, versus 4.35 for the GPT-5 tokenizer. Meaning 11.2% fewer tokens on German net textual content. In English, Kolibri reaches 4.58 bytes per token in opposition to 4.67 for GPT-5.
Coaching: 24T Tokens, Then SFT and RL
Pre-training coated 20T tokens on 768 NVIDIA B200 GPUs, adopted by 3.44T mid-training tokens at 65,536 sequence size. A 201B-token long-context stage then educated on 262,144-token sequences. Aleph Alpha added greater than 2T German tokens it curated from the net or generated synthetically. Put up-training mixed supervised fine-tuning, blended with MergeMix, with reinforcement studying on greater than 1.2M inner duties. The Merlin-Arthur protocol trains the mannequin to abstain when retrieved context doesn’t help a solution.
Interactive Explainer: How Kolibri Works
window.addEventListener(“message”,perform(e){if(e.knowledge&&e.knowledge.kolibriH){var f=doc.getElementById(“kolibri-xp”);if(f)f.model.peak=e.knowledge.kolibriH+”px”;}});
Benchmarks
Aleph Alpha crew evaluated each mannequin with the identical eval-framework setup. In English, Kolibri leads GPQA Diamond (84.3), AIME 2025 (96.9) and AIME 2026 (96.0). It ties Qwen3.5 35B-A3B on the English agentic common at 63.4. It trails on BFCL v4, scoring 61.4 in opposition to 70.5 for Qwen3.5. The dense Qwen3.8 27B scores larger total (80.2 EN, 79.9 DE), however prompts about 8 occasions extra parameters per token. Towards its inner predecessor Kolibri Origin, Kolibri decodes about 2.7 occasions extra textual content per GPU whereas scoring 21.4 factors larger in English.
Kolibri vs Closest Opponents
| Characteristic | Kolibri-1 | Qwen3.6 35B-A3B | Nemotron 3 Tremendous | Mistral Small 4 |
|---|---|---|---|---|
| Developer | Aleph Alpha (Germany) | Alibaba Qwen | NVIDIA | Mistral AI (France) |
| Complete / energetic params | 78.1B / 3.46B | ~35B / 3B | 120B / 12B | 119B / 6.5B |
| Structure | MoE, sliding-window + full consideration | MoE, Gated DeltaNet hybrid | Mamba-2 + MoE + consideration | MoE |
| Max context | 1,048,576 tokens | 262,144 native, ~1M with YaRN | 1M tokens | 256k tokens |
| Reasoning management | none / low / medium / excessive | Considering on / off | On / off, low-effort mode | none / excessive |
| License | Apache 2.0 | Apache 2.0 | NVIDIA Nemotron Open Mannequin License | Apache 2.0 |
| General rating (EN / DE) | 75.5 / 70.8 | 71.4 / 67.3 | 73.0 / 67.9 | 63.1 / 61.4 |
| Agentic avg (EN) | 63.4 | 62.1 | 54.9 | 40.7 |
| Business RAG avg (DE) | 67.5 | 65.8 | 57.6 | 53.4 |
| Code avg (EN) | 89.3 | 87.7 | 88.3 | 82.0 |
Specs: mannequin playing cards for Kolibri-1, Qwen3.6-35B-A3B, Nemotron 3 Super and Mistral Small 4.
How you can Deploy Kolibri
Set up the aleph-alpha-inference bundle, then serve with vLLM:
pip set up 'aleph-alpha-inference>=1'
vllm serve Aleph-Alpha/Kolibri-1 --kv-cache-dtype fp8
--reasoning-parser kolibri1 --tool-call-parser kolibri1
--enable-auto-tool-choice
The default context is 262,144 tokens. Cross --max-model-len 1048576 with a max_position_embeddings override for the complete 1M window. Set reasoning_effort via chat_template_kwargs. Aleph Alpha recommends temperature 1.0, top-p 0.97 and top-k 128.
Key Takeaways
- 78.1B whole, 3.46B energetic: every token makes use of 6 of 384 routed specialists plus 1 shared skilled.
- 40 sliding-window and 10 full-attention layers preserve a 1M-token context inexpensive.
- Skilled on 24T tokens, with German above 20% of the combo.
- Prime General rating in English (75.5) and German (70.8) among the many 12 MoE fashions Aleph Alpha in contrast.
- Apache 2.0 FP8 weights, serveable on a single B200 or H200 by way of vLLM.
Try the Model Weights and Technical Report. All credit score goes to the researcher of this challenge. Additionally, be at liberty to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Must accomplice with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so forth.? Connect with us
The submit Aleph Alpha Releases Kolibri: A 78.1B Open-Weight English-German MoE Model With Only 3.46B Active Parameters appeared first on MarkTechPost.

