” style=”width:100%;border:0;height:600px;display:block;overflow:hidden;” scrolling=”No,” loading=”lazy” title=”Bonsai 27B interactive explainer”>
The architecture below shapes the compression technique.
The Compression Process
Every weight has a unique code. There is only one FP16 scalar per 128-member group. Effective weight is w_i = s_g · t_i.
It is possible to carry a value in ternary form. log2(3) ≈ 1.585 bits. Additions of 128 weights per FP16 scale 16/128, giving ≈1.71 bits per weight. That is a ~9.4× reduction against FP16. Binary costs 1 + 16/128 = 1.125 bits, a ~14.2× reduction.
This representation is a continuous line that runs from the matrices to all of their components. These are the embeddings and attention projections. MLPs projections also appear. The normalization and scaling parameters are only marginally higher in precision.
Qwen3.627B is the true average. “4-bit” Build (Q4_K_XL), which is 5.2 bit per weight. The weight of the build (Q4_K_XL) is 5.2 bits. “2-bit” Build (IQ2_XXS), is 2.8. Bonsai is also a departure from BitNet. BitNet can only be avoided by pretraining.
It is obvious to ask what accuracy costs when compressed.
The Performance of a
PrismML evaluated fifteen benchmarks, in thinking mode using EvalScope on H100 graphics cards. Ternary Bonsai27B maintains 94.6% and 1-bit Bonsai27B retains 89.6% respectively of the FP16 benchmark.
| Variant | True bpw | Footprint | Think avg | Density (in g/kg) |
|---|---|---|---|---|
| Qwen3.6-27B FP16 | 16.0 | 54GB | 85.07 | 0.051 |
| Qwen3.6-27B Q4_K_XL (“4-bit”) | 5.2 | 17.6GB | 84.99 | 0.155 |
| Qwen3.6-27B IQ2_XXS (“2-bit”) | 2.8 | 9.4GB | 72.73 | 0.199 |
| Ternary Bonsai 28B | 1.71 | 5.9GB | 80.49 | 0.400 |
| Bonsai 1 Bit 27B | 1.125 | 3.9GB | 76.11 | 0.530 |
| Categories | FP16 | Ternary | 1-bit |
|---|---|---|---|
| It is a good idea to use the word “Me” instead of “mean”. | 95.33 | 93.40 | 91.66 |
| Coding | 88.74 | 85.96 | 81.88 |
| Understanding and Reasoning | 83.15 | 76.96 | 73.39 |
| The call of the Agentic Tool | 80.00 | 74.01 | 66.03 |
| Please follow the instructions below | 78.47 | 71.77 | 65.74 |
| Visualise the Future | 72.61 | 65.19 | 59.57 |
The failure of conventional sub-4 bit builds is different. LiveCodeBench and AIME26 both drop IQ2_XXS to 56.5 on LiveCodeBench. The benchmarks are still 88.93 for MMLU Redux. The pattern is repeated on the second model, Gemma-4-3B Q2_K_XL.
However, scores alone do not explain release. Memory is the key.
Memory as a Binding Constrained
Storage numbers are not the only thing to consider when fitting a new phone. iOS only allows a single program to use about half of the memory. The 12GB iPhone exposes approximately 6GB.
Second budget is KV cache. Only 16 of 64 layers carry a growing full-attention cache, so FP16 costs ≈64 KiB/token. A 262K window costs ≈17.2GB, and a 4-bit KV cache cuts that to ≈4.3GB.
The tolerance is then measured. Ternary Bonsai, a FP16 KV-based company, shows output-forward-KL of 0.0011 nanometers on MATH 500. Q4_K_XL displays 0.0146.
Peaks follow. Peaks follow. The derived Q4_K_XL row needs ≈25.6GB.
After a model is found to fit, the question of throughput arises.
Throughput and DSpark Speculative decoding
| The Platform | A Variant | tg128 | pp512 |
|---|---|---|---|
| M5 Max | The Binary System | 66.4 | 874 |
| M5 Pro | Ternary | 26.2 | 393 |
| iPhone 17 Pro Max | The Binary System | 11.0 | 111 |
| H100 (CUDA) | The Binary System | 104.8 | 2755 |
Generation is memory-bandwidth-bound, so fewer bytes per step means more tokens per second. Prefill has a lower gain because it is compute bound.
PrismML also includes a DSpark drafting system that has been trained to use the Bonsai 27B. On an H100 at draft depth k=4, the binary target reaches accepted length τ=3.6. That is 143.8 tok/s, a 1.37× speedup. Verification is lossless, so output stays distribution-identical. Apple Silicon’s drafter defaults to batch size 1, so output is distribution-identical.
Run It
Demo repo default is Ternary27B. You can start the server or directly generate:
./scripts/start_llama_server.sh # OpenAI-compatible API + chat/vision UI on :8080
./llama-cli -m ./Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf
--mmproj ./Ternary-Bonsai-27B-gguf/mmproj.gguf -c 0
-p "Explain KV cache growth."
mlx_lm.generate --model prism-ml/Ternary-Bonsai-27B-mlx-2bit
--prompt "Explain KV cache growth."
OpenAI tool calling is standard. You can also find out more about array:
curl http://localhost:8080/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
}
}]
}'
Return of the call The following are alternatives to[0].message.tool_calls. By default, the thinking mode is activated; thinking_budget_tokens It can be toggled on or off per your request.
Four deployment patterns are mapped onto this map.
Use Cases
Agents on laptops run the ternary builds for work involving full repository code over 262K tokens. The 1-bit build is run by phone-local reasoning; according to the whitepaper, 672 tokens are required for every 1% iPhone battery. By construction, privacy sensitive and offline workflows maintain prompts within the device. Combining the 4-bit KV Cache, one-GPU service fits 27B class quality onto a 24GB memory card.
What you need to know
- Bonsai 28B converts Qwen3.6 to 27B into binary and ternary weights. This is not a pretrain.
- The Ternary version retains 95.6% at 5.9GB, while the 1-bit version retains an 89.9% at 3.9GB.
- PrismML says the 27B class model is first to be able to fit into a smartphone.
- AIME LiveCodeBench & agentic tasks can cause the conventional sub-4 bit builds to fail.
- The latest version of Apache 2.0 is shipped with llama.cpp and MLX (CUDA/Metal).
Take a look at the Technical details, GitHub Repo The following are some examples of how to get started: Model Weight. Also, feel free to follow us on Twitter Join our Facebook group! 150k+ML SubReddit Subscribe Now our Newsletter. Wait! What? now you can join us on telegram as well.
Want to promote your GitHub repo, Hugging Face page, Product release or Webinar?? Connect with us

