Kyutai has launched Voice of Motive, 2 open-weight speech-to-speech fashions that remedy math issues out loud. Each begin from GLM-4-Voice-9B and add supervised fine-tuning (SFT) and reinforcement studying (RL). There isn’t a transcription step and no separate textual content LLM within the loop. On spoken GSM8K, accuracy climbs from 27.3% for the bottom mannequin to 77.1%.
Is it deployable? Sure, for self-hosting. Kyutai ran each BF16 checkpoints on a single H100. You additionally want the GLM-4-Voice repo for its speech tokenizer and decoder. Weights inherit the GLM-4-Voice license, and no Hugging Face inference supplier hosts them but.
Why Speech Fashions Lag on Math
Cascaded pipelines (speech-to-text, textual content LLM, text-to-speech) nonetheless lead on reasoning. Nevertheless, every stage provides latency, and the pipeline loses paralinguistic cues like tone. Speech-native fashions should emit audio at common intervals to remain interactive. That limits what number of hidden reasoning tokens they will afford.
Base GLM-4-Voice scores 27.3% on GSM8K. The sooner STITCH methodology raised that to 58.7% by including reasoning chunks. The analysis group name their work the primary software of RL to math reasoning in speech-native fashions.
How the Coaching Works
GLM-4-Voice interleaves its output: 13 textual content tokens, then 26 audio tokens, repeating.
- Stage-1 SFT: Coaching makes use of 150,616 issues from Orca-Math. Qwen3-235B rewrote every downside for speech. Kyutai’s DSM TTS then voiced them in lots of voices. SFT alone lifts GLM-4-Voice from 27.3% to 61.7%.
- Stage-2 RL: For every spoken query, the mannequin samples 4 replies at temperature 0.9. A decide, Qwen3-235B-A22B-2507, scores the decoded textual content with a binary reward. The decide by no means sees the reference reply. On 100 hand-checked instances, it agreed with people 88% of the time.
Rewards are centered inside every group, forming a group-relative REINFORCE goal. It’s associated to GRPO however drops PPO clipping and KL regularization. Coaching ran on 16 H100 GPUs, with 1,500 RL updates.
2 design decisions matter most:
- Temperature correction: Logits are divided by the sampling temperature earlier than the log-softmax within the loss. With out it, GSM8K collapsed from 65.5% to 12.3%.
- Audio-token merging: At every audio place, all audio-vocabulary possibilities are summed into 1 summary token. The loss asks solely whether or not audio got here subsequent, not which audio token. The paper proves this estimator is unbiased and lower-variance below a value-invariance assumption.
Interactive Explainer
‘;d.appendChild(lab);d.appendChild(pr);});
q(‘#kvr-lkey’).fashion.background=repair?’#7DB7FF’:’#FF6B6B’;
q(‘#kvr-tv’).textContent=”Mismatch (total variation): “+(television*100).toFixed(1)+’%’;
q(‘#kvr-tnote’).textContent=repair?’Logits are divided by T earlier than the log-softmax within the RL loss, so the gradient is computed for the precise coverage that produced the samples.’:’Samples are drawn from softmax(z / T) however scored below softmax(z). The gradient estimator not matches the sampler. Within the paper this collapsed GSM8K from 65.5% to 12.3%.’;
resize();}
perform drawAbl(){var rows=[[‘Default setup’,’merged audio, correction on, group 4′,65.5,’65.5 u00b1 1.1′,’ours’],[‘Group size 8′,”,67.0,’67.0’,”],[‘Loss on all tokens’,”,64.4,’64.4′,”],[‘Loss on text tokens only’,”,63.8,’63.8′,”],[‘No temperature correction’,”,12.3,’12.3′,fix?”:’hot’]];paint(q(‘#kvr-abl’),rows);}
q(‘#kvr-T’).addEventListener(‘enter’,temp);
seg(‘kvr-fix’,perform(b){repair=b.getAttribute(‘data-f’)===’1′;temp();drawAbl();});
temp();
/* slide 4: outcomes */
var dec=”paper”;
perform drawRes(){var P=dec===’paper’;var rows=[
[‘PersonaPlex’,’8B, full-duplex’,3.2,’3.2′,”],
[‘GLM-4-Voice (base)’,’9B’,27.3,’27.3′,”],
[‘STITCH (Chiang et al.)’,’9B, reasoning chunks’,58.7,’58.7′,”],
[‘Voice of Reason’,’9B, direct answers’,P?65.5:70.3,P?’65.5 u00b1 1.1′:’70.3′,’ours’],
[‘Voice of Reason (Stitch)’,’9B, silent reasoning’,P?74.8:77.1,P?’74.8 u00b1 1.1′:’77.1′,’ours’],
‘Prime traces, not matched on dimension or structure’,
[‘Qwen2.5-Omni’,’7B, text output’,84.7,’84.7′,”],
[‘Qwen3-Omni’,’30B, text output’,94.6,’94.6′,”],
[‘Cascaded ASR, LLM, TTS, ASR’,’Gemma-4-31B-IT + Kokoro’,95.7,’95.7′,”]];paint(q(‘#kvr-res’),rows);}
perform paint(el,rows){el.innerHTML=”;var fills=[];rows.forEach(perform(r){if(typeof r===’string’){var s=doc.createElement(‘div’);s.className=”kvr-sep”;s.textContent=r;el.appendChild(s);return;}
var n=doc.createElement(‘div’);n.className=”kvr-rname”;n.innerHTML=r[0]+(r[1]?’‘+r[1]+’‘:”);
var tr=doc.createElement(‘div’);tr.className=”kvr-track”;var f=doc.createElement(‘div’);f.className=”kvr-fill “+r[4];tr.appendChild(f);
var v=doc.createElement(‘div’);v.className=”kvr-val”;v.textContent=r[3]+’%’;var row=doc.createElement(‘div’);row.className=”kvr-rrow”;row.appendChild(n);row.appendChild(v);row.appendChild(tr);el.appendChild(row);fills.push([f,r[2]]);});
requestAnimationFrame(perform(){requestAnimationFrame(perform(){fills.forEach(perform(x){x[0].fashion.width=x[1]+’%’;});});});resize();}
seg(‘kvr-dec’,perform(b){dec=b.getAttribute(‘data-d’);drawRes();});
})();
