The constant modification of algorithms is the hallmark of agentic reinforcement learning. New estimation methods, pipeline stages and rollout plans. Each change is woven through the layers of distributed backends, trainers, and glue for rollouts in mainstream frameworks. The researcher pays for this cost at each iteration.
MoltThe NeMo Team at NVIDIA has developed targets which cost money directly. This is a PyTorch native agentic RL Framework with an uncommon design goal. The codebase must be compact for researchers to keep in their heads, as well as for AI coding assistants to read it and understand its reasoning. According to the stated footprint, there are approximately 8.6K lines RL code. This is measured by tracing an import graph starting from each framework’s RL entry. The same method accounts for 62K lines of verl code, 25K lines for slime and 7.2K line for OpenRLHF.
Is it deployable?
Yes. Molt is shipped under Apache 2.0, with Slurm Scripts and a container. The research paper, however, positions Molt as research infrastructure and not production training, with hardware being the key. Shipped recipes use 2 nodes each with 8 H100 graphics cards, 8 split for training and eight for rolling out.
This puts Molt within reach of labs at the frontiers and adjacent to them, as well as startups that are doing post-training. It also makes it available for enterprise AI groups working in robotics, finance, healthcare and AI who train agents in proprietary environments. The applications include code-execution, multi-turn tools, vision-language environments, LLM reward loops and policy distillation on a smaller student.
Three parts, one loop
Molt composes Ray For placement and asynchronous queuing vLLM NVIDIA’s AutoModel and FSDP2 are used to train, while NVIDIA is for rolling out. The three are not forked. Upstream improvements come as container pins rather than being rebased.
Runtime consists of an agentpool, a collection of vLLM-engines behind a request router, and one trainable policy actor. The streaming pool ensures that the engines are never drained while an actor is training. Partially rolling out pauses engines. It broadcasts actor shards to all the engines over NCCL and resumes requests that were previously retained.
Agent is a normal program
A RL run names a Python module which exports. AgentRunner. The reward is also code.
There are two forms of support. Two forms are supported. EnvThe LLM loop is owned by the framework in a Gymnasium aligned Step(). As With ChatAgentThe user can own the loop by using an OpenAI SDK or Anthropic SDK. Molt launches both wire and loopback servers, decoding each request at the server. If a agent with a longer horizon compacts their context, rewrites prefixes it and then reopens a new segment automatically.
Do not train using a token which you didn’t generate
The design is organized around three correctness invariants. Token identity – sampled tokens ids are used to define the path, and not just a retokenized version of the transcript. Semantics of the policy-version: Trainable tokens retain their log-probabilities for behavior and policies, while asynchronous usage is corrected by token behind an sequence-level gate. Consistency in the future: both rollout and actor agree upon model semantics.
In policies that mix experts, it is important to consider the final invariant. Both the rollout and training routes select experts separately, but small numerical changes can cause top-k decisions to change. Molt applies rollout routing replayvLLM will return its expert ids per token and then the forward training replays these.Function Tok (Mode)
Interactive explainer
/* —- panel 3 : benchmark —- */
var BM={step:{a:119.4,as:2.3,b:109.5,bs:10.3,max:145,u:’ s’,c:’Mean u00b1 s.d. of steady-state wall time per optimizer step, three independent runs per configuration’},tok:{a:461,as:0,b:502,bs:0,max:620,u:’ tok/GPU/s’,c:’Generated tokens per GPU-second on a common basis of about 880K tokens per step across 16 GPUs’}};
function bench(k){
var d=BM[k],W=560,x0=150;
var w1=d.a/d.max*W,w2=d.b/d.max*W;
anim($(‘mxbar1’),w1);anim($(‘mxbar2’),w2);
$(‘mxval1′).textContent=d.a+(d.as?’ u00b1 ‘+d.as:”)+d.u;
$(‘mxval2′).textContent=d.b+(d.bs?’ u00b1 ‘+d.bs:”)+d.u;
$(‘mxval1’).setAttribute(‘x’,x0+w1+8);$(‘mxval1’).setAttribute(‘y’,60);
$(‘mxval2’).setAttribute(‘x’,x0+w2+8);$(‘mxval2’).setAttribute(‘y’,130);
$(‘mxbench-cap’).textContent=d.c;
err($(‘mxerr1′),x0+w1,55,d.as/d.max*W,’#76B900’,d.as);
err($(‘mxerr2′),x0+w2,125,d.bs/d.max*W,’#4a8fd6’,d.bs);
$(‘mxm-step’).classList.toggle(‘on’,k===’step’);
$(‘mxm-tok’).classList.toggle(‘on’,k===’tok’);
setTimeout(sz,40);
}
function err(g,cx,cy,half,col,v){
if(!v){g.setAttribute(‘opacity’,’0′);g.innerHTML=”;return}
g.setAttribute(‘opacity’,’1′);
g.innerHTML=’
‘
‘
}
function anim(el,to){
var from=parseFloat(el.getAttribute(‘width’))||0,s=null;
function f(ts){if(!s)s=ts;var p=Math.min(1,(ts-s)/500);el.setAttribute(‘width’,from+(to-from)*(1-Math.pow(1-p,3)));if(p’;
}
if(n>64){s+=’
$(‘mxexperts’).innerHTML=s;
setTimeout(sz,40);
}
$(‘mxepsl’).addEventListener(‘input’,scale);scale();
var spec=false,off=false;
$(‘mxspec’).addEventListener(‘click’,function(){
spec=!spec;this.classList.toggle(‘on’,spec);
$(‘mxgen’).textContent=spec?’64 s’:’329 s’;
$(‘mxgend’).textContent=spec?’MTP head enabled: the recipe moves from generation-bound to training-bound’:’Baseline, Qwen3.6-35B-A3B multimodal MoE recipe’;
});
$(‘mxoff’).addEventListener(‘click’,function(){
off=!off;this.classList.toggle(‘on’,off);
$(‘mxmem’).textContent=off?’46.4 GB’:’64.7 GB’;
$(‘mxmemd’).textContent=off?Adam states in host-memory; Policy_Train rises from 213s to 251s (+18 %)’:’Adam States on Device’);
});
/* —- auto-resize —- */
function sz(){
try{
var h=(document.body?document.body.offsetHeight:R.offsetHeight)+40;
if(window.parent&&window.parent!==window){window.parent.postMessage({mtpMoltHeight:h},’*’)}
}catch(e){}
}
window.addEventListener(‘load’,sz);setTimeout(sz,300);setTimeout(sz,1200);
if(window.ResizeObserver){new ResizeObserver(sz).observe(R)}
})();
You can partner with us to promote your GitHub Repository OR Hugging Page OR New Product Launch OR Webinar, etc.? Connect with us

