How can you teach a model to do tasks which it has difficulty with, but without having them repeat the same mistakes over and over again? A team of researchers from Google Cloud AI Research and UCLA have released a training framework, ‘Supervised Reinforcement Learning’ (SRL)This allows 7B models to learn by analyzing very complex math problems and agents’ trajectories, which are not possible with normal supervised learning and reinforcement-based learning.
The teacher trace of small open-source models, such as Qwen2.57B Instruct, is not good enough to solve the most difficult problems on s1K1.1. When we use supervised fine-tuning on full DeepSeek R1 solutions, our model will imitate token by token. The sequence is too long and there are only 1,000 data items. Our final scores fall below those of the base model.
Core idea of ‘Supervised Reinforcement Learning’ SRL
‘Supervised Reinforcement Learning’ (SRL) keeps the RL style optimization, but it injects supervision into the reward channel instead of into the loss. Every expert trajectory in s1K version 1.1 has been parsed to a series of actions. Each prefix is a separate training example. The model produces the private reasoning span first. It then outputs an action that corresponds to the step. This action, in turn, is only compared against the teacher’s action by using a similarity measure based on difflib. Every step is rewarded, even if the answer to the question is incorrect. Because the reasoning portion of the text is unconstrained, the model has the freedom to search through its own chain, without needing to copy any teacher tokens.
Mathematics results
To ensure that comparisons are accurate, all models have been initialized with Qwen2.5/7B Instruct. They are also trained using the DeepSeek R1 formatted s1K 1.1 sets. In Table 1, the exact numbers are as follows:
- AMC23 greedy 50, AIME24, AIME25, AIME26, AIME27, AIME28, AIME29, AIME30, AIME31, AMC23, AIME24, AIME25, AIME29, AIME30, AIME30, AMC23, AIME24, AIME25, AIME30, AIME24, AIME25, AIME29, AIME30, AIME30, AIME29, AIME30, AIME25, AIME25
- AMC23 is greedy at 50.0. AIME24 and AIME25 are greedy at 16.7.
- AMC23 greedy 57,5, AIME24 greedy 20,0, AIME25 greedy 10.

The key is that SRL by itself removes SFT degradation, raises AIME24, AIME25 and, when RLVR runs after SRL, it achieves the highest open-source scores. This research team has stated that the pipeline should be SRL, then RLVR and not SRL by itself.
The results of software engineering
Researchers also apply SRL to Qwen2.5 Coder 7B Instruct, using 5,000 verified agents trajectories produced by claude 3-7 sonnet. Every trajectory is then decomposed and 134,000 items of step are generated. The evaluation is done on the SWE Bench Verified. In the Oracle file editing mode, base models get 5.8% and 3.2% respectively. SWE Gym7B receives 8.4 percent, and 4.2 per cent. SRL receives 14.8 and 8.6 percent. This is more than twice the baseline model, and higher than SFT’s.

The Key Takeaways
- SRL reformulates difficult reasoning by generating a stepwise action. First, it produces an internal monologue and then outputs one single action. Only that action will be rewarded with sequence similarity.
- SRL uses the DeepSeek R1 formatted s1K 1.1 s1K data of SFT and RLVR. But unlike SFT, SRL does not overfit for long demonstrations and, unlike RLVR does not collapse if no rollouts are correct.
- For maths, it is best to initialize the Qwen2.57B Instruct using SRL and then use RLVR. Both methods will push reasoning benchmarks above either alone.
- SWE Bench verified is a step above the baseline Qwen2.5 coder 7B instruct and SFT-style SWE Gym7B.
- This SRL is easier to use on smaller datasets than other step-wise RL methods, which require an additional reward model. It uses actions only from expert trajectories, and a light string similarity.
‘Supervised Reinforcement Learning’ (SRL) is a practical contribution by the research team. The GRPO-style reinforcement learning is retained, but the fragile outcomes are replaced with supervised stepwise rewards, which are calculated directly from expert trajectories. This ensures that the model receives an informative signal even when in D.As a result, it is hard to imagine how you could possibly get along with someone else. RLVR regimes where SFT and RLVR both stall. This is why it is crucial that researchers show the SWE Bench Verified and SRL with the exact same recipe. They should also make sure that SRL and RLVR are the most effective configurations. The open model can learn difficult tasks using SRL. SRL can be used to bridge the gap between process control and RL.
Take a look at the Paper. Check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.
Asif Razzaq serves as the CEO at Marktechpost Media Inc. As an entrepreneur, Asif has a passion for harnessing Artificial Intelligence’s potential to benefit society. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. This platform has over 2,000,000 monthly views which shows its popularity.

