Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
  • Anthropic Shares Drop 8.5% in Second Session Throughout AI Selloff – Unite.AI
  • OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min
AI-trends.todayAI-trends.today
Home»Tech»Google AI Introduces Supervised Reinforcement Learning, a Framework that Uses Expert Trajectories and Step-Wise Frameworks for Teaching Small Language Models How to Solve Hard Problems

Google AI Introduces Supervised Reinforcement Learning, a Framework that Uses Expert Trajectories and Step-Wise Frameworks for Teaching Small Language Models How to Solve Hard Problems

Tech By Gavin Wallace01/11/20255 Mins Read
Facebook Twitter LinkedIn Email
A Coding Implementation to Build an Interactive Transcript and PDF
A Coding Implementation to Build an Interactive Transcript and PDF
Share
Facebook Twitter LinkedIn Email

How can you teach a model to do tasks which it has difficulty with, but without having them repeat the same mistakes over and over again? A team of researchers from Google Cloud AI Research and UCLA have released a training framework, ‘Supervised Reinforcement Learning’ (SRL)This allows 7B models to learn by analyzing very complex math problems and agents’ trajectories, which are not possible with normal supervised learning and reinforcement-based learning.

The teacher trace of small open-source models, such as Qwen2.57B Instruct, is not good enough to solve the most difficult problems on s1K1.1. When we use supervised fine-tuning on full DeepSeek R1 solutions, our model will imitate token by token. The sequence is too long and there are only 1,000 data items. Our final scores fall below those of the base model.

https://arxiv.org/pdf/2510.25992

Core idea of ‘Supervised Reinforcement Learning’ SRL

‘Supervised Reinforcement Learning’ (SRL) keeps the RL style optimization, but it injects supervision into the reward channel instead of into the loss. Every expert trajectory in s1K version 1.1 has been parsed to a series of actions. Each prefix is a separate training example. The model produces the private reasoning span first. … It then outputs an action that corresponds to the step. This action, in turn, is only compared against the teacher’s action by using a similarity measure based on difflib. Every step is rewarded, even if the answer to the question is incorrect. Because the reasoning portion of the text is unconstrained, the model has the freedom to search through its own chain, without needing to copy any teacher tokens.

Mathematics results

To ensure that comparisons are accurate, all models have been initialized with Qwen2.5/7B Instruct. They are also trained using the DeepSeek R1 formatted s1K 1.1 sets. In Table 1, the exact numbers are as follows:

  • AMC23 greedy 50, AIME24, AIME25, AIME26, AIME27, AIME28, AIME29, AIME30, AIME31, AMC23, AIME24, AIME25, AIME29, AIME30, AIME30, AMC23, AIME24, AIME25, AIME30, AIME24, AIME25, AIME29, AIME30, AIME30, AIME29, AIME30, AIME25, AIME25
  • AMC23 is greedy at 50.0. AIME24 and AIME25 are greedy at 16.7.
  • AMC23 greedy 57,5, AIME24 greedy 20,0, AIME25 greedy 10.
https://arxiv.org/pdf/2510.25992

The key is that SRL by itself removes SFT degradation, raises AIME24, AIME25 and, when RLVR runs after SRL, it achieves the highest open-source scores. This research team has stated that the pipeline should be SRL, then RLVR and not SRL by itself.

The results of software engineering

Researchers also apply SRL to Qwen2.5 Coder 7B Instruct, using 5,000 verified agents trajectories produced by claude 3-7 sonnet. Every trajectory is then decomposed and 134,000 items of step are generated. The evaluation is done on the SWE Bench Verified. In the Oracle file editing mode, base models get 5.8% and 3.2% respectively. SWE Gym7B receives 8.4 percent, and 4.2 per cent. SRL receives 14.8 and 8.6 percent. This is more than twice the baseline model, and higher than SFT’s.

https://arxiv.org/pdf/2510.25992

The Key Takeaways

  1. SRL reformulates difficult reasoning by generating a stepwise action. First, it produces an internal monologue and then outputs one single action. Only that action will be rewarded with sequence similarity.
  2. SRL uses the DeepSeek R1 formatted s1K 1.1 s1K data of SFT and RLVR. But unlike SFT, SRL does not overfit for long demonstrations and, unlike RLVR does not collapse if no rollouts are correct.
  3. For maths, it is best to initialize the Qwen2.57B Instruct using SRL and then use RLVR. Both methods will push reasoning benchmarks above either alone.
  4. SWE Bench verified is a step above the baseline Qwen2.5 coder 7B instruct and SFT-style SWE Gym7B.
  5. This SRL is easier to use on smaller datasets than other step-wise RL methods, which require an additional reward model. It uses actions only from expert trajectories, and a light string similarity.

‘Supervised Reinforcement Learning’ (SRL) is a practical contribution by the research team. The GRPO-style reinforcement learning is retained, but the fragile outcomes are replaced with supervised stepwise rewards, which are calculated directly from expert trajectories. This ensures that the model receives an informative signal even when in D.As a result, it is hard to imagine how you could possibly get along with someone else. RLVR regimes where SFT and RLVR both stall. This is why it is crucial that researchers show the SWE Bench Verified and SRL with the exact same recipe. They should also make sure that SRL and RLVR are the most effective configurations. The open model can learn difficult tasks using SRL. SRL can be used to bridge the gap between process control and RL.


Take a look at the Paper. Check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.


Asif Razzaq serves as the CEO at Marktechpost Media Inc. As an entrepreneur, Asif has a passion for harnessing Artificial Intelligence’s potential to benefit society. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. This platform has over 2,000,000 monthly views which shows its popularity.

🙌 Follow MARKTECHPOST: Add us as a preferred source on Google.

AI ar Google learning models work x
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

Microsoft, Meta, and Google Triple Up on AI Spending

Bloomberg Terminal is Getting a Makeover with AI, Whether You Like It Or Not

Google Gemini in Chrome: How to disable it

Meta’s Feel-Good AI Ad uses a song about the world ending

Your Online Trends Are All Wrong

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

Anthropic Shares Drop 8.5% in Second Session Throughout AI Selloff – Unite.AI

21/11/2026

DeepSeek V3.1: Why are people talking about it?

21/08/2025
Latest News

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.