Human supervision is required to define desired behaviors in post-training methods of language models that have been pre-trained. This can be done through either demonstrations or feedback on preferences. This approach is limited by the complexity of tasks and behaviors. The human supervisor is not reliable in such scenarios, as LMs can mimic errors in demos and exploit feedback system flaws. In order to overcome this challenge, LMs must be trained for tasks which are beyond the human ability in terms of reliability. Recently, research has revealed a variety of failure modes including the reward-hacking or human-designed signals for supervision and real humans.
LLM post-training: Limitations in Human Supervision
Researchers have investigated several ways to increase scale without human supervision. Standard methods include high-quality rewards that can be verified, like matching outputs of models with real-world solutions. While there is evidence to suggest that the pre-trained models are capable of performing downstream tasks with minimal post-training, the challenge remains in eliciting latent knowledge. Contrast Consistent Search is a method of unsupervised elicitation that relies on logical consistency in order to discover latent knowledge. CCS, however, underperforms the supervised approach and is often unable to identify latent knowledge as other prominent features satisfy consistency properties.
Introduction to Internal Coherence Maximization
Internal Coherence Maximization is a method that researchers from Anthropic have developed. It allows them to fine tune pre-trained model on labels they generate themselves, without any labels provided. ICM finds label sets that satisfy the requirements of both the pre-trained and logically consistent model. ICM’s simulated-annealing search algorithm approximates the maximum goal, since optimal label set recognition is computationally impossible. The method also matches training with golden labels for TruthfulQA or GSM8K. It even outperforms the crowdsourced labels used by Alpaca.
What is the ICM Algorithm?
ICM follows a three-step iterative process. (a) The system selects a sample of a newly unlabeled instance from the dataset to be considered for inclusion. (b) It determines an optimal label while also resolving logical inconsistencies. (c) Finally, the algorithm decides whether or not it accepts this newly labeled case based on a scoring function. ICM’s performance is assessed across three datasets, including TruthfulQA, which assesses truthfulness, GSM8K, which verifies mathematical accuracy, and Alpaca, which measures helpfulness and harmlessness. The researchers tested four baselines, including Zero-shot, Golden Label and Human Label. The Experiments also used Llama 3.1, 8B, 70B models as well as two proprietary models: Claude 3.5 Haiku, and Claude 3.4 Haiku.
Benchmark performance and model comparisons
ICM is more accurate than humans in superhuman capabilities elicitation. It matches the golden standard of supervision at 80%. Researchers successfully trained a chatbot assistant without the need for human supervision using ICM reward models. Unsupervised reward models achieve 75.0% accuracy in RewardBench compared with 72.2% when trained by humans using production data. Using both unsupervised and human supervised RMs, two policies were trained using RL in order to produce helpful, innocent, and honest Assistants. A policy that is trained using the unsupervised RM has a win rate of 60%. These policies are still behind the previously released Claude 3.5 Haiku which has a 92% success rate.
The Future Outlook
In this paper, we introduce Internal Coherence Maximization as an advance in unsupervised LM to fine-tune pre-trained models for self-generated labels. It consistently outperforms human supervision, including crowdsourced, in GSM8K Verification, TruthfulQA Tasks, and Alpaca Reward Modeling. ICM has limitations, including a dependency on the salience of concepts within pre-trained model and an inability to handle long inputs because context window restrictions. ICM is a promising alternative to traditional RLHF as LMs progress beyond the human evaluation capability. It ensures model alignment with intent, without any human supervision.
Click here to find out more Paper. The researchers are the sole owners of all credit. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe Now our Newsletter.
