Specialized data is a resource that’s quietly disappearing. While the internet provided a seemingly infinite supply of text and images to train today’s generalist models, the next wave of AI breakthroughs — in cybersecurity, legal reasoning, healthcare, and other niche domains — requires data that simply doesn’t exist in sufficient volume, or can’t be accessed due to privacy concerns.
Researchers from Google and EPFL present SimulaThe framework is driven by reasoning and prioritizes transparency as well as fine-grained controls and scaling. Unlike conventional approaches, Simula doesn’t rely on seed data from the target distribution, hand-crafted prompts, or evolutionary algorithms — it constructs each dataset from first principles, treating data generation as a problem of mechanism design.
The Synthetic Data Generation Process is Not as Easy As It Seems
If you’ve worked with fine-tuning pipelines or domain-specific model training, you’ve likely run into the ‘not enough data’ wall. Manually collecting, annotating, and analyzing specialized data sets is time consuming, expensive, and prone to errors. But the obvious workaround — just prompt a large language model (LLM) to generate training data — runs into its own set of problems.
The researchers have defined three main axes for synthetic data optimization. ‘good’ data: quality, diversity, You can also find out more about the following: Complexity. The quality of a datapoint is determined by whether it meets certain semantic and syntactic criteria. Diversity is both global (does the data set include all concepts? Diversity includes both global coverage (does your collection include examples from across the entire concept space?) A given example’s complexity is measured in terms of how complicated, unusual, and confusing it may be. Simula’s unsolved problem is to simultaneously control all three at scale and with understandability.
Simula’s Workings: Meta-Prompts & Dual Critiques
Simula divides the creation process into There are four steps that you can control.The data properties are targeted by each of the.
It is important to note that the word “you” means “you”. The first step Addresses global diversity by using hierarchical taxesonomies. Given a dataset description — say, ‘a dataset of cybersecurity threat intelligence questions’ — a multi-modal model (referred to as M3) is prompted to identify the prime factors of variation for that domain (e.g., attack type, threat actor, vulnerability class). After each factor has been expanded, it is categorised into hierarchical trees. The system employs a Best of N proposal strategy in conjunction with a refinement stage that critiques the child nodes for their completeness, specificity, and soundness. It is important to note that the word “you” means “you”. resulting taxonomies function as structured sampling scaffolds — ensuring that when you draw 512,000 training examples, they genuinely cover the long tail of the domain rather than clustering around common modes.
The Second step handles local diversity. Sampled combinations of taxonomy nodes — called ‘mixes’ — are passed to an M3 to generate ‘meta prompts.’ A mix of {house cat, poem, travel enthusiast} becomes ‘Compose an exciting haiku about a house cat who goes on an adventure.’ Simula creates several meta prompts at once and then subsamples them to ensure distinct instances rather than a repetition of the same prompt.
It is important to note that the word “you” means “you”. Third Step Complexification is a user-configurable fraction. A user-Configurable fraction, cThe complexity control of generated meta prompts, while still maintaining the other requirements, is then passed to the M3. This separates complexity control from coverage control — you can raise the difficulty ceiling without sacrificing breadth.
It is important to note that the word “you” means “you”. Fourth step enhances quality through a ‘dual-critic’ approach. Simula asks the model whether an answer generated by the model is correct or incorrect, rather than just asking it once. This dual-verification design mitigates sycophancy bias — the tendency of LLMs to agree with plausible-sounding outputs — and is particularly important for tasks with a defined notion of correctness, such as multiple-choice questions or math problems.
The Experiments Show
Researchers generated data sets of up to 516K points in five different domains. These included CTI MCQ (a dataset with multiple choice questions for understanding the standards and threats associated with CTI), as well as LoRA Fine-Tuning, which was run 10 times with each configuration using different seeds. The researchers reported mean accuracy and 95% confidence intervals. The researchers generated data sets of up to 512K points in five different domains. These included CTI MCQ (a dataset of multiple-choice questions for assessing CTI standards and threats; CTI RCM (an open-ended task requiring the models to generate a Common Weakness Enumeration category from a Common Vulnerabilities & Exposures description); LEXam (covering Swiss, EU and international law exams in English and German); GSM8k, (grade-school mathematics);
Across all datasets and data sizes, the full Simula system — combining global diversification, local diversification, complexification, and critiquing — consistently outperformed simpler baseline configurations. Noteworthy, the combination of Global and Local diversifications was essential; either alone produced suboptimal outcomes depending on the dataset and scale.
These results are particularly interesting. In GSM8k the High Complexity data split resulted in a 10% increase in accuracy over the Low Complexity at 64K item. But on LEXam, where the teacher model achieved only 57% accuracy, higher complexity data actually hurt performance — demonstrating that complex data is only beneficial when the teacher model is strong enough to generate reliable labels for it. LEXam’s critic rejection rate reached 61% compared with just 2% of CTI MCQ, only 9% of CTI RCM and 9% of GSM8k. This directly reflects the weakness in this domain.
The Student-Teacher Gap Effect on Scaling Laws is a separate but equally important result. CTI-RCM showed that student model performance reached saturation at 128K points after approximately 83% had been bridged between student accuracy starting out (40%) compared to the performance of the teacher model (70%). GSM8k did not show saturation as the student’s performance peak (75%) was still sufficiently distant from the teacher model (88%).
The Intrinsic Assessment is Rethought
The research team introduces Beyond Generation two new evaluation approaches. Taxonomic Coverage measures what fraction of taxonomy nodes at each level are represented in a dataset — a structured alternative to coarse embedding-based cosine distance metrics that fail to provide actionable insights. Calibrated Complexity Scoring assigns Elo ratings to individual data points by running batch-wise pairwise comparisons, a method the research team call ‘calibrated attribute scoring,’ which proved to align well with human-annotated complexity labels on the MATH dataset.
A striking finding is that, on the basis of taxonomic cover, the real-world datasets are almost always less comprehensive than Simula’s generated variants. This holds true even when diversity metrics based on embeddings show the contrary. The cosine-based distance is not a good proxy for data quality.
The Key Takeaways
- Simula’s reasoning-first, seedless framework controls quality, diversity, and complexity as independent axes — enabling fine-grained synthetic dataset design without relying on manual prompts, evolutionary algorithms, or seed data from the target distribution.
- The combination of Global and Local Diversification is crucial: each component alone produces suboptimal results but when combined, they improve downstream model performance for all datasets and sizes tested.
- Data complexity helps model performance in most domains, but can hurt when the teacher model is weak — on LEXam, where Gemini 2.5 Flash (non-thinking) achieved only 57% accuracy, the Low Complexity split outperformed the High Complexity split.
- The taxonomic coverage of real-world datasets is almost never the same as that generated by Simula, despite standard metrics such as cosine-based embedding distances.
- Data scaling laws are driven by data properties, not size alone — the full Simula system reached higher downstream performance with fewer samples compared to baseline approaches, making it more cost-effective across the full data lifecycle despite requiring up to 5x more inference calls per data point.
Check out the Paper You can also find out more about the following: Technical details. Also, feel free to follow us on Twitter Join our Facebook group! 130k+ ML SubReddit Subscribe Now our Newsletter. Wait! Are you using Telegram? now you can join us on telegram as well.
Want to promote your GitHub repo, Hugging Face page, Product release or Webinar?? Connect with us
