Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI
  • Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
AI-trends.todayAI-trends.today
Home»Robotics»Scale AI Stories ROK-FORTRESS Findings on Multilingual AI Security – Unite.AI

Scale AI Stories ROK-FORTRESS Findings on Multilingual AI Security – Unite.AI

Robotics By Gavin Wallace18/09/20265 Mins Read
Facebook Twitter LinkedIn Email
The Rise of Smarter Robots: How LLMs Are Changing Embodied
The Rise of Smarter Robots: How LLMs Are Changing Embodied
Share
Facebook Twitter LinkedIn Email

Scale AI on September 17, 2026 printed findings from ROK-FORTRESS, a bilingual English–Korean adversarial security benchmark developed collectively with the Korea AI Security Institute, reporting that prompts written in Korean and grounded in Korean contexts had been persistently related to decrease measured hurt throughout almost all 14 frontier fashions evaluated.

Benchmark Design: The Transcreation Matrix

Most multilingual security benchmarks translate a hard and fast immediate into one other language whereas protecting the state of affairs it describes unchanged. Within the analysis submit, authored by Madhu Sehwag, Scale AI describes ROK-FORTRESS as a managed transcreation matrix: every adversarial immediate is evaluated throughout as much as 4 variants that independently differ language, English versus Korean, and geopolitical grounding, that means U.S. versus Korean entities, establishments, and operational particulars. Every adversarial immediate is paired with a benign counterpart so the benchmark may measure over-refusal.

The total dataset covers 1,235 duties throughout 4 nationwide safety and public security domains: chemical, organic, radiological, nuclear, and explosive (CBRNE) threats; political violence and terrorism; prison and monetary exercise; and knowledge leakage. Responses are scored by calibrated LLM-as-judge panels validated in opposition to expert-written reference labels, utilizing prompt-specific binary rubrics developed by professional red-teamers.

The submit illustrates the design with a mass-casualty assault state of affairs: the identical underlying intent can invoke the 1995 bombing of the Oklahoma Federal Constructing in the USA or the 1987 bombing of Korean Air Flight 858 in Korea, and a translation-only analysis can’t reveal how that shift in grounding modifications mannequin conduct.

ROK-FORTRESS is the most recent benchmark from a broader Scale AI and Korea AI Security Institute partnership masking joint analysis, LLM evaluations, and crimson teaming, and it builds on FORTRESS, Scale AI’s nationwide safety and public security benchmark for frontier fashions.

Reported Findings Throughout 14 Fashions

The analysis coated a dual-track set of frontier and Korean-optimized fashions, in line with the paper. Scale AI stories that English prompts usually produced the best tier-weighted threat rating and totally transcreated Korean prompts the bottom, with intermediate variants falling in between, and that the sample held even for Korean-specialized regional fashions. Essentially the most and least dangerous fashions differed by almost 9 occasions of their threat scores.

A direct-request ablation sophisticated that sample. The benchmark’s predominant checks use elaborate adversarial prompts that disguise dangerous requests inside role-play, invented backstories, or emotional appeals; when these wrappers had been stripped and the identical info was requested in plain, direct language, the Korean benefit largely vanished. Proprietary fashions from OpenAI, Anthropic, and Google stayed modestly safer in Korean, whereas 5 open-source frontier fashions grew to become extra prone to comply in Korean. The paper states this cut up suggests a part of the Korean suppression displays immediate specialization, that means adversarial wrappers shedding effectiveness via transcreation, relatively than intrinsic language-based security alignment.

Scale AI additionally stories that the impact of switching from English to Korean was roughly 2.5 occasions as giant because the impact of switching from U.S. to Korean grounding, about ten share factors versus 4, and provides one interpretation per the outcomes: that Korean features as a conservative threat sign. In 4 of the 14 fashions, including Korean context considerably weakened the discount in dangerous responses related to Korean language, and no mannequin confirmed a statistically important impact in the other way. Contemplating solely instances the place fashions answered relatively than declined, 12 of the 14 nonetheless gave much less dangerous responses in Korean, whereas fashions additionally refused innocent Korean requests extra usually, in some instances about twice as usually.

Preprint, Public Dataset, and Said Implications

The underlying preprint on arXiv, titled “ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety,” lists Michael S. Lee and 15 co-authors. It was first submitted on Might 13, 2026 and final revised on July 7, 2026, and it runs 16 pages of predominant textual content plus an appendix, 74 pages in whole, with 4 figures and two tables in the primary textual content. Its summary frames the outcomes as proof that, at the least within the English–Korean case, security conduct is formed by language-as-risk alerts and context interactions that translation-only evaluations miss, and it states that the transcreation-matrix methodology is designed to generalize to different language-culture pairs.

A public subset of the dataset is out there on Hugging Face beneath a CC-BY-4.0 license, behind an entry settlement that requires contact info. The subset accommodates 791 of the 1,235 duties, 64 %, whereas the remaining 444 duties, 36 %, are withheld as a personal holdout based mostly on professional red-teamer evaluation of hurt potential, each to limit prompts judged to pose better threat of real-world misuse and to stop benchmark contamination. The discharge contains 359 Tradition Agnostic duties with two variants every and 432 Tradition Particular duties with 4 variants every, yielding 1,519 efficient task-variant pairs, and every job carries one to seven binary rubric gadgets mapped to seven hurt dimensions with domain-specific threat tiers 1 via 3. The general public subset spans CBRNE (251 duties), prison and monetary illicit actions (248), political violence and terrorism (209), and knowledge leakage (83), with 450 duties tailored from FORTRESS and 341 newly authored. The dataset is offered in Parquet format with a TSV model included.

Within the submit, Scale AI states that translation-only evaluations can misestimate real-world security gaps, that benchmarks ought to take a look at transcreated prompts that adapt each language and geopolitical grounding whereas preserving the underlying intent, and that post-training information and red-teaming practices ought to incorporate culturally grounded variants relatively than solely translated variations of English adversarial prompts. For allied authorities contexts, Scale AI states, a mannequin that performs nicely in English security evaluations might behave in another way when queried within the native language about regionally grounded threats. The submit states that additional testing is required to find out whether or not the identical patterns maintain in different languages and nations.

AI security
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI

22/11/2026

Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI

22/11/2026

OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI

22/11/2026

Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI

22/11/2026
Top News

Carl Pei believes that the phone of the future will only have one app

A New Chatbot Needs to Unlock the Secrets and techniques in Tattered Historical Greek Information

A Dark Horse AI is rewriting rules of game design

OpenAI Engineer Accused of Spying on Executives Ordered to Stay in Custody

OpenAI Re-acquires Two Thinking Machines Lab cofounders

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

DeepSeek releases R1-0528 – An Open Source Reasoning AI model that delivers enhanced math and code performance with single-GPU efficiency

30/05/2025

Ansible automation lab design with playbooks, inventories, roles, vault, dynamic inventory, and custom modules

29/05/2026
Latest News

Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI

22/11/2026

Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.