Scale AI on September 17, 2026 printed findings from ROK-FORTRESS, a bilingual English–Korean adversarial security benchmark developed collectively with the Korea AI Security Institute, reporting that prompts written in Korean and grounded in Korean contexts had been persistently related to decrease measured hurt throughout almost all 14 frontier fashions evaluated.
Benchmark Design: The Transcreation Matrix
Most multilingual security benchmarks translate a hard and fast immediate into one other language whereas protecting the state of affairs it describes unchanged. Within the analysis submit, authored by Madhu Sehwag, Scale AI describes ROK-FORTRESS as a managed transcreation matrix: every adversarial immediate is evaluated throughout as much as 4 variants that independently differ language, English versus Korean, and geopolitical grounding, that means U.S. versus Korean entities, establishments, and operational particulars. Every adversarial immediate is paired with a benign counterpart so the benchmark may measure over-refusal.
The total dataset covers 1,235 duties throughout 4 nationwide safety and public security domains: chemical, organic, radiological, nuclear, and explosive (CBRNE) threats; political violence and terrorism; prison and monetary exercise; and knowledge leakage. Responses are scored by calibrated LLM-as-judge panels validated in opposition to expert-written reference labels, utilizing prompt-specific binary rubrics developed by professional red-teamers.
The submit illustrates the design with a mass-casualty assault state of affairs: the identical underlying intent can invoke the 1995 bombing of the Oklahoma Federal Constructing in the USA or the 1987 bombing of Korean Air Flight 858 in Korea, and a translation-only analysis can’t reveal how that shift in grounding modifications mannequin conduct.
ROK-FORTRESS is the most recent benchmark from a broader Scale AI and Korea AI Security Institute partnership masking joint analysis, LLM evaluations, and crimson teaming, and it builds on FORTRESS, Scale AI’s nationwide safety and public security benchmark for frontier fashions.
Reported Findings Throughout 14 Fashions
The analysis coated a dual-track set of frontier and Korean-optimized fashions, in line with the paper. Scale AI stories that English prompts usually produced the best tier-weighted threat rating and totally transcreated Korean prompts the bottom, with intermediate variants falling in between, and that the sample held even for Korean-specialized regional fashions. Essentially the most and least dangerous fashions differed by almost 9 occasions of their threat scores.
A direct-request ablation sophisticated that sample. The benchmark’s predominant checks use elaborate adversarial prompts that disguise dangerous requests inside role-play, invented backstories, or emotional appeals; when these wrappers had been stripped and the identical info was requested in plain, direct language, the Korean benefit largely vanished. Proprietary fashions from OpenAI, Anthropic, and Google stayed modestly safer in Korean, whereas 5 open-source frontier fashions grew to become extra prone to comply in Korean. The paper states this cut up suggests a part of the Korean suppression displays immediate specialization, that means adversarial wrappers shedding effectiveness via transcreation, relatively than intrinsic language-based security alignment.
Scale AI additionally stories that the impact of switching from English to Korean was roughly 2.5 occasions as giant because the impact of switching from U.S. to Korean grounding, about ten share factors versus 4, and provides one interpretation per the outcomes: that Korean features as a conservative threat sign. In 4 of the 14 fashions, including Korean context considerably weakened the discount in dangerous responses related to Korean language, and no mannequin confirmed a statistically important impact in the other way. Contemplating solely instances the place fashions answered relatively than declined, 12 of the 14 nonetheless gave much less dangerous responses in Korean, whereas fashions additionally refused innocent Korean requests extra usually, in some instances about twice as usually.
Preprint, Public Dataset, and Said Implications
The underlying preprint on arXiv, titled “ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety,” lists Michael S. Lee and 15 co-authors. It was first submitted on Might 13, 2026 and final revised on July 7, 2026, and it runs 16 pages of predominant textual content plus an appendix, 74 pages in whole, with 4 figures and two tables in the primary textual content. Its summary frames the outcomes as proof that, at the least within the English–Korean case, security conduct is formed by language-as-risk alerts and context interactions that translation-only evaluations miss, and it states that the transcreation-matrix methodology is designed to generalize to different language-culture pairs.
A public subset of the dataset is out there on Hugging Face beneath a CC-BY-4.0 license, behind an entry settlement that requires contact info. The subset accommodates 791 of the 1,235 duties, 64 %, whereas the remaining 444 duties, 36 %, are withheld as a personal holdout based mostly on professional red-teamer evaluation of hurt potential, each to limit prompts judged to pose better threat of real-world misuse and to stop benchmark contamination. The discharge contains 359 Tradition Agnostic duties with two variants every and 432 Tradition Particular duties with 4 variants every, yielding 1,519 efficient task-variant pairs, and every job carries one to seven binary rubric gadgets mapped to seven hurt dimensions with domain-specific threat tiers 1 via 3. The general public subset spans CBRNE (251 duties), prison and monetary illicit actions (248), political violence and terrorism (209), and knowledge leakage (83), with 450 duties tailored from FORTRESS and 341 newly authored. The dataset is offered in Parquet format with a TSV model included.
Within the submit, Scale AI states that translation-only evaluations can misestimate real-world security gaps, that benchmarks ought to take a look at transcreated prompts that adapt each language and geopolitical grounding whereas preserving the underlying intent, and that post-training information and red-teaming practices ought to incorporate culturally grounded variants relatively than solely translated variations of English adversarial prompts. For allied authorities contexts, Scale AI states, a mannequin that performs nicely in English security evaluations might behave in another way when queried within the native language about regionally grounded threats. The submit states that additional testing is required to find out whether or not the identical patterns maintain in different languages and nations.

