Polish tops a surprising new leaderboard for long-context AI, outperforming English and Chinese when large language models must reason over very long documents. In a multilingual benchmark spanning 26 languages and context windows stretching to 64,000 tokens and beyond, Polish achieved the highest average accuracy at long-context scales, around 88%. English slipped to sixth place, while Chinese landed among the bottom four.
The results come from the OneRuler benchmark introduced in a COLM 2025 paper, which evaluates retrieval and aggregation tasks at multiple sequence lengths. As context windows grow, the ranking of languages shifts dramatically—suggesting that how text is tokenized and the underlying writing system can matter more than the volume of training data.
Languages using Latin-based scripts, such as Polish, French, and Spanish, consistently held up better as sequences lengthened. In contrast, languages with logographic or abugida scripts, including Chinese, Korean, and Tamil, showed only moderate performance at shorter lengths and deteriorated further at longer ones. This flips the usual expectation that English-heavy training datasets guarantee superior results across the board. When models have to locate, combine, or summarize information buried deep in long contexts, structural aspects of the language and tokenization efficiency appear to take the lead.
The study also highlights how the performance gap widens with scale. The difference between the best- and worst-performing languages grew from about 11% at 8,000 tokens to roughly 34% at 128,000 tokens. Sensitivity to prompt design was another notable finding: allowing the model to respond with “none” if a target string was absent caused English accuracy to drop by about 32% at 128k tokens, underscoring how small instruction tweaks can have outsized effects in long-context settings.
Key takeaways for teams building or choosing long-context LLMs:
– Don’t rely on English-only evaluations. Multilingual testing is essential once context windows reach tens of thousands of tokens.
– Tokenization and script matter. Latin-based scripts tend to yield higher accuracy at long lengths than logographic or abugida systems.
– Expect widening disparities as context grows. Performance differences that look small at 8k tokens can become large at 64k–128k.
– Prompt design is critical. Minor instruction changes can significantly shift outcomes in long-context retrieval and aggregation tasks.
As context windows keep expanding, the benchmark suggests we should rethink how we evaluate and compare models across languages. English dominance in typical benchmarks may not reflect real-world performance when the task requires deep retrieval or synthesis across very long documents. For more reliable outcomes, developers may need to optimize tokenization strategies and evaluation protocols with script and language structure in mind.
