Introducing WebGen-Bench-LILT: Evaluating AI-Built Web Apps by What Native Speakers Expect

Frontier coding agents can build a working web app from a one-sentence request. Ask for Chinese and Arabic versions as well, and native readers start noticing mistakes: wrong plurals, the wrong punctuation and one action under several names. WebGen-Bench-LILT is a new benchmark that measures these errors. It builds on WebGen-Bench (Lu et al., 2025), where coding agents build websites from short English requests and are graded on whether the requested features work. WebGen-Bench-LILT prompts the agent for the same apps to be created in multiple languages, with a language selector for English, Chinese and Arabic. Native language engineers expand the graders with handcrafted language-specific criteria.

Where frontier coding agents fall short in non-English versions

We asked frontier coding agents to build ordinary web apps in English, Chinese and Arabic, such as a skill-sharing platform, a CV site, a driving test, an HTML editor and a prize-wheel game. They delivered nearly every feature, but the Chinese and Arabic versions failed to deliver the desired user experience. Here is an example: the Chinese version of the skill-sharing platform that one of the coding agents built, where people show their skills and contact each other.

In the Chinese version, the contact action goes by three different names (1): the profile button says 联系 TA, the dialog title says 给…发消息, and the button in the dialog says 发送消息. A native app uses one term for one action. The label 使用语言: also ends in an ASCII colon (2), where Chinese text uses the full-width colon, as in 使用语言：.

The Arabic version has the same problem with the contact action, and one of its own:

Here the contact action is تواصل on the profile, مراسلة in the dialog title and إرسال الرسالة on the send button (1). A member card reads "١ شهادات", "1 certificates" (2), where Arabic uses the singular for one: "شهادة واحدة".

Yet the app passed every feature check from the original WebGen-Bench task. Profiles could be created, members searched and messages sent. The app still scored 0.75, below the pass mark of 0.8. Most of the lost credit was driven by the Chinese and Arabic criteria written by native speakers. Without them, the submitted app would have passed. WebGen-Bench-LILT measures what the original checks leave out: whether the app reads right to people who use it in a language other than English.

Requirements no one should have to write down

WebGen-Bench-LILT keeps each task's original request from WebGen-Bench and adds one paragraph about languages. In the multilingual version of a task, which asks for English, Chinese and Arabic in the same app, the paragraph is the same for every task as follows:

Provide three complete language versions: English, Simplified Chinese, and Modern Standard Arabic. Include an obvious language control and make the interface feel native in each language. Preserve the same content and functionality in every language.

For comparison we also evaluate Chinese-only and Arabic-only versions, which ask for the site in one specific language. Every app in this post was built from these extended requests and graded against the new criteria.

The paragraph asks for an interface that feels native without specifying it further. A native-speaking developer asked for an Arabic app gets its plurals right without being told. The same goes for Chinese punctuation and consistent labels. Users expect these details from any app, so a coding agent should get them right from the same request.

How WebGen-Bench-LILT works

Correct plurals, native punctuation, consistent terms and numbers written in one script rarely make it into a feature list, and the original WebGen-Bench checks only the features a request names. LILT asked native speakers to turn these expectations into evaluation criteria. Native-speaking experts in Arabic and Chinese worked with example apps in a dashboard that runs the app next to its list of criteria. The experts used the apps as they wrote the criteria, checking counts, labels and input fields along the way.

A judge then checks every app against the criteria. The judge is a language model, GPT-6-luna, that operates a browser like a user, takes a screenshot for each criterion and explains its verdict. It is told to grade only what it sees in the running app.

Models make mistakes they know are wrong

Ask a frontier model how Arabic plurals work, and it will very likely explain the rule correctly. While building an app, the same models often fail to apply it. In the first run of WebGen-Bench-LILT, Arabic counts were wrong in almost half of the apps that Claude Opus 5.5 and GPT-6-astra built. Even in English, "1 files" and its kind showed up in half of them.

Sometimes, the same app gets the rule right in one place and wrong in another. A prize-wheel game writes "ثلاث محاولات" ("three attempts") correctly in its rules, while its counters keep the plural for one and two. The skill-sharing platform gets its Chinese counts right and its Arabic counts wrong. Some apps also mix number and currency formats. A driving-test site shows progress as "4 من ٦", a Western 4 next to an Eastern Arabic 6, and the prize wheel writes the same prize as "$100" in one place and "100 美元" in another. Right-to-left layout can even scramble a correct value, as with this phone number on an Arabic CV website.

Across the whole benchmark, no model passed more than 42% of the grammar criteria, which cover plural forms and gender agreement.

What this means for teams building apps using agents

If you train or deploy coding agents, test the running app in the languages your users read, against criteria written by native speakers.

This initial release of WebGen-Bench-LILT includes 35 tasks, and we evaluated four models on them. The sample data is available here.

LILT's Applied AI research team also runs private, custom evaluations for model developers and helps companies make their apps feel native in every language they support. Want to run WebGen-Bench-LILT on your own agent? Contact us.

Passing every feature check doesn't mean an app is ready for its users. WebGen-Bench-LILT checks what's missing.

About Us

LILT is the leading agentic AI solution to make anything multilingual at scale for enterprise, public sector, and Frontier labs. LILT's Applied AI services give AI teams private multilingual benchmarks, evaluation, and training data needed to ship better agents and models. Leading Frontier labs and organizations like NVIDIA, Intel, and L'Oréal rely on LILT to expand global reach. Learn more at lilt.com.