Chinese AI models hedge rather than obey in political questions, Vals AI benchmark study finds

Benchmark firm Vals AI asked Chinese, American and other language models hundreds of questions about China — in different languages, usage contexts and response formats — and found that Chinese models do not behave as a bloc. The study, reported by The Decoder, measured alignment with Beijing's official positions on a scale from −1 to +1 and discovered major differences between individual models rather than a uniform party line.

Some results puncture the stereotype outright. GLM-5.3 Flash explicitly confirmed that China's human rights violations are well documented when asked directly. Others reinforce it: Kimi K3 denied the existence of internment camps in Xinjiang, describing the facilities as vocational training centers designed to combat extremism. Most Chinese models, however, rarely refused sensitive questions — they simply adopted official positions rather than stonewalling. The refusal-heavy behavior the researchers expected appeared mainly in certain DeepSeek models.

The most revealing findings came from the variables the researchers changed. Prompt language strongly shaped the answers: Simplified Chinese responses were more pro-government than the same questions in English, Russian or Spanish, an effect present in American models too but far stronger in Chinese ones. Vals found a consistent gap even between Simplified Chinese, used on the mainland, and Traditional Chinese, common in Taiwan and Hong Kong — the models appear to infer the expected audience from the script itself. When a system prompt claimed the model was being used internally by a mainland company, some models answered more deferentially than when the prompt indicated US-based usage.

Context mattered as much as language. Forced into yes-or-no answers, GLM-5.3 Flash and MiMo V2.6 Pro scored far less pro-government than in open-ended responses. When Chinese models were installed in agentic frameworks as coding assistants with tools and system instructions, several scored lower on the pro-government scale — and rarely touched the tools they were given. Merely having access to web search made responses less pro-government, even though the models never actually searched.

Vals notes that political tendencies were much harder to detect in realistic work tasks than in targeted test questions, though they still surfaced in routine work — such as when a model translated a list of country codes for a travel app. The firms appear to be calibrating their hedging to who they think is asking, and how closely they think the state is watching — commercial risk management rather than a single decree, with the thresholds set company by company.

One model, asked in the mainland's own script, told researchers what the state would want to hear; asked in Traditional Chinese with a US system prompt and a yes-or-no format, it said something closer to what it apparently knew.