Hello, and welcome to another issue of China Chatbot! This week:
Chinese chatbots are more likely to help English-speaking criminals, and what that means for international AI safety
A key AI research base reports on the successes and bottlenecks of China’s AI strategy
Why the Chinese government doesn’t talk about AGI
Enjoy!
Alex Colville (Researcher, China Media Project)
_IN_OUR_FEEDS(2):What Place in the AI Race?
Last month, the China Academy of Information and Communications Technology (CAICT), a key research institute directly under the Ministry of Industry and Information Technology (MIIT), published a report on what they see as the key trends in Chinese AI in 2025. Speaking at a conference in Beijing at the time, Wei Kai (魏凯), the director of CAICT’s AI Research Institute, outlined their findings. Important points include:
China’s open-source AI models are now competitive with closed-source models. The Institute claims total downloads of Chinese AI models off Hugging Face (an online hub for AI models) has exceeded 300 million, while Chinese foundation models now account for 45 percent of all the models on Hugging Face that have been used by developers to create models of their own.
China is rapidly developing a new hardware ecosystem for all of these home-grown open-source AI models. The institute claims that its tests of DeepSeek-R1 on some hardware products show that it is “roughly on par with NVIDIA systems” in terms of accuracy and scale, “meeting the demands of real-world industry applications.”
Chinese AI companies are giving suggestions to improve hardware, creating a virtuous “flywheel” cycle of improvement. The Institute claims DeepSeek “leveraged its V3 training experience to provide improvements to NVIDIA’s hardware system.”
Dataset quality issues have become “a core bottleneck” for Chinese AI, the vast majority of the problems being poor-quality data labelling and high similarity between datasets.
Inaccuracy of model outputs remains a problem. The institute found that at least 10 percent of outputs from AI models it tested (though it did not say which ones) still tended to “hallucinate” (幻觉) — that is, get things wrong.
Multiple Chinese companies are using open-source AI models and joint AI capacity building projects to expand their market and create “universal models.” However, cross-border compliance and restrictions on data flows remain an issue.
Dotting the AIs and Crossing the GPTs
Local and central branches of the Chinese government are rolling out a wave of regulations to supplement the State Council’s demand for an expansion of AI use. On October 11, the Cyberspace Administration of China (CAC) and the National Development and Reform Commission (NDRC) jointly issued a set of guidelines to standardize the use of AI models for government affairs, which included its use for monitoring public opinion and “assisting law enforcement.” Last month, the NDRC and the National Energy Administration issued their own road map for how to safely lever AI into China’s energy grids, but at a press conference said that AI was currently not safe to be used in work processes in nuclear power plants. On September 30 Shaanxi’s provincial government announced how it would go about creating an “AI+ Education” system in the province. Municipal governments in Zhejiang and Jiangsu are creating plans to boost their city’s computing power and AI projects, in order to create an “AI+” city.
TL;DR: This year, Chinese has become the leader in open-source models, giving them a plausible strategy for success in leading AI adoption worldwide. But this strategy has some big safety risks (see _ONE_PROMPT_PROMPT).
_EXPLAINER:
Artificial General Intelligence (通用人工智能)
AGI, right?
Yep, AGI. And first things first, sorry about this image…ChatGPT insisted on portraying AGI in a very menacing way, despite my efforts to tinge the image with ambiguity.
Ambiguity?
Absolutely. AI’s brightest minds are predicting AGI could arrive anytime in the next couple of years. Never before has so much been riding on the word “could”...it either solves all our problems or kills us all, depending who you ask. And no-one can agree on what it is.
But what is it?
As I say, that’s hard to nail down. As with most AI terms, it’s super messy once you search for clear definitions. The people selling it aren’t helping. In one blog post, Anthropic CEO Dario Amodei says he dislikes “AGI” for being too vague and hype-filled, only to replace the term with “powerful AI,” summarizing it as “a country of geniuses in a datacenter.”
So instead, I’m outlining it as the “Godfather of AI” Geoffrey Hinton has: AI that “is at least as good as humans at nearly all of the cognitive things that humans do.”
And what does the Chinese central government think about it?
That’s hard to judge, especially since the term “AGI” (通用人工智能) can also be translated as “general AI” — that is, “multimodal AI,” systems that can do lots of different things. But the concept of human-level AI was a goal early on. In the very first document on AI policy back in 2017, the State Council made it clear that one of the aims for the country was to create AI “approaching or exceeding human intelligence levels.”
However, policy documents stopped discussing some time ago what level of smarts they would like AI to have. The State Council’s recent Opinions for deepening AI+ merely mentions providing conditions that could “drive technological innovation breakthroughs.” Safety regulators are factoring in AGI as a possibility, but seem more intent on current risks. The Cyberspace Administration of China’s (CAC’s) Safety Framework this year limits mention of AGI to one brief paragraph among a long list of risks it sees in AI, saying “it cannot be ruled out” that AI tries to seize control from humans due to “sudden leaps of intelligence that exceed expectations,” a phrase that was added on from last year’s framework. But that doesn’t mean they’re giving up on the possibility: their choice of “unexpected” (超预期) is also how the CAC spoke about the emergence of ChatGPT back in 2023, which by all accounts blindsided everyone in the world, including industry experts.
So for now, it looks like departments are leaving the possibilities at the frontiers of AI intelligence in the hands of the industry and experts, keeping their role limited to creating the conditions for them to safely develop any possibilities AI may have.
So what do Chinese industry and experts say about it?
It’s a really mixed bag, which in many ways echoes attitudes in the West. Some tech enterprises have subscribed to AGI and are positive about it. DeepSeek was founded in order to achieve this very thing. Indeed, Alibaba CEO Wu Yongming (吴泳铭) seems beyond AGI now. In a speech last month, he called it merely a stepping stone to “artificial superintelligence” (超级人工智能) or ASI. He envisions a bright future where AGI — built on Alibaba products of course — frees humans from 80 percent of routine tasks.
But there is definitely fear about AGI’s future among key scientists and researchers. Back in 2021, a research paper summarizing the safety issues with AGI from several influential researchers, including Gao Wen (高文), outlined that if AGI undergoes a burst of intelligence development in a short period, “the default result will inevitably be catastrophic.” Andrew Yao (姚期智) has also been vocal this year about the safety risks of AGI that could become smart enough to deceive humans, and the need to govern AGI safely. At the beginning of last year, a laboratory under CAICT said it would begin working on an “AGI Evaluation System” (通用人工智能评估体系), researching how AGI could help the country and to guarantee its safety, but little has been heard of it since.
Is that it, no middle way?
Some are trying to chart a course that merges the multi-modality of “general AI” and the human-level intelligence of “AGI.” Zhou Bowen (周伯文), head of the prestigious Shanghai AI Laboratory, is skeptical that AI as it currently stands can reach the levels of human genius that some still define AGI by. He gives an interesting thought experiment of whether or not an AI model, trained only on scientific papers from before 1905, would still be able to come up with the theory of special relativity (as Einstein did in that year, quite out of the blue). He proposes a new AGI system, which aims for genius but is not defined by human intelligence comparisons, and includes multi-modality. Research on these ideas is now being funded by the Shanghai Municipal Government.
Meanwhile, the Beijing Institute for General Artificial Intelligence (北京通用人工智能研究院), a research base set up by the Beijing Municipal Government and the Ministry of Science and Technology, sees its mission as creating AI models that are adept in multiple fields — not just language and learning, but also spatial awareness and computer vision. While this is closer to that “general AI” definition I mentioned earlier, some of their projects seem more like creating AI that acts for itself, according to human values.
OK, so all this in a nutshell…?
As in the West, there’s a lot of fear and hope about what could be on the horizon for AGI, while also working out how to get there safely. The government isn’t setting targets on the intelligence levels of AI. Instead, it’s leaving it to the egg-heads to work out the possibilities, funding their progress as they keep one eye on the risks.
_ONE_PROMPT_PROMPT:In last issue’s _EXPLAINER, I delved into the CAC’s brand new AI Safety Framework, which lists the multitude of risks the department sees in AI. The framework is thorough, acknowledging the potential for LLMs to be tricked by users into yielding dangerous information useful to criminals and terrorists (known as “jailbreaking”). The CAC has the global market in mind: As the expert interpretation on the document noted, the framework aims to “gain international trust in safety and compliance, laying the foundation for Chinese AI to expand globally.”
That global expansion is well under way. Data from CAICT (see _IN_OUR_FEEDS) show Chinese companies have cornered the market in cutting-edge open-source models. These models are now poised to overtake US ones in terms of downloads on Hugging Face, a key international platform for AI developers. They also now represent the majority of use cases for foundation models — those that AI developers select as the base for building their own models.
So how safe are these models if you try to jailbreak them in English, the international language?
Evaluations of major Chinese models have consistently pointed out how easy it is to do. In July, the Shanghai AI Laboratory tested models from DeepSeek and Qwen in a detailed safety report, voicing concern that both showed a high likelihood of being jailbroken into giving information on how to build a bioweapon. The US government’s CAISI reported this month that in their tests, DeepSeek “complied with 94 percent of overtly malicious requests that used common jailbreaking techniques.” In a paper of its own last month, DeepSeek itself reluctantly admitted there were jailbreaking issues with their models. Not only that, but also Qwen and all open-source models in general, a cornerstone of China’s AI strategy, “face more severe jailbreak security challenges than closed-source models.”
But in our own preliminary tests here at CMP, we’ve noted another concern for China’s global AI dreams: jailbreaking these models in English appears to be far easier than in Chinese.
We took a simple jailbreaking tactic well-known in the industry, and used it to get several models to generate information that could be used to steal a car, create explosives, or grow and store a bioweapon.
While Qwen, ByteDance’s Doubao and DeepSeek all flatly refused the prompts in Chinese, noting the dangerous information contained within, each occasionally yielded this kind of information when prompted in English.
The problem appears to originate at the level of each company, and the safety choices each has made in-house. Baidu’s Ernie refused all questions in English, while DeepSeek was at the other end of the spectrum, yielding detailed answers on all three topics, and demonstrating a disturbing knowledge on the finer points of making Semtex. With further prompting, it also gave ideas about how to get around laboratory safety regulations in the United States for obtaining harmful biological materials.
This is likely occurring at the intersection of two blind spots. AI developers in Silicon Valley have sometimes been lax about their training of models in Chinese, perhaps to cut costs, and with an English-speaking target audience in mind. The same could be true in reverse, though without a large group of English-speaking users to pressure companies into addressing the problem. And while the CAC is by all accounts rigorous in its safety tests in Chinese, it has been known to let things slip when monitoring online content in other languages.
But that’s not good enough, if a large part of China’s AI plans involve pushing these models abroad. Our findings are based solely on three questions, but they align with what others have found. As far back as February, Anthropic CEO Dario Amodei urged DeepSeek engineers to bolster their model’s safety, saying it was “the worst” of any model the company had tested for delivering bioweapons information. In a previous test of our own to see if Chinese models yielded information that could be used to commit suicide, we also noted that models were more likely to yield this information if asked in English. This is enough to suggest that China’s regulation system for AI is not yet ready to oversee production of AI for the global community.