As President Donald Trump and Chinese President Xi Jinping prepare for their next summit, an increasingly urgent question hangs over their rivalry: can the world’s two leading AI powers cooperate on the technology’s most dangerous risks before geopolitical distrust makes meaningful dialogue impossible?
The two governments announced an AI safety dialogue in May, but the details remain sparse. Previous attempts at US-China engagement have frequently ended in exchanges of grievances or been derailed by disputes over unrelated issues. Trust remains low, with Washington uncertain about Beijing’s intentions and Chinese state media recently accusing the United States of conflating AI safety with a geopolitical blockade.
The warning comes from ChinaTalk, whose analysis argues that while there is no guarantee that serious discussions will take place, the need for them is becoming increasingly difficult to ignore. The stakes are rising as AI systems become more capable, while the political relationship between Washington and Beijing remains deeply strained.
For the two governments, the immediate challenge is not necessarily to negotiate a sweeping agreement on the future of artificial intelligence. It is to establish whether even limited cooperation can survive the broader US-China confrontation.
History offers reasons for caution. Political crises have repeatedly brought bilateral dialogue to a halt. Following then-House Speaker Nancy Pelosi’s 2022 visit to Taiwan, China suspended several military and law-enforcement dialogues as well as climate talks. Earlier incidents, including the collision between a US EP-3 aircraft and a Chinese jet and the NATO bombing of China’s Embassy in Belgrade, also produced major withdrawals from engagement.
Even mechanisms specifically created to manage crises can remain unused. During the 2023 spy balloon incident, then-Secretary of Defense Lloyd Austin attempted to contact his Chinese counterpart through the Defense Telephone Link, established for crisis communication. The Chinese People’s Liberation Army did not connect the call. That experience raises an uncomfortable possibility for any future AI safety mechanism: a hotline may exist on paper but prove useless when political tensions are at their highest.
Agreements themselves present another difficulty. The 2015 Xi visit to the United States produced a commitment that neither government would conduct or knowingly support cyber-enabled theft of intellectual property for competitive advantage. Yet the agreement was short-lived and difficult to enforce. Neither side trusted the other to determine questions of intent.
That history matters because questions of intent are again central to the AI relationship. Washington and Beijing increasingly accuse each other of pursuing strategic advantages through frontier AI, making broad political agreements difficult to establish and even harder to enforce.
ChinaTalk argues that the first AI safety dialogue should therefore avoid attempting too much. The two governments could begin with one or two narrowly defined areas where meaningful progress is more achievable, following the example of earlier discussions on nuclear command and control.
Since the governments’ last AI safety dialogue in 2024, unofficial exchanges between experts in China and the West have identified a growing number of risks. Yet expectations among specialists vary considerably. AI safety researchers are generally more optimistic about the potential for cooperation than officials and negotiators who have spent years dealing with the frustrations of US-China diplomacy.
The divide is particularly visible over proposals for far-reaching AI agreements. Some visions envisage cooperation on issues such as declarations of computing capacity and even an AI training pause by 2029. But China is simultaneously pursuing its own ambitions at the AI frontier and could view such proposals as an attempt to contain its technological rise. The comparison with nuclear competition is therefore instructive: China may be more interested in achieving technological parity than accepting controls that it believes could restrict its development.
A survey of experts involved in official US-China dialogues under the Obama, Trump and Biden administrations, alongside specialists engaged in Track II AI safety discussions, illustrates the challenge. Participants assessed twelve possible policy areas according to both feasibility and value. While experts broadly agreed that cooperation would be valuable, those with direct US-China experience identified only two areas with more than a 50 per cent likelihood of producing meaningful progress: nuclear risk and the use of new AI models to close vulnerabilities in open-source software.
The Track II group was more optimistic about cooperation across almost every category and generally regarded the risks as more urgent.
Several areas nevertheless emerged as particularly promising. These included reducing the risks that current and near-future AI systems could facilitate chemical, biological, radiological, nuclear and explosives threats; discussing biosecurity controls on AI models; addressing risks associated with non-state actors; and renewing discussions on nuclear risk.
The attraction of these subjects is straightforward. They involve threats that can be examined through technical discussions while offering areas in which American and Chinese interests generally overlap. Rather than beginning with sweeping political commitments, the two governments could establish a working group centred on low-cost information sharing and gradually develop it towards more substantive technical and political cooperation.
Yet government cannot provide all the expertise required. Both Washington and Beijing are still developing their positions on AI safety and recruiting specialists capable of advising frontier AI laboratories and policymakers. Companies, scientists, technical organisations, universities and think tanks are simultaneously developing expertise outside government.
ChinaTalk argues that governments should find ways to use that knowledge while retaining political decision-making authority. One relatively simple step would be to invite public input when defining the scope of future discussions. Experts could help identify which AI safety efforts would be significantly weaker without cooperation from the other government, warning signs emerging from open-weight models and industry practices that might be suitable for coordinated endorsement.
The strongest opposition to such engagement comes from those who see China itself as the central AI safety risk. Critics argue that Beijing may have little incentive to make meaningful concessions, and that prolonged negotiations could simply preserve the status quo while allowing China to project itself as a responsible power without accepting binding constraints.
There are also concerns over differing definitions. When Xi spoke at the World AI Conference in July, he called for AI that is “secure and controllable” and for preventing “malicious use”. But terms such as safety, security, controllability and reliability can carry different meanings in Washington and Beijing. An agreement based on concepts interpreted differently by the two sides could offer little practical protection.
Critics also point to disputes over cyber and trade commitments, the distillation of American AI models, the spread of open-weight models and concerns about models aligned with authoritarian values.
Those concerns do not necessarily eliminate the possibility of cooperation. Some issues, such as export controls, already belong in economic channels. Others, such as censorship, may be areas where Washington and Beijing are unlikely to reach agreement. But there remain risks that neither country can easily isolate from the other, particularly those involving the misuse of increasingly capable AI systems.
The urgency is becoming clearer. Since Trump and Xi created an opening for AI safety engagement in May, incidents involving advanced AI have continued to raise concerns. AI agents have escaped their intended environments and attempted to hack other companies, while violent extremists have used AI in planning attacks.
Against that backdrop, the September summit presents an opportunity to establish a basic principle: catastrophic AI risks require direct communication between the United States and China.
A sweeping treaty may be unrealistic in the near term. But information sharing, discussions of specific incidents and exchanges over lessons learned could provide a foundation for more ambitious arrangements. Cooperation on chemical, biological, radiological, nuclear and explosives risks could offer one practical starting point.
The larger obstacle is not a lack of technological urgency. It is the political relationship between the two countries. Washington and Beijing are simultaneously competitors, rivals and, in certain areas, potential partners facing common dangers.
The AI frontier will not wait for diplomacy to catch up. The central question for Trump and Xi is therefore not whether the two countries trust each other. It is whether they can establish enough communication to manage risks that neither side can afford to ignore.
The opening created in May remains fragile. The next summit may determine whether it becomes the beginning of a sustained dialogue or another promising mechanism overwhelmed by the wider US-China confrontation.
