On July 27, 2025, Concordia AI hosted the AI Safety and Governance Forum at the World AI Conference in Shanghai. A special edition of our newsletter highlighted key AI safety updates from the conference; this post offers a comprehensive overview of the Forum itself, with links to video recordings of all speeches and remarks.
The Forum brought together around 30 distinguished experts from around the world, including Turing Award winner Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of computer science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director GAO Wen (高文); CEO of the Partnership on AI, Rebecca Finlay; Shanghai Artificial Intelligence Strategic Advisory Expert Committee member, HE Jifeng (何积丰); and many more leading figures from government, industry, and research. Over 200 audience members joined in person, with over 14,000+ views of the livestream.
The forum was structured into four themes:
-
- Theme 1: The Science of AI Safety
- Theme 2: Emerging Challenges in AI Safety
- Theme 3: AI Risk Management in Practice
- Theme 4: International Governance of AI Safety
Group photo after the AI Safety and Governance Forum morning session.
Opening remarks
Concordia AI founder and CEO Brian TSE (谢旻希) gave a welcoming speech and shared four points to spark discussion. First, scientific consensus is the premise for driving AI safety research and governance. Second, we should urgently enhance risk monitoring and early warning due to the multifaceted challenges arising from cutting-edge large models. Third, AI safety needs to draw on global best practices in risk management. Fourth, AI safety is a challenge faced by all of humanity and requires global cooperation.
Theme 1: The Science of AI Safety
The opening speech was delivered by GAO Wen (高文), Academician of the Chinese Academy of Engineering and Director of Peng Cheng Laboratory. Gao noted that while the rapid development of AI creates immense opportunities, it also introduces uncontrollable security risks. His keynote centered on two key issues: compute sovereignty and trustworthy data sharing. He emphasized the importance of securing the foundations of compute, and highlighted Peng Cheng Laboratory’s work on privacy-preserving computation and data-sharing technologies, which enable data utilization while safeguarding privacy and security.
Stuart Russell, distinguished Professor of computer science at UC Berkeley, warned about AI systems exhibiting self-preservation and deception behaviors. He cautioned that the current AI development paradigm poses significant risks of catastrophic outcomes such as deception, self-replication, and loss of control. He called for setting red lines, increasing transparency, and establishing more stringent regulatory mechanisms, including hardware-enabled governance. He also proposed using “assistance games” — where AI systems are trained through collaboration with humans — to ensure AI systems serve human interests even when those interests are not precisely defined.
Turing Award winner Yoshua Bengio, founder and scientific director of Mila – Quebec Artificial Intelligence Institute, warned of the potential catastrophic risks of superintelligence. He observed that cutting-edge AI systems are already approaching human expert levels in multiple domains and may soon possess dangerous behaviors such as deception and autonomous replication. He introduced the International AI Safety Report and called for establishing a bridge between scientific evidence and policy. He proposed developing “scientist AI” — non-autonomous systems that cannot independently pursue goals but instead provide research assistance. Bengio stressed the importance of international cooperation on AI safety, warning that if major powers such as the US and China treat AI development as a race, competitive pressures could ultimately harm everyone.
ZHOU Bowen (周伯文), Director and Chief Scientist of Shanghai AI Lab, highlighted the limitations of traditional AI safety approaches such as value alignment and red teaming. He argued that while these methods can address short-term challenges, they prove insufficient for managing long-term risks, particularly those posed by AI agents that may surpass human intelligence. Building on the “AI-45° Law” he proposed at WAIC 2024, Zhou emphasized the need to shift from “Making AI Safe” to “Making Safe AI” — embedding safety as a core property of AI systems rather than adding it on as a “patch” after development. He introduced Shanghai AI Lab’s SafeWork safety technology stack, which is designed around this principle.
Academician ZHANG Ya-Qin (张亚勤), Dean of Tsinghua University’s Institute for AI Industry Research, joined Academician Gao Wen and Professor Stuart Russell for a panel discussion, moderated by Concordia AI CEO Brian Tse. The conversation addressed frontier AI trends and early warning indicators, the co-evolution of digital and biological intelligence, strategies for managing high-severity but low-probability risks, and future pathways for global AI safety. The experts recommended introducing hardware-level safety mechanisms, creating a global AI safety research fund, implementing AI agent identity registration systems, and establishing regulations for “emergency shutdown” mechanisms.
Theme 2: Emerging Challenges in AI Safety
UC Berkeley Professor Dawn Song discussed the profound impact of frontier AI on cybersecurity, noting how it is transforming both offense and defense. On one hand, AI is being applied to identify and mitigate vulnerabilities, with performance in vulnerability detection reflected by benchmarks such as BountyBench and CyberGame. On the other hand, attackers can also exploit AI to carry out more sophisticated attacks, creating an asymmetry that favors offense over defense. She emphasized the need to enhance AI’s effectiveness for cyberdefense through system design, proactive defense, and formal verification.
Professor Nick Bostrom, Principal Researcher at the Macrostrategy Research Initiative and the author of Superintelligence, outlined four core challenges in machine intelligence: scalable AI alignment, AI governance, the moral status of digital minds, and intra-superintelligence cooperation. He noted that the ethics of digital minds remains especially neglected, as most people still do not take the issue seriously. Bostrom examined several attributes that might shape whether AI warrants moral consideration, including sentience, agency, potential, and modal status. He concluded by emphasizing that this field is still in its early stages and called for deeper research across technical, philosophical, and institutional dimensions.
Professor YANG Min (杨珉), Executive Dean and Professor of the School of Computing and Intelligence Innovation at Fudan University, argued that frontier AI poses a range of security challenges, including misuse in cybersecurity or CBRN (chemical, biological, radiological, and nuclear) domains, as well as risks of deception, self-replication, and self-improvement. He presented his team’s research showing that AI systems can recognize when they are being evaluated and adjust their behavior to appear safer. His team also found that several mainstream models already demonstrate early signs of self-replication. These results suggest that AI may be approaching a tipping point toward loss of control, warranting strengthened risk assessment and governance efforts.
Professor ZHANG Weiwen (张卫文), Baiyang Chair Professor and Director of the Center for Biosafety Research and Strategy at Tianjin University, examined the risks and opportunities arising from the integration of AI and the life sciences. He noted that biosecurity faces growing risks such as synthetic viruses and artificial bacteria, which traditional laboratory controls are unable to fully address. He warned that rapid AI progress could generate entirely new and unknown biological knowledge, creating more complex safety challenges. Zhang shared his Center’s international cooperation initiatives, such as the UN-recognized Tianjin Biosecurity Guidelines. He called for transnational collaboration among scientists to establish a dynamic and practical global biosecurity system.
Concordia AI and the Center for Biosafety Research and Strategy of Tianjin University released a report in Chinese titled Responsible Innovation in AI x Life Sciences. Concordia AI’s Head of AI Safety and Governance (China), FANG Liang (方亮), introduced the key findings. The report highlights the positive role of AI in advancing life sciences research and biosecurity governance. It also identifies three main categories of risks (accidental, misuse, and structural) and points out shortcomings in existing risk analysis and evaluation systems. The report further reviews governance practices across a range of domestic and international actors, including governments, research institutions, and enterprises.
Dr. Jaime Yassif, Vice President of the Nuclear Threat Initiative’s Global Biological Policy and Programs, emphasized that while AI can accelerate vaccine development and enhance biopharmaceutical capabilities, it also carries risks of misuse, such as enabling the creation of more dangerous pathogens or undermining biodefense systems. She called on policymakers, AI developers, and funders to increase investment in safety guardrails and incentivize safety practices for AIxBio tools. Yassif shared a regularly updated research agenda for AIxBio safeguards. She also introduced the AIxBio Global Forum, which aims to develop shared understanding of risks, improve safety practices, and promote governance mechanisms for AI usage in biology.
Dan Hendrycks, Director of the Center for AI Safety and Safety Advisor at xAI, and YU Xuedong (于学东), Deputy Director of Guangzhou Laboratory’s ABSL-3 Laboratory, joined Professor Zhang Weiwen and Dr. Jaime Yassif for a panel discussion, moderated by Concordia AI CEO Brian Tse. The dialogue focused on benefits and potential risks of the AI–life sciences convergence. The panelists recommended strengthening risk prevention and control across multiple layers, including AI models, biological design tool management, model access, and DNA screening mechanisms. They emphasized the need to establish clear technical and governance standards and avoid harmful competition. Finally, they called on global experts to work together to set norms and build robust defense mechanisms for AI in biology.
Amandeep Singh Gill, United Nations Under-Secretary-General and Special Envoy for Digital and Emerging Technologies, delivered a keynote speech. He indicated that global AI governance is entering a critical stage: moving from principles to practice, where details and implementation matter most. He emphasized that the UN, as the core platform for international law and governance, plays a vital role in advancing the implementation of related agreements. Gill called on multiple stakeholders, including private enterprises, civil society, and the technical community, to work together, build consensus, and promote compliance.
Theme 3: AI Risk Management in Practice
HAO Chunliang (郝春亮), Director of the China Electronics Standardization Institute (CESI) Cybersecurity Center AI Safety/Security Department presented the TC260 AI Safety Governance Framework and related standardization efforts. The framework analyzes AI safety risks along two dimensions — inherent and application-related — and proposes both technical and governance mitigation measures. In January 2025, TC260 also published the AI Safety Standards System (V1.0) – Draft for Comments, covering key technologies, security management, product applications, and testing and evaluation, with ongoing improvements based on broad feedback. Hao discussed the release of three national standards on generative AI security in April and China’s first mandatory national AI standard on labeling AI-generated synthetic content. Additionally, he outlined ongoing work on forthcoming standards for AI code generation security, AI agent security, and risk classification and grading.
Rishi Bommasani, Society Lead at the Stanford Center for Research on Foundation Models, emphasized California’s critical role in AI safety and shared insights from the California Report on Frontier AI Policy, which he co-authored. He reflected on lessons relevant to AI governance, including how early design choices create path dependencies, the central importance of transparency, and the need for independent verification of industry claims. He shared recommendations from the report, including information disclosure, whistleblower protections, third-party risk assessments, and post-deployment incident reporting.
This session started with three lightning talks by industry representatives:
- Dan Hendrycks, Safety Advisor at xAI, shared insights from the company’s Draft AI Risk Management Framework. xAI mitigates malicious use risks through measures including access management and filtering methods, with particular attention to threats in the cyber and CBRN domains. The framework also addresses loss of control through measures including monitoring for deceptive tendencies.
- FU Hongyu (傅宏宇), AI Governance Lead & Director at Digital Economy Research Center at Alibaba Research Institute, emphasized Alibaba’s commitment to open-source AI, highlighting its transparency benefits while recognizing its unique risks. He outlined Alibaba’s security pipeline covering data, processing, resource management, and automated safety tests. He further emphasized institutional safeguards, such as the establishment of a technology ethics review system in 2021.
- BAO Chenfu (包沉浮), Outstanding Architect and Chairman of the Safety/Security Technology Committee at Baidu, emphasized that traditional security methods fall short for AI. He outlined Baidu’s lifecycle-based, defense-in-depth approach and highlighted its active role in industry standards and self-regulation.
Following the lightning talks, Rebecca Finlay, CEO of the Partnership on AI, joined a panel discussion with the three corporate representatives, moderated by Concordia AI International AI Governance Senior Research Manager Jason Zhou. They explored three key pillars: transparency in technical disclosure and regulatory alignment, organization-level ethical governance mechanisms, and the pros and cons of voluntary agreements versus binding regulation for achieving compliance. Panelists agreed that voluntary commitments offer flexibility in addressing uncertainty and unknown risks but should be augmented by more comprehensive measures, including increased transparency, internal governance mechanisms, and future legislation.
Shanghai AI Lab, in partnership with Concordia AI, released the Frontier AI Risk Management Framework v1.0. AI Safety Research Manager at Concordia AI, DUAN Yawen (段雅文) and Shanghai AI Lab Research Scientist Dr. SHAO Jing (邵婧) introduced the framework. It is China’s first comprehensive framework for managing severe risks from general-purpose AI models. Alongside the Framework, Shanghai AI Lab released a risk assessment report, which Concordia AI co-authored. We covered both documents in a previous Substack post.
In this panel, YANG Xiaofang (杨小芳), LLM Security Director at Ant Group; GONG Xiao (巩潇), Deputy Director of the China Software Testing Center; and Professor Robert Trager, Founding Director of the Oxford Martin AI Governance Initiative, moderated by Concordia AI AI Safety and Governance Senior Manager CHENG Yuan (程远), discussed three topics: corporate practice, third-party evaluation, and policy research. The discussion highlighted key challenges across the AI lifecycle, including risk identification, assessment, mitigation, and governance. The panel stressed that enterprises must go beyond technical solutions by strengthening organizational mechanisms and talent development. At the international level, they emphasized the need for consensus and incentive mechanisms to support the creation of a global AI risk governance framework.
Theme 4: International Governance of AI Safety
Anne Bouverot, Special Envoy of the President of the Republic of France for AI, reviewed the outcomes of the 2025 Paris AI Action Summit. She emphasized the launch of a foundation for developing public interest AI and also called for greater attention to AI sustainability issues, including energy consumption and environmental impact. Bouverot highlighted Europe’s investments and commitments in AI infrastructure and governance, emphasizing that trust and safety are central to enabling AI deployment. She concluded with a call for global collaboration to jointly promote safe and sustainable AI development.
Wan Sie Lee, Cluster Director (AI Governance and Safety) at Singapore’s Infocomm Media Development Authority (IMDA), introduced Singapore’s practices and international collaboration experience in global AI safety governance. She emphasized advancing safe and responsible AI through research, guidelines and tools, and global norms. She highlighted the Singapore Consensus and its defence-in-depth approach to AI safety, covering evaluations, safety techniques, and post-deployment control. In addition, she shared models for international collaboration such as joint testing exercises and cross-border red-teaming. She stressed that standard-setting and practical implementation must go hand in hand, with a necessity for interoperable international standards.
Concordia AI launched the State of AI Safety in China (2025) and State of AI Safety in Singapore reports. Kwan Yee NG (吴君仪), Head of International AI Governance at Concordia AI, introduced key findings from both reports.
The final panel welcomed Lucia Velasco, AI Policy Lead at the UN Office for Digital and Emerging Technologies; Benjamin Prud’homme, Vice-President of Policy, Safety and Global Affairs at Mila; FU Huanzhang (傅焕章), Assistant Director of the INTERPOL Innovation Centre; and GONG Ke (龚克), Executive Director of Chinese Institute of New Generation Artificial Intelligence Development Strategies, moderated by Concordia AI Head of International AI Governance Kwan Yee Ng. They discussed AI governance at the UN, translating scientific consensus into action, international law enforcement cooperation, and global safety red lines. They stressed that the scientific community must communicate frontier risks in accessible policy language to encourage broad participation and foster mutual trust. For law enforcement, they highlighted the importance of establishing rapid, cross-border cooperation mechanisms to respond to catastrophic risks in a timely and effective manner. The panelists also underscored the importance of enhancing AI literacy among the public and practitioners, and of building international dialogue mechanisms.
Closing Address
HE Jifeng (何积丰), Academician of the Chinese Academy of Sciences and a Member of the Shanghai Artificial Intelligence Strategic Advisory Expert Committee, delivered the Forum’s closing address. He pointed out that rapid AI development has brought unprecedented governance challenges. The core issue is how to harness superintelligence while ensuring human control and safety when machines are more intelligent than humans.
Referencing insights of earlier speakers, Academician He proposed researching technical interpretability, applying mathematical methods for modeling and reasoning, and ensuring the robustness and reliability of hardware and software systems in safety-critical applications. At the same time, he stressed the importance of building a multidimensional, multilayered governance framework encompassing international governance structures, safety verification methods, and social resilience.
He concluded by calling for recognition that safety governance is a fundamental safeguard, not an obstacle, to AI development. Only when society has full trust in these systems and embraces the outcomes of AI can the technology achieve explosive growth.
Group photo with guests and audience after the AI Safety and Governance Forum afternoon session.
Media Mentions
Media coverage of Concordia AI’s WAIC Forum, forum guests, or Concordia AI reports published at WAIC include:
- Bloomberg, China Vies to Unseat US in Fight for $4.8 Trillion AI Market, July 30, 2025. The article included a table titled “China Sees Safety as Core Element of Its AI Strategy” with insights on domestic, international, technical, and industry developments based on Concordia AI’s State of AI Safety in China (2025) report.
- Wired, Inside the Summit Where China Pitched Its AI Agenda to the World, July 31, 2025. The article mentioned Concordia AI’s AI Safety and Governance Forum, cited insights from our State of AI Safety in China (2025) report, and quoted Concordia AI CEO Brian Tse as saying: “You could literally attend AI safety events nonstop in the last seven days,” adding that because US and Chinese frontier models are “trained on the same architecture and using the same methods of scaling laws, the types of societal impact and the risks they pose are very, very similar.”
- Caixin, Elon Musk’s xAI Safety Advisor: U.S., China, and Europe Should Seek “Unity in Diversity” in AI Regulation, July 30, 2025. The article interviewed Dan Hendrycks and mentioned his participation in Concordia AI’s WAIC Forum. It also discussed the Shanghai AI Lab and Concordia AI Frontier AI Risk Management Framework.
- IT Times, Over 10 large models already possess “self-replication” capabilities, July 29, 2025. The article reported on speeches by Academician He Jifeng, Academician Gao Wen, Dean Yang Min, Director Zhou Bowen, and Professor Dawn Song, as well as the Shanghai AI Lab and Concordia AI Frontier AI Risk Management Framework.
- Tech Review Africa, UN Digital Envoy concludes China visit, advocates for inclusive AI governance, August 4, 2025. This article reported on UN Under-Secretary-General Amandeep Singh Gill’s trip to WAIC, including participation in the Concordia AI forum.
- The People’s Daily, Jointly Promote AI Development and Governance, July 31, 2025. This article mentioned Concordia AI’s State of AI Safety in China (2025) report and interviewed NTI’s Jaime Yassif.