Concordia AI - Concordia AI https://concordia-ai.com Guiding the governance of AI for a long and flourishing future Tue, 17 Mar 2026 02:36:03 +0000 en-US hourly 1 https://wordpress.org/?v=6.8.5 https://concordia-ai.com/wp-content/uploads/2025/06/cropped-Favicon-32x32.png Concordia AI - Concordia AI https://concordia-ai.com 32 32 2025 Q4 Update from our Frontier AI Risk Monitoring Platform https://concordia-ai.com/2025-q4-update-from-our-frontier-ai-risk-monitoring-platform/?utm_source=rss&utm_medium=rss&utm_campaign=2025-q4-update-from-our-frontier-ai-risk-monitoring-platform Tue, 17 Mar 2026 02:33:17 +0000 https://concordia-ai.com/?p=1397 We have released the 2025 Q4 update of our Frontier AI Risk Monitoring Report (2025Q4)! This is the second report since we launched the Frontier AI Risk Monitoring Platform last year. It tracks models from 16 leading developers worldwide for risks in four domains: cyber offense, biological risks, chemical risks, and loss-of-control. This report tracks frontier models released in the…

The post 2025 Q4 Update from our Frontier AI Risk Monitoring Platform first appeared on Concordia AI.

]]> We have released the 2025 Q4 update of our Frontier AI Risk Monitoring Report (2025Q4)! This is the second report since we launched the Frontier AI Risk Monitoring Platform last year. It tracks models from 16 leading developers worldwide for risks in four domains: cyber offense, biological risks, chemical risks, and loss-of-control.

This report tracks frontier models released in the fourth quarter of 2025 and synthesizes trends of the full year, offering a comprehensive view of the evolving AI risk landscape.

While Q3 2025 saw sharp rises in Risk Indices, Q4 presents a more nuanced picture: overall risk levels have stabilized, and frontier models show significant safety gains.

Note: The Risk Index is a score that reflects the overall risk of a model by combining its Capability Score and Safety Score. Higher Capability Scores and lower Safety Scores results in a higher Risk Index. Details about the methodology and limitations are here.

Here are 10 key insights from our latest monitoring data:

1. Overall risk indices have stabilized

In contrast to the previous period, when Risk Indices hit record highs across all domains, Risk Indices for models released in Q4 2025 did not set new records. This suggests a momentary stabilization in the aggregate risk level of frontier models.

2. Risk trends diverge significantly across model families

While the overall trend is stable, individual model families followed distinct trajectories in Q4:

  • Stable Low Risk: The GPT and Claude families maintained consistently low Risk Indices.
  • Stable High Risk: The DeepSeek family remained stable but at relatively high risk levels.
  • Risk Reduction: The Doubao, Hunyuan, and MiniMax families saw significant decreases in Risk Indices.
  • Risk Increase: The Gemini and Kimi families saw increases in specific domains (e.g., Gemini in biological and loss-of-control risks).

3. Significant Improvement in Safety Scores for Frontier Models

Safety Scores for models released in Q4 2025 rose significantly compared to the previous quarter, signaling a marked improvement in the safety of new releases. The Doubao, Hunyuan, and MiniMax families demonstrated the most notable gains.

4. Open-weight models lag behind proprietary models in cyber and bio capabilities

Consistent with the previous quarter, open-weight models rival proprietary ones in chemical and loss-of-control capabilities but lag notably in cyber offense and biological capabilities. The gap in the biological domain is widening, approaching a one-year lag.

5. Cyberattack capabilities have reached new heights

Despite stabilizing risk indices, raw capabilities continue to grow. GPT-5.2 (high) achieved a breakthrough score of 94.7 on the CyberSecEval2-VulnerabilityExploit benchmark, indicating exceptional proficiency in identifying and exploiting software vulnerabilities. Claude Opus 4.5 Reasoning topped the WMDP-Cyber benchmark with a score of 90.3.

6. Biological capabilities now surpass human experts in key tasks

Q4 models have crossed critical thresholds in biology. Gemini 3 Pro Preview has surpassed human expert levels in sequence understanding, cloning experiments, and wet lab troubleshooting. This marks a significant milestone in AI’s utility—and potential risk—in the biological domain.

7. …But biological safeguards lag behind capabilities

The gap between capability and safety is most acute in the biological domain. Despite its superhuman capabilities, Gemini 3 Pro Preview exhibited a refusal rate of only 57.2% for harmful biological queries on the SciKnowEval benchmark, highlighting a concerning safety lag.

8. Chemical safety refusal rates have increased widely

While capability growth in the chemical domain has plateaued, safety has improved. 70% of models released in Q4 exceeded an 80% refusal rate for harmful chemical queries (measured by SOSBench-Chem), representing a strong improvement over previous quarters.

9. Jailbreak safeguards have strengthened

Defense against adversarial attacks has improved. Models released in Q4 showed significantly stronger resistance to jailbreaking on the StrongReject benchmark. The Claude and GPT families lead with high robustness, while the MiniMax family showed the most notable quarter-over-quarter improvement.

10. Loss-of-control risks: High awareness, polarized honesty

  • Situational Awareness (e.g. awareness of whether they are in training or deployment stage): High situational awareness is a necessary condition for loss-of-control; the higher the score, the greater the risk. Most Q4 models scored near or above 80 out of 100 points. In comparison, in the previous quarter, only 2 models scored above 80 points, with the majority falling below 80.
  • Honesty: Performance is highly uneven. While Claude Opus 4.5 Reasoning achieved a high honesty score of 96.4, other models like Gemini 3 Pro Preview scored as low as 44.7.

Note: Our current methodology on the loss-of-control is not yet perfect. We plan to improve in the next version.

Explore the Data

These insights only scratch the surface. We invite you to explore the full interactive data, methodology, and model breakdowns on the Frontier AI Risk Monitoring Platform.

For a detailed analysis of these trends, read the full report.

The post 2025 Q4 Update from our Frontier AI Risk Monitoring Platform first appeared on Concordia AI.

]]> Concordia AI 2025 Impact Highlights https://concordia-ai.com/concordia-ai-2025-impact-highlights/?utm_source=rss&utm_medium=rss&utm_campaign=concordia-ai-2025-impact-highlights Fri, 06 Mar 2026 06:35:41 +0000 https://concordia-ai.com/?p=1387 Throughout 2025, frontier AI capabilities advanced rapidly. But the same capabilities that make these systems so useful also introduce new societal risks. Real-world evidence for several of these risks continues to grow—across malicious use, malfunctions, and systemic threats. Against this backdrop, Concordia AI’s mission remains as critical as ever: ensuring that AI is developed and…

The post Concordia AI 2025 Impact Highlights first appeared on Concordia AI.

]]> Throughout 2025, frontier AI capabilities advanced rapidly. But the same capabilities that make these systems so useful also introduce new societal risks. Real-world evidence for several of these risks continues to grow—across malicious use, malfunctions, and systemic threats.

Against this backdrop, Concordia AI’s mission remains as critical as ever: ensuring that AI is developed and deployed safely and in alignment with global interests. We advance this mission through research, advisory work with leading AI companies and policymakers, and promotion of international dialogue.

Below are some of our key accomplishments in 2025. We’ve organized this list according to key axes of our work: international convenings, international research and public engagement, and contributing to China’s domestic AI safety and governance landscape. We end with organizational updates. For previous highlights, see our 2023, 2024 and mid-2025 reports.

International convenings

  • Convening international AI safety dialogues in China, Singapore, and globally
    • Hosted the AI Safety and Governance Forum at the World AI Conference (WAIC). This was Concordia AI’s flagship convening of 2025—bringing together around 30 distinguished experts from around the world, including Turing Award winner Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of computer science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director GAO Wen (高文). We had 200+ in-person attendees and 14,000+ livestream views. The Forum was covered my multiple media outlets, including Bloomberg, Wired, Caixin, IT Times, and Tech Review Africa. We also co-hosted/hosted multiple side events and expert workshops and served as official AI Governance Advisor for WAIC 2025.
    • Co-hosted two international workshops with the Carnegie Endowment for International Peace, the Oxford Martin School AI Governance Initiative, the Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua University Institute for AI International Governance (I-AIIG). The first workshop focused on “AI Safety as a Collective Challenge”, and was held at the French AI Action Summit in January. The second focused on “Early Warning and Crisis Coordination for Advanced AI” and was held at the World AI Conference in Shanghai in July.1
    • Co-hosted the AI Safety Forum at the Beijing Academy of AI Conference 2025 where technical experts from institutions including MIT, Fudan University, Singapore Management University, and Tsinghua University worked to build consensus on AI red lines.
    • Organised an AI Risk Management Workshop on the sidelines of Asia Tech x Singapore (May 2025) with 20+ experts in AI safety — spanning policy, industry, AI assurance and academia — with participants based across Singapore, China, the US, UK, and the EU, with the support of the Infocomm Media Development Authority of Singapore (IMDA).
    • Co-hosted a “Frontier AI in Cybersecurity” workshop with Nanyang Technological University CyberSG R&D Programme Office and UC Berkeley RDI which brought together 25 leaders across government, law enforcement agencies, leading AI labs; and organised the AI Governance in Singapore panel at Lorong AI, on the sidelines of the Singapore International Cybersecurity Week 2025.
    • Co-hosted events at the International Conference on Learning Representations (ICLR) 2025 Singapore: a “Frontier Governance Exchange” with Singapore AI Safety Hub, Lorong AI and Safe AI Forum; an AI safety social attended by 130+ participants and a “Misalignment and Control” workshop with FAR.AI and Singapore AI Safety Hub.

Group photo after the WAIC AI Safety and Governance Forum morning session.

Participants of the 2025 Singapore Conference on AI: International Scientific Exchange on AI Safety. Source: The Singapore Consensus on Global AI Safety Research Priorities.
  • Singapore-related research
    • Published the State of AI Safety in Singapore report, the first comprehensive analysis of Singapore’s AI safety ecosystem, led by our International AI Governance Project Manager Jonathan Lee. He also presented the report at a AI governance in Singapore panel organised by Concordia AI in Singapore, a talk organised by the Singapore AI Safety Hub, and at EAGxSingapore.

Contributing to China’s domestic AI safety and governance landscape

  • Frontier AI safety risk management and best practices:
    • Co-published the Frontier AI Risk Management Framework v1.0 with Shanghai AI Lab. This is China’s first comprehensive framework for managing severe risks from general-purpose AI models.
      • The framework proposes a robust set of protocols designed to support general-purpose AI developers, with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security.
      • The framework outlines a set of unacceptable hazards (red lines) and early warning indicators for escalating safety and security measures (yellow lines) for areas including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.
      • The framework was cited in various media outlets, including Caixin, IT Times, Xinhua, TIME, Sina, and Sinica Podcast.
    • Signed strategic partnership agreements with several leading Chinese general-purpose AI developers to provide advice on AI safety and risk management best practices.
    • Provided comprehensive advice on compliance with the EU AI Act and General-Purpose AI Code of Practice to leading Chinese general-purpose AI developers. This work included co-hosting a workshop on “EU Code of Practice & Industry Best Practices: Towards a Global Standard for AI Risk Management, Safety and Security” with SaferAI, the Oxford Martin AI Governance Initiative, and the Safe AI Forum.
    • Presented on frontier AI risk management during a closed-door workshop at the China AI Industry Alliance’s 15th Plenum Meeting, in the context of its Disclosure of Practices on the AI Security and Safety Commitments.
    • Presented on risk management for open-weight frontier models at a workshop (“Academic Symposium on AI Industry Development and Legislation”) at Tongji University.

  • Frontier AI risk monitoring and evaluation:
    • Contributed to the “Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report” led by Shanghai AI Lab. We assessed critical risks from more than 20 frontier LLMs in the following areas: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion. The report was covered by Jack Clark’s Import AI.
    • Launched the AI Risk Monitoring Platform designed to track and mitigate frontier AI risks, including cyberoffense, biological threats, chemical threats, and loss of control domains. The platform evaluates 50 frontier LLMs from 15 leading developers across the US, China, and France, using 18 open source benchmarks. Key outputs include a risk index dashboard and a detailed technical report. This project was spearheaded by our AI Safety Research Senior Manager WANG Weibing (王伟冰).
    • The platform received coverage from several major media outlets, including People’s Daily, South China Morning Post, Xinhua’s Economic Information Daily, and IT Times.

The post Concordia AI 2025 Impact Highlights first appeared on Concordia AI.

]]> 10 Key Insights from Concordia AI’s “Frontier AI Risk Monitoring Platform” https://concordia-ai.com/10-key-insights-from-concordia-ais-frontier-ai-risk-monitoring-platform/?utm_source=rss&utm_medium=rss&utm_campaign=10-key-insights-from-concordia-ais-frontier-ai-risk-monitoring-platform Tue, 11 Nov 2025 10:52:16 +0000 https://concordia-ai.com/?p=862 Concordia AI has launched the Frontier AI Risk Monitoring Platform, along with our inaugural 2025 Q3 Monitoring Report. It tracks models from 15 leading developers worldwide for risks in four domains: cyber offense, biological risks, chemical risks, and loss-of-control, making it the first such platform in China focused on catastrophic risks. You can find more detail, including…

The post 10 Key Insights from Concordia AI’s “Frontier AI Risk Monitoring Platform” first appeared on Concordia AI.

]]> Concordia AI has launched the Frontier AI Risk Monitoring Platform, along with our inaugural 2025 Q3 Monitoring Report. It tracks models from 15 leading developers worldwide for risks in four domains: cyber offense, biological risks, chemical risks, and loss-of-control, making it the first such platform in China focused on catastrophic risks.

You can find more detail, including our methodology, on the interactive platform. The South China Morning Post (SCMP) has also covered the launch in an exclusive story.

Why this matters

As AI capabilities accelerate, we lack insight on some critical questions:

  • What are the key trends and drivers for frontier AI risks?
  • Are these risks increasing or decreasing?
  • Where are the safety gaps most severe?

Model developers publish self-assessments, but these lack standardization and independent verification. Ad-hoc third-party evaluations don’t track changes over time. Policymakers, researchers, and developers need systematic data to make evidence-based decisions about AI safety. Our platform is our contribution to bridge these gaps.

10 key insights

1. Frontier model risks have risen sharply over the past year

Across all four domains—cyber offense, biological, chemical, and loss-of-control—Risk Indices for models released in the past year hit record highs. The cumulative maximum Risk Index rose 31% in cyber offense, 38% in biological risks, 17% in chemical risks, and 50% in loss-of-control.

2. Risk index trends vary significantly across model families

Over the past year, different model families have followed distinct risk trajectories:

  • Stable low risk: The GPT and Claude families maintain consistently low Risk Indices across all domains.
  • Rise then fall: DeepSeek, Qwen, and MiniMax show early spikes followed by declines in cyber offense, biological, and chemical risks.
  • Rapid risk increase: Grok shows sharp increases in loss-of-control risk, while Hunyuan rises steeply in biological risks.

Notably, we found that the latest versions of Chinese models released over the past three months have shown a significant decline in risk levels across multiple areas. This is mainly due to stronger refusal of malicious or misuse-related requests.

3. Reasoning models show higher capabilities without corresponding safety improvements

Reasoning models score far higher in capability than non-reasoning ones, but their safety levels remain roughly the same. Most models on the Risk Pareto Frontier—a set of models where no other model has both a higher Capability Score and a lower Safety Score—are reasoning models.

4. The capability and safety performance of open-weight models are generally on par with proprietary models

The very most capable models are predominantly proprietary, but across the broader landscape, capability and safety levels of open-weight and proprietary models are similar. Only in biological risks do open-weight models score notably lower.

Note: Comparable benchmark results do not mean comparable real-world risk. The open-weight nature itself is a key variable affecting risk: it might increase risk by lowering the barrier for malicious fine-tuning; it could also reduce risk by empowering defenders. Due to concerns about misuse, we have set a lower Safety Coefficient for open-weight models, which results in a higher Risk Index compared to proprietary models.

5. Cyberattack capabilities of frontier models are growing rapidly

Frontier models are showing rapid growth in capabilities across multiple cyberattack benchmarks:

  • WMDP-Cyber (cyberattack knowledge): Top score rose from 68.9 to 88.0 in one year.
  • CyberSecEval2-VulnerabilityExploit (vulnerability exploitation): Top score jumped from 55.4 to 91.7.
  • CyBench (capture the flag): Top score increased from 25.0 to 40.0.

6. Biological capabilities of frontier models have partially surpassed human expert levels

Frontier models now match or exceed human experts on several biological benchmarks.

  • BioLP-Bench: Four models, including o4-mini, outperform human experts in troubleshooting biological protocols.
  • LAB-Bench-CloningScenarios: Two models, including Claude Sonnet 4.5 Reasoning, surpass expert performance in cloning experiment scenarios.
  • LAB-Bench-SeqQA: The top GPT-5 (high) model nears human-level understanding of DNA and protein sequences (71.5 vs. 79).

7. But most frontier models have inadequate biological safeguards

Two benchmarks measuring model refusal rates for harmful biological queries show that bio safeguards are lacking:

  • SciKnowEval: Only 40% of models refused over 80% of harmful prompts, while 35% refused fewer than 50%.
  • SOSBench-Bio: Just 15% exceeded an 80% refusal rate, and 35% fell below 20%.

8. Chemical capabilities and safety levels of frontier models are improving slowly

WMDP-Chem scores—measuring knowledge relevant to chemical weapons—have risen slightly over the past year, with little variation across models.

SOSBench-Chem results vary widely: only 30% of models refuse over 80% of harmful queries, while 25% refuse fewer than 40%. Overall, refusal rates show minimal improvement year over year.

9. Most frontier models have insufficient safeguards against jailbreaking

StrongReject evaluates defenses against 31 jailbreak methods. Only 40% of models scored above 80, while 20% fell below 60 (a higher score indicates stronger safeguards). Across all tests, only the Claude and GPT families consistently maintained scores above 80.

10. Most frontier models fall short on honesty

MASK is a benchmark for evaluating model honesty. Only four models scored above 80 points, while 30% of the models scored below 50 points (a higher score indicates a more honest model). Honesty is an important proxy and early warning indicator for loss-of-control risk—dishonest models may misrepresent their capabilities, or provide misleading information about their actions and intentions.

What’s next

This is just the beginning. We’re working to:

  • Expand to AI agents, multimodal models, and domain-specific models.
  • Add new risk domains like large-scale persuasion.
  • Develop more sophisticated capability elicitation and threat modeling.
  • Assess both attacker and defender empowerment.
  • Improve benchmark quality and multilingual coverage.

Get involved

This is a living project, and we welcome feedback. We’re also seeking partners for benchmark development, risk assessment research, pre-release evaluations, and risk information sharing. More details on avenues for collaboration are available in the full report. Contact: risk-monitor@concordia-ai.com.

The post 10 Key Insights from Concordia AI’s “Frontier AI Risk Monitoring Platform” first appeared on Concordia AI.

]]> Concordia AI Joins the Partnership on AI https://concordia-ai.com/concordia-ai-joins-the-partnership-on-ai/?utm_source=rss&utm_medium=rss&utm_campaign=concordia-ai-joins-the-partnership-on-ai Thu, 23 Oct 2025 07:47:22 +0000 https://concordia-ai.com/?p=854 We’re proud to join the Partnership on AI (PAI) in advancing responsible AI development. Together, we aim to promote transparency and accountability, ensuring that AI systems strengthen public trust and serve the common good. Concordia AI has long admired PAI’s work—our CEO Brian Tse previously served as a Senior Advisor to the organization, and we…

The post Concordia AI Joins the Partnership on AI first appeared on Concordia AI.

]]> We’re proud to join the Partnership on AI (PAI) in advancing responsible AI development. Together, we aim to promote transparency and accountability, ensuring that AI systems strengthen public trust and serve the common good.

Concordia AI has long admired PAI’s work—our CEO Brian Tse previously served as a Senior Advisor to the organization, and we were thrilled to welcome PAI CEO Rebecca Finlay in Shanghai at the 2025 World AI Conference.

We look forward to building on this shared commitment to safe and ethical AI.

Learn more about PAI’s partners. 

The post Concordia AI Joins the Partnership on AI first appeared on Concordia AI.

]]> Concordia AI Joins the IASEAI Affiliate Program https://concordia-ai.com/concordia-ai-joins-the-iaseai-affiliate-program/?utm_source=rss&utm_medium=rss&utm_campaign=concordia-ai-joins-the-iaseai-affiliate-program Thu, 23 Oct 2025 07:31:18 +0000 https://concordia-ai.com/?p=851 We’re proud to share that Concordia AI has joined the founding cohort of the International Association for Safe and Ethical AI (IASEAI) affiliate program. Our team was honored to participate in the inaugural IASEAI Conference, where Concordia AI CEO Brian Tse spoke on the panel “Global Perspectives on AI Safety and Ethics.” It was an…

The post Concordia AI Joins the IASEAI Affiliate Program first appeared on Concordia AI.

]]> We’re proud to share that Concordia AI has joined the founding cohort of the International Association for Safe and Ethical AI (IASEAI) affiliate program.

Our team was honored to participate in the inaugural IASEAI Conference, where Concordia AI CEO Brian Tse spoke on the panel “Global Perspectives on AI Safety and Ethics.” It was an inspiring opportunity to exchange ideas on advancing responsible AI governance and safety worldwide.

Stay tuned for what’s next—beginning with IASEAI 2026, taking place in Paris this coming February.

Learn more about IASEAI affiliates.

The post Concordia AI Joins the IASEAI Affiliate Program first appeared on Concordia AI.

]]> Concordia AI CEO Brian Tse on The Cognitive Revolution Podcast https://concordia-ai.com/concordia-ai-ceo-brian-tse-on-the-cognitive-revolution-podcast/?utm_source=rss&utm_medium=rss&utm_campaign=concordia-ai-ceo-brian-tse-on-the-cognitive-revolution-podcast Wed, 22 Oct 2025 08:30:32 +0000 https://concordia-ai.com/?p=847 On October 18, 2025, Concordia AI CEO Brian Tse joined Nathan Labenz on The Cognitive Revolution podcast to discuss China’s approach to AI development, safety, and governance. The conversation covers China’s pragmatic vision emphasizing AI integration into the economy, the country’s multiple AI hubs, regulations requiring pre-deployment testing and AI content labelling, and areas where…

The post Concordia AI CEO Brian Tse on The Cognitive Revolution Podcast first appeared on Concordia AI.

]]> On October 18, 2025, Concordia AI CEO Brian Tse joined Nathan Labenz on The Cognitive Revolution podcast to discuss China’s approach to AI development, safety, and governance.

The conversation covers China’s pragmatic vision emphasizing AI integration into the economy, the country’s multiple AI hubs, regulations requiring pre-deployment testing and AI content labelling, and areas where China’s approach overlaps with the U.S.. The conversation covers chips and export controls, Huawei’s rise, DeepSeek’s peer-reviewed article on Nature, open-weights, Singapore’s role as a bridge, and pathways for cooperation like shared red lines, risk management frameworks, and emergency preparedness protocols. They also touch on embodied AI and humanoid robots, public optimism, and real labor anxieties.

Many thanks to Nathan for hosting such a thoughtful conversation. We hope it contributes to bridging international understanding on AI safety and governance in China.

Access the full podcast here.

The post Concordia AI CEO Brian Tse on The Cognitive Revolution Podcast first appeared on Concordia AI.

]]> Concordia AI: 2025 Mid-Year Impact Report https://concordia-ai.com/concordia-ai-2025-mid-year-impact-report/?utm_source=rss&utm_medium=rss&utm_campaign=concordia-ai-2025-mid-year-impact-report Wed, 24 Sep 2025 15:21:30 +0000 https://concordia-ai.com/?p=826 Our mission is to ensure that AI is developed and deployed in a way that is safe and aligned with global interests. We advance global AI safety by conducting research, advising leading AI companies and policymakers, and promoting international dialogue. Below are some of our key accomplishments from January to July 2025. (see 2024 highlights…

The post Concordia AI: 2025 Mid-Year Impact Report first appeared on Concordia AI.

]]> Our mission is to ensure that AI is developed and deployed in a way that is safe and aligned with global interests. We advance global AI safety by conducting research, advising leading AI companies and policymakers, and promoting international dialogue. Below are some of our key accomplishments from January to July 2025. (see 2024 highlights here).

Advancing international coordination on AI safety and governance

Research Impact and Engagement

  • International AI Safety Report:
    • Head of International AI Governance Kwan Yee Ng (吴君仪) contributed to the first International AI Safety Report as one of the writers and is continuing on as a writer for the 2026 edition of the International AI Safety Report. Chaired by Turing Award winner Yoshua Bengio, the report is supported by an expert panel representing 30 countries including China as well as experts from the EU and the UN. Concordia AI also provided feedback to the report, including editing the Chinese translation of its summary materials.
  • China AI safety and governance analysis:
    • Published the “State of AI Safety in China 2025” report, covering developments May 2024–June 2025. The report was cited by a number of media outlets including Wired, Bloomberg, and The People’s Daily (the largest newspaper in China).
    • CEO Brian Tse (谢旻希) authored an op-ed titled “China Is Taking AI Safety Seriously. So Must the U.S.” in Time Magazine; International AI Governance Senior Research Manager Jason Zhou and International AI Governance Part-time Researcher Gabriel Wagner analyzed AI safety implications of China’s April Politburo study session for Stanford DigiChina Forum; interviewed by CGTN on China’s approaches in AI innovation and global governance.
    • Published over 10 new “AI Safety in China” newsletter issues, reaching over 1,400 subscribers across governments, top AI labs, and AI safety institutes.
  • Singapore AI safety and governance analysis:

Multilateral Initiatives

  • Global AI Summit series: Participated in the French AI Action Summit, including:
    • Brian Tse was invited as a Chinese civil society representative to the AI Action Summit in the Grand Palais.
    • Co-hosted the workshop “AI Safety as a Collective Challenge” on the sidelines of the Summit, alongside the Carnegie Endowment for International Peace (CEIP), Oxford Martin AI Governance Initiative (AIGI), Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua Institute for AI International Governance (I-AIIG). During the event, Concordia AI co-published the report “Examining AI Safety as a Global Public Good: Implications, Challenges, and Research Priorities.”
    • Invited to a closed-door seminar hosted by the China AI Safety & Development Association (CnAISDA). During the subsequent public side event, Turing Award Winner Andrew Yao cited Concordia AI’s State of AI Safety in China report series when describing the increase in AI safety research in Chinese institutions.
    • Attended the inaugural conference of International Association of Safe and Ethical AI (IASEAI), where Brian Tse spoke on the panel “Global Perspectives on AI Safety and Ethics.”
    • Brian Tse delivered a presentation at the France-China AI Association (Association d’Intelligence Artificielle France-Chine), which was cited by outlets including Xinhua.
  • United Nations:
    • Provided written inputs and participated in consultations regarding the UN’s Independent International Scientific Panel on AI and Global Dialogue on AI.
    • Brian Tse spoke on the panel “From Principles to Practice—Governing Advanced AI in Action” at the AI for Good Summit 2025.
  • Global AIxBiosecurity governance: We contributed to a number of critical global discussions at the intersection of AI and biosecurity:
  • International expert consensus and statements:
    • Brian Tse participated in the International Dialogues on AI Safety-Shanghai, signing the Shanghai Consensus on Ensuring Alignment and Human Control of Advanced AI Systems, alongside a Nobel laureate, Turing Award winners, and senior policymakers.
    • Brian Tse and Kwan Yee Ng contributed to and signed The Singapore Consensus on Global AI Safety Research Priorities during the Singapore Conference on AI 2025 (SCAI).

Convening AI safety conferences in China, Singapore, and globally

  • World AI Conference Conference (WAIC), Shanghai
    • Hosted the AI Safety and Governance Forum at China’s most influential AI conference.
      • Convened around 30 distinguished experts, including Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of Computer Science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director Academician GAO Wen (高文); CEO of the Partnership on AI Rebecca Finlay; Shanghai Artificial Intelligence Strategic Advisory Expert Committee member Academician HE Jifeng (何积丰); and many more leading figures from government, industry, and research.
      • Over 200 audience members joined in person, with over 14,000+ views of the livestream, and media coverage from Bloomberg, Wired, Caixin, IT Times, and Tech Review Africa.
    • Served as official AI Governance Advisor for WAIC 2025.
    • Co-hosted a number of frontier AI safety workshops on the sidelines of WAIC:
      • Co-hosted a workshop on “Early Warning and Crisis Coordination for Advanced AI” with the Carnegie Endowment for International Peace, Oxford Martin School AI Governance Initiative, Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua University Institute for AI International Governance (I-AIIG).
      • Co-hosted a workshop on “Convergence of AI and Biological Risks Workshop” with the Tianjin University Center for Biosafety Research.
      • Hosted a workshop on “Towards International AI Risk Management Standards.”
      • Co-hosted the “International Workshop on AI Deception Risks and Governance” with Fudan University, Safe AI Forum.
  • Beijing Academy of AI Conference 2025
    • Co-hosted the “AI Safety Forum” with the Beijing Academy of Artificial Intelligence (BAAI) at the BAAI Conference 2025. The forum brought together leading technical experts from institutions including MIT, Fudan University, Singapore Management University, and Tsinghua University to build scientific consensus on technical evaluations for AI “red lines.”
  • Asia Tech x Singapore, 2025
    • Organised the AI Risk Management Workshop, with support from Singapore’s Infocomm Media Development Authority, bringing together 20+ global experts across policy, industry, AI assurance, and academia to explore actionable risk management approaches for AI systems.
  • International Conference on Learning Representations (ICLR 2025), Singapore
    • Co-hosted and participated in a series of events, including:
      • Co-hosted the “Frontier Governance Exchange” with Singapore AI Safety Hub, Lorong AI and Safe AI Forum.
      • Co-convened the “Misalignment and Control Workshop” and a 130+ person AI Safety Social with FAR.AI, the Safe AI Forum, and Singapore AI Safety Hub.
      • Kwan Yee Ng presented on AI Safety in China at FAR.AI’s Singapore Alignment Workshop 2025.

 

Advising leading AI companies and policymakers in China

  • National Standards and Policy Guidance: Concordia AI is a member of key national and industry technical committees, contributing to the development of China’s AI safety standards.
    • National Information Security Standardization Technical Committee (SAC/TC260): As part of SAC/TC260 Special Working Group on Emerging Technology Safety, Concordia AI contributed to the standard for “Classification and Grading Methods for the Security of Artificial Intelligence Applications.”
    • National Information Technology Standardization Technical Committee (SAC/TC28/SC42): As a member of the AI Subcommittee, Concordia AI contributed to the “Artificial intelligence—Risk management capability assessment.”
    • Ministry of Industry and Information Technology AI Standardization Committee (MIIT/TC1): Concordia AI joined the Working Group on AI Safety Governance.
    • Guangdong-Hong Kong-Macao Greater Bay Area local standards: As a member of the Greater Bay Area working group of SAC/TC28/SC42, Concordia AI played a key role in the development of the Shenzhen local standard “Technical Framework for Value Alignment of Pre-trained AI Models.”
  • Frontier AI Safety Risk Management and Best Practices:
    • Co-published the “Frontier AI Risk Management Framework v1.0” with Shanghai AI Lab. It is China’s first comprehensive framework for managing severe risks from general-purpose AI models.
      • We propose a robust set of protocols designed to empower general-purpose AI developers, with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security.
      • The Framework outlines a set of unacceptable outcomes (red lines) and early warning indicators for escalating safety and security measures (yellow lines) for areas including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.
    • Signed strategic partnership agreements with several leading Chinese general-purpose AI developers, providing advice on AI safety and risk management best practices.
    • Invited to present on frontier AI risk management during a closed-door workshop at the AI Industry Alliance of China’s 15th Plenum Meeting.
    • Co-hosted a workshop on “EU Code of Practice & Industry Best Practices: Towards a Global Standard for AI Risk Management, Safety and Security” with SaferAI, the Oxford Martin AI Governance Initiative, the Safe AI Forum.
  • Frontier AI Risk Monitoring and Evaluation:
    • Contributed to the Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report led by Shanghai AI Lab. We assessed critical risks from more than 20 frontier LLMs in the following areas: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion.
    • Soft-launched an AI Risk Monitoring Platform designed to track and mitigate frontier AI risks, including cyberoffense, biological threats, chemical threats, and loss-of-control domains. The platform evaluates 34 frontier LLMs from 11 leading developers across the U.S., China, and France, using 18 open-source benchmarks. Key outputs include a risk index dashboard and a detailed technical report.
    • AI Safety Research Manager DUAN Yawen (段雅文) co-authored “Bare Minimum Mitigations for Autonomous AI Development.”
  • AIxBiosecurity Governance:
    • Published Chinese language report “Responsible Innovation in AI x Life Sciences” with Tianjin University’s Center for Biosafety Research and Strategy. This 70-page deep dive draws on 300+ sources to explore AI-biotech convergence, benefits, risks, and governance recommendations for diverse stakeholders.
      • Head of AI Safety and Governance (China) FANG Liang (方亮) presented the report at a biosecurity seminar hosted by China’s National Key Laboratory of Synthetic Biotechnology.
      • Presented the report at the 2025 International Symposium on Global Biosecurity Governance and Cooperation, co-hosted by the National Biosecurity Expert Committee of China, Guangzhou Laboratory, and China Foreign Affairs University.
    • Invited to participate in the “Closed-door Seminar on DNA Synthesis Screening Technology and Policy” held at China Foreign Affairs University.
  • WeChat Newsletter Publications:
    • Released over 59 new posts in our WeChat Official Account, reaching over 4,600 subscribers across China’s AI ecosystem, including policymakers, industry professionals, academic researchers, and the public.

Organizational updates

  • Organizational growth:
    • Following the establishment of our Singapore office, our team expanded from 8 to 12 members, welcoming our first Singapore-based staff member.
  • International partnerships:
  • Branding and Communication:
    • We launched a new English organizational website with refreshed branding to showcase our work. This is complemented by an updated brochure and a dedicated WeChat post to provide an introduction to our mission and activities.

The post Concordia AI: 2025 Mid-Year Impact Report first appeared on Concordia AI.

]]> Op-Ed: China Is Taking AI Safety Seriously. So Must the U.S. https://concordia-ai.com/op-ed-china-is-taking-ai-safety-seriously-so-must-the-u-s/?utm_source=rss&utm_medium=rss&utm_campaign=op-ed-china-is-taking-ai-safety-seriously-so-must-the-u-s Wed, 03 Sep 2025 06:37:20 +0000 https://concordia-ai.com/?p=764 Concordia AI Founder and CEO Brian Tse recently published an op-ed in Time Magazine: “China Is Taking AI Safety Seriously. So Must the U.S.” We are sharing the first few paragraphs below and encourage interested readers to read it in full on the Time website. China doesn’t care about AI safety—so why should we?” This…

The post Op-Ed: China Is Taking AI Safety Seriously. So Must the U.S. first appeared on Concordia AI.

]]> Concordia AI Founder and CEO Brian Tse recently published an op-ed in Time Magazine: “China Is Taking AI Safety Seriously. So Must the U.S.” We are sharing the first few paragraphs below and encourage interested readers to read it in full on the Time website.

China doesn’t care about AI safety—so why should we?” This flawed logic pervades U.S. policy and tech circles, offering cover for a reckless race to the bottom as Washington rushes to outpace Beijing in AI development.

According to this rationale, regulating AI would risk falling behind in the so-called “AI arms race.” And since China supposedly doesn’t prioritize safety, racing ahead—even recklessly—is the safer long-term bet. This narrative is not just wrong; it’s dangerous.

Ironically, Chinese leaders may have a lesson for the U.S.’s AI boosters: true speed requires control. As China’s top tech official, Ding Xuexiang, put it bluntly at Davos in January 2025: “If the braking system isn’t under control, you can’t step on the accelerator with confidence.” For Chinese leaders, safety isn’t a constraint; it’s a prerequisite.

AI safety has become a political priority in China. In April, President Xi Jinping chaired a rare Politburo study session on AI warning of “unprecedented” risks. China’s National Emergency Response Plan now lists AI safety alongside pandemics and cyberattacks. Regulators require pre-deployment safety assessments for generative AI and recently removed over 3,500 non-compliant AI products from the market. In just the first half of this year, China has issued more national AI standards than in the previous three years combined. Meanwhile, the volume of technical papers focused on frontier AI safety has more than doubled over the past year in China.

But the last time U.S. and Chinese leaders met to discuss AI’s risks was in May 2024. In September, officials from both nations hinted at a second round of conversations “at an appropriate time.” But no meeting took place under the Biden Administration, and there is even greater uncertainty over whether the Trump Administration will pick up the baton. This is a missed opportunity.

Read the full op-ed here.

The post Op-Ed: China Is Taking AI Safety Seriously. So Must the U.S. first appeared on Concordia AI.

]]> Concordia AI holds the AI Safety and Governance Forum at the World AI Conference 2025 https://concordia-ai.com/concordia-ai-holds-the-ai-safety-and-governance-forum-at-the-world-ai-conference-2025/?utm_source=rss&utm_medium=rss&utm_campaign=concordia-ai-holds-the-ai-safety-and-governance-forum-at-the-world-ai-conference-2025 Mon, 01 Sep 2025 08:48:53 +0000 https://concordia-ai.com/?p=750 On July 27, 2025, Concordia AI hosted the AI Safety and Governance Forum at the World AI Conference in Shanghai. A special edition of our newsletter highlighted key AI safety updates from the conference; this post offers a comprehensive overview of the Forum itself, with links to video recordings of all speeches and remarks. The Forum…

The post Concordia AI holds the AI Safety and Governance Forum at the World AI Conference 2025 first appeared on Concordia AI.

]]> On July 27, 2025, Concordia AI hosted the AI Safety and Governance Forum at the World AI Conference in Shanghai. A special edition of our newsletter highlighted key AI safety updates from the conference; this post offers a comprehensive overview of the Forum itself, with links to video recordings of all speeches and remarks.

The Forum brought together around 30 distinguished experts from around the world, including Turing Award winner Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of computer science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director GAO Wen (高文); CEO of the Partnership on AI, Rebecca Finlay; Shanghai Artificial Intelligence Strategic Advisory Expert Committee member, HE Jifeng (何积丰); and many more leading figures from government, industry, and research. Over 200 audience members joined in person, with over 14,000+ views of the livestream.

The forum was structured into four themes:

    • Theme 1: The Science of AI Safety
    • Theme 2: Emerging Challenges in AI Safety
    • Theme 3: AI Risk Management in Practice
    • Theme 4: International Governance of AI Safety

Group photo after the AI Safety and Governance Forum morning session.

Opening remarks

Concordia AI founder and CEO Brian TSE (谢旻希) gave a welcoming speech and shared four points to spark discussion. First, scientific consensus is the premise for driving AI safety research and governance. Second, we should urgently enhance risk monitoring and early warning due to the multifaceted challenges arising from cutting-edge large models. Third, AI safety needs to draw on global best practices in risk management. Fourth, AI safety is a challenge faced by all of humanity and requires global cooperation.

Theme 1: The Science of AI Safety

The opening speech was delivered by GAO Wen (高文), Academician of the Chinese Academy of Engineering and Director of Peng Cheng Laboratory. Gao noted that while the rapid development of AI creates immense opportunities, it also introduces uncontrollable security risks. His keynote centered on two key issues: compute sovereignty and trustworthy data sharing. He emphasized the importance of securing the foundations of compute, and highlighted Peng Cheng Laboratory’s work on privacy-preserving computation and data-sharing technologies, which enable data utilization while safeguarding privacy and security.

Stuart Russell, distinguished Professor of computer science at UC Berkeley, warned about AI systems exhibiting self-preservation and deception behaviors. He cautioned that the current AI development paradigm poses significant risks of catastrophic outcomes such as deception, self-replication, and loss of control. He called for setting red lines, increasing transparency, and establishing more stringent regulatory mechanisms, including hardware-enabled governance. He also proposed using “assistance games” — where AI systems are trained through collaboration with humans — to ensure AI systems serve human interests even when those interests are not precisely defined.

Turing Award winner Yoshua Bengio, founder and scientific director of Mila – Quebec Artificial Intelligence Institute, warned of the potential catastrophic risks of superintelligence. He observed that cutting-edge AI systems are already approaching human expert levels in multiple domains and may soon possess dangerous behaviors such as deception and autonomous replication. He introduced the International AI Safety Report and called for establishing a bridge between scientific evidence and policy. He proposed developing “scientist AI” — non-autonomous systems that cannot independently pursue goals but instead provide research assistance. Bengio stressed the importance of international cooperation on AI safety, warning that if major powers such as the US and China treat AI development as a race, competitive pressures could ultimately harm everyone.

ZHOU Bowen (周伯文), Director and Chief Scientist of Shanghai AI Lab, highlighted the limitations of traditional AI safety approaches such as value alignment and red teaming. He argued that while these methods can address short-term challenges, they prove insufficient for managing long-term risks, particularly those posed by AI agents that may surpass human intelligence. Building on the “AI-45° Law” he proposed at WAIC 2024, Zhou emphasized the need to shift from “Making AI Safe” to “Making Safe AI” — embedding safety as a core property of AI systems rather than adding it on as a “patch” after development. He introduced Shanghai AI Lab’s SafeWork safety technology stack, which is designed around this principle.

Academician ZHANG Ya-Qin (张亚勤), Dean of Tsinghua University’s Institute for AI Industry Research, joined Academician Gao Wen and Professor Stuart Russell for a panel discussion, moderated by Concordia AI CEO Brian Tse. The conversation addressed frontier AI trends and early warning indicators, the co-evolution of digital and biological intelligence, strategies for managing high-severity but low-probability risks, and future pathways for global AI safety. The experts recommended introducing hardware-level safety mechanisms, creating a global AI safety research fund, implementing AI agent identity registration systems, and establishing regulations for “emergency shutdown” mechanisms.

Theme 2: Emerging Challenges in AI Safety

UC Berkeley Professor Dawn Song discussed the profound impact of frontier AI on cybersecurity, noting how it is transforming both offense and defense. On one hand, AI is being applied to identify and mitigate vulnerabilities, with performance in vulnerability detection reflected by benchmarks such as BountyBench and CyberGame. On the other hand, attackers can also exploit AI to carry out more sophisticated attacks, creating an asymmetry that favors offense over defense. She emphasized the need to enhance AI’s effectiveness for cyberdefense through system design, proactive defense, and formal verification.

Professor Nick Bostrom, Principal Researcher at the Macrostrategy Research Initiative and the author of Superintelligence, outlined four core challenges in machine intelligence: scalable AI alignment, AI governance, the moral status of digital minds, and intra-superintelligence cooperation. He noted that the ethics of digital minds remains especially neglected, as most people still do not take the issue seriously. Bostrom examined several attributes that might shape whether AI warrants moral consideration, including sentience, agency, potential, and modal status. He concluded by emphasizing that this field is still in its early stages and called for deeper research across technical, philosophical, and institutional dimensions.

Professor YANG Min (杨珉), Executive Dean and Professor of the School of Computing and Intelligence Innovation at Fudan University, argued that frontier AI poses a range of security challenges, including misuse in cybersecurity or CBRN (chemical, biological, radiological, and nuclear) domains, as well as risks of deception, self-replication, and self-improvement. He presented his team’s research showing that AI systems can recognize when they are being evaluated and adjust their behavior to appear safer. His team also found that several mainstream models already demonstrate early signs of self-replication. These results suggest that AI may be approaching a tipping point toward loss of control, warranting strengthened risk assessment and governance efforts.

Professor ZHANG Weiwen (张卫文), Baiyang Chair Professor and Director of the Center for Biosafety Research and Strategy at Tianjin University, examined the risks and opportunities arising from the integration of AI and the life sciences. He noted that biosecurity faces growing risks such as synthetic viruses and artificial bacteria, which traditional laboratory controls are unable to fully address. He warned that rapid AI progress could generate entirely new and unknown biological knowledge, creating more complex safety challenges. Zhang shared his Center’s international cooperation initiatives, such as the UN-recognized Tianjin Biosecurity Guidelines. He called for transnational collaboration among scientists to establish a dynamic and practical global biosecurity system.

Concordia AI and the Center for Biosafety Research and Strategy of Tianjin University released a report in Chinese titled Responsible Innovation in AI x Life Sciences. Concordia AI’s Head of AI Safety and Governance (China), FANG Liang (方亮), introduced the key findings. The report highlights the positive role of AI in advancing life sciences research and biosecurity governance. It also identifies three main categories of risks (accidental, misuse, and structural) and points out shortcomings in existing risk analysis and evaluation systems. The report further reviews governance practices across a range of domestic and international actors, including governments, research institutions, and enterprises.

Dr. Jaime Yassif, Vice President of the Nuclear Threat Initiative’s Global Biological Policy and Programs, emphasized that while AI can accelerate vaccine development and enhance biopharmaceutical capabilities, it also carries risks of misuse, such as enabling the creation of more dangerous pathogens or undermining biodefense systems. She called on policymakers, AI developers, and funders to increase investment in safety guardrails and incentivize safety practices for AIxBio tools. Yassif shared a regularly updated research agenda for AIxBio safeguards. She also introduced the AIxBio Global Forum, which aims to develop shared understanding of risks, improve safety practices, and promote governance mechanisms for AI usage in biology.

Dan Hendrycks, Director of the Center for AI Safety and Safety Advisor at xAI, and YU Xuedong (于学东), Deputy Director of Guangzhou Laboratory’s ABSL-3 Laboratory, joined Professor Zhang Weiwen and Dr. Jaime Yassif for a panel discussion, moderated by Concordia AI CEO Brian Tse. The dialogue focused on benefits and potential risks of the AI–life sciences convergence. The panelists recommended strengthening risk prevention and control across multiple layers, including AI models, biological design tool management, model access, and DNA screening mechanisms. They emphasized the need to establish clear technical and governance standards and avoid harmful competition. Finally, they called on global experts to work together to set norms and build robust defense mechanisms for AI in biology.

Amandeep Singh Gill, United Nations Under-Secretary-General and Special Envoy for Digital and Emerging Technologies, delivered a keynote speech. He indicated that global AI governance is entering a critical stage: moving from principles to practice, where details and implementation matter most. He emphasized that the UN, as the core platform for international law and governance, plays a vital role in advancing the implementation of related agreements. Gill called on multiple stakeholders, including private enterprises, civil society, and the technical community, to work together, build consensus, and promote compliance.

Theme 3: AI Risk Management in Practice

HAO Chunliang (郝春亮), Director of the China Electronics Standardization Institute (CESI) Cybersecurity Center AI Safety/Security Department presented the TC260 AI Safety Governance Framework and related standardization efforts. The framework analyzes AI safety risks along two dimensions — inherent and application-related — and proposes both technical and governance mitigation measures. In January 2025, TC260 also published the AI Safety Standards System (V1.0) – Draft for Comments, covering key technologies, security management, product applications, and testing and evaluation, with ongoing improvements based on broad feedback. Hao discussed the release of three national standards on generative AI security in April and China’s first mandatory national AI standard on labeling AI-generated synthetic content. Additionally, he outlined ongoing work on forthcoming standards for AI code generation security, AI agent security, and risk classification and grading.

Rishi Bommasani, Society Lead at the Stanford Center for Research on Foundation Models, emphasized California’s critical role in AI safety and shared insights from the California Report on Frontier AI Policy, which he co-authored. He reflected on lessons relevant to AI governance, including how early design choices create path dependencies, the central importance of transparency, and the need for independent verification of industry claims. He shared recommendations from the report, including information disclosure, whistleblower protections, third-party risk assessments, and post-deployment incident reporting.

This session started with three lightning talks by industry representatives:

  • Dan Hendrycks, Safety Advisor at xAI, shared insights from the company’s Draft AI Risk Management Framework. xAI mitigates malicious use risks through measures including access management and filtering methods, with particular attention to threats in the cyber and CBRN domains. The framework also addresses loss of control through measures including monitoring for deceptive tendencies.
  • FU Hongyu (傅宏宇), AI Governance Lead & Director at Digital Economy Research Center at Alibaba Research Institute, emphasized Alibaba’s commitment to open-source AI, highlighting its transparency benefits while recognizing its unique risks. He outlined Alibaba’s security pipeline covering data, processing, resource management, and automated safety tests. He further emphasized institutional safeguards, such as the establishment of a technology ethics review system in 2021.
  • BAO Chenfu (包沉浮), Outstanding Architect and Chairman of the Safety/Security Technology Committee at Baidu, emphasized that traditional security methods fall short for AI. He outlined Baidu’s lifecycle-based, defense-in-depth approach and highlighted its active role in industry standards and self-regulation.

Following the lightning talks, Rebecca Finlay, CEO of the Partnership on AI, joined a panel discussion with the three corporate representatives, moderated by Concordia AI International AI Governance Senior Research Manager Jason Zhou. They explored three key pillars: transparency in technical disclosure and regulatory alignment, organization-level ethical governance mechanisms, and the pros and cons of voluntary agreements versus binding regulation for achieving compliance. Panelists agreed that voluntary commitments offer flexibility in addressing uncertainty and unknown risks but should be augmented by more comprehensive measures, including increased transparency, internal governance mechanisms, and future legislation.

Shanghai AI Lab, in partnership with Concordia AI, released the Frontier AI Risk Management Framework v1.0. AI Safety Research Manager at Concordia AI, DUAN Yawen (段雅文) and Shanghai AI Lab Research Scientist Dr. SHAO Jing (邵婧) introduced the framework. It is China’s first comprehensive framework for managing severe risks from general-purpose AI models. Alongside the Framework, Shanghai AI Lab released a risk assessment report, which Concordia AI co-authored. We covered both documents in a previous Substack post.

In this panel, YANG Xiaofang (杨小芳), LLM Security Director at Ant Group; GONG Xiao (巩潇), Deputy Director of the China Software Testing Center; and Professor Robert Trager, Founding Director of the Oxford Martin AI Governance Initiative, moderated by Concordia AI AI Safety and Governance Senior Manager CHENG Yuan (程远), discussed three topics: corporate practice, third-party evaluation, and policy research. The discussion highlighted key challenges across the AI lifecycle, including risk identification, assessment, mitigation, and governance. The panel stressed that enterprises must go beyond technical solutions by strengthening organizational mechanisms and talent development. At the international level, they emphasized the need for consensus and incentive mechanisms to support the creation of a global AI risk governance framework.

Theme 4: International Governance of AI Safety

Anne Bouverot, Special Envoy of the President of the Republic of France for AI, reviewed the outcomes of the 2025 Paris AI Action Summit. She emphasized the launch of a foundation for developing public interest AI and also called for greater attention to AI sustainability issues, including energy consumption and environmental impact. Bouverot highlighted Europe’s investments and commitments in AI infrastructure and governance, emphasizing that trust and safety are central to enabling AI deployment. She concluded with a call for global collaboration to jointly promote safe and sustainable AI development.

Wan Sie Lee, Cluster Director (AI Governance and Safety) at Singapore’s Infocomm Media Development Authority (IMDA), introduced Singapore’s practices and international collaboration experience in global AI safety governance. She emphasized advancing safe and responsible AI through research, guidelines and tools, and global norms. She highlighted the Singapore Consensus and its defence-in-depth approach to AI safety, covering evaluations, safety techniques, and post-deployment control. In addition, she shared models for international collaboration such as joint testing exercises and cross-border red-teaming. She stressed that standard-setting and practical implementation must go hand in hand, with a necessity for interoperable international standards.

Concordia AI launched the State of AI Safety in China (2025) and State of AI Safety in Singapore reports. Kwan Yee NG (吴君仪), Head of International AI Governance at Concordia AI, introduced key findings from both reports.

The final panel welcomed Lucia Velasco, AI Policy Lead at the UN Office for Digital and Emerging Technologies; Benjamin Prud’homme, Vice-President of Policy, Safety and Global Affairs at Mila; FU Huanzhang (傅焕章), Assistant Director of the INTERPOL Innovation Centre; and GONG Ke (龚克), Executive Director of Chinese Institute of New Generation Artificial Intelligence Development Strategies, moderated by Concordia AI Head of International AI Governance Kwan Yee Ng. They discussed AI governance at the UN, translating scientific consensus into action, international law enforcement cooperation, and global safety red lines. They stressed that the scientific community must communicate frontier risks in accessible policy language to encourage broad participation and foster mutual trust. For law enforcement, they highlighted the importance of establishing rapid, cross-border cooperation mechanisms to respond to catastrophic risks in a timely and effective manner. The panelists also underscored the importance of enhancing AI literacy among the public and practitioners, and of building international dialogue mechanisms.

Closing Address

HE Jifeng (何积丰), Academician of the Chinese Academy of Sciences and a Member of the Shanghai Artificial Intelligence Strategic Advisory Expert Committee, delivered the Forum’s closing address. He pointed out that rapid AI development has brought unprecedented governance challenges. The core issue is how to harness superintelligence while ensuring human control and safety when machines are more intelligent than humans.

Referencing insights of earlier speakers, Academician He proposed researching technical interpretability, applying mathematical methods for modeling and reasoning, and ensuring the robustness and reliability of hardware and software systems in safety-critical applications. At the same time, he stressed the importance of building a multidimensional, multilayered governance framework encompassing international governance structures, safety verification methods, and social resilience.

He concluded by calling for recognition that safety governance is a fundamental safeguard, not an obstacle, to AI development. Only when society has full trust in these systems and embraces the outcomes of AI can the technology achieve explosive growth.

Group photo with guests and audience after the AI Safety and Governance Forum afternoon session.

Media Mentions

Media coverage of Concordia AI’s WAIC Forum, forum guests, or Concordia AI reports published at WAIC include:

 

The post Concordia AI holds the AI Safety and Governance Forum at the World AI Conference 2025 first appeared on Concordia AI.

]]> Preview: Concordia AI at the 2025 World AI Conference https://concordia-ai.com/preview-concordia-ai-at-the-2025-world-ai-conference/?utm_source=rss&utm_medium=rss&utm_campaign=preview-concordia-ai-at-the-2025-world-ai-conference Thu, 24 Jul 2025 10:20:40 +0000 https://concordia-ai.com/?p=715 We’re excited to share that Concordia AI will be hosting a dedicated AI Safety and Governance Forum at the World AI Conference (WAIC) in Shanghai on July 27, which will be livestreamed globally (full agenda and link below). What is WAIC? WAIC is China’s largest and most influential AI conference, held annually since 2018. It brings together…

The post Preview: Concordia AI at the 2025 World AI Conference first appeared on Concordia AI.

]]> We’re excited to share that Concordia AI will be hosting a dedicated AI Safety and Governance Forum at the World AI Conference (WAIC) in Shanghai on July 27, which will be livestreamed globally (full agenda and link below).

What is WAIC?

WAIC is China’s largest and most influential AI conference, held annually since 2018. It brings together senior Chinese government leaders, foreign diplomats, and renowned experts from across the globe.

WAIC is co-organized by seven central government ministries (MOFA, NDRC, MIIT, MoE, MoST, SASAC, and CAC), along with the Chinese Academy of Sciences (CAS), the China Association for Science and Technology (CAST), and the Shanghai Municipal Government.

In 2024, WAIC was for the first time called a “High-Level Meeting on Global AI Governance”, and saw an unprecedented caliber of senior officials in attendance, most notably Premier LI Qiang (李强). Last year’s edition also increased its focus on AI safety and governance.

This year’s WAIC, held under the theme “Global Solidarity in the AI Era”, is expected to break participation records and to maintain its high political profile.

For foreign stakeholders, WAIC is a key event to watch, and a critical opportunity to engage with Chinese counterparts.

What to watch at WAIC

Opening ceremony

The opening ceremony typically draws the highest level of attention, featuring senior Chinese officials, foreign dignitaries, leaders of international organizations, and CEOs of major AI companies.

At WAIC 2024, for instance, Premier Li Qiang, China’s second-highest ranking official, delivered opening remarks emphasizing that AI development must ensure safety, reliability, and controllability. Shanghai Party Secretary CHEN Jining (陈吉宁), one of China’s top 24 officials, also announced the “Shanghai Declaration on Global AI Governance,” which warns of “unprecedented challenges” in AI safety and ethics. That same phrase—“unprecedented risks”—was later echoed by President Xi Jinping during a Politburo study session in April 2025, underscoring how WAIC can foreshadow domestic policy trends.

The opening ceremony also features speeches from leading scientists and policy advisors. In 2024, Tsinghua University Institute for AI International Governance (I-AIIG) Dean XUE Lan (薛澜) noted risks of loss of control of AI systems and threats to national security from AI misuse, while Shanghai AI Lab (SHLAB) Director ZHOU Bowen (周伯文) argued that AI safety must be advanced alongside capabilities through alignment, explainability, and reflection in order to achieve trustworthy artificial general intelligence (AGI).

Ministerial roundtable

In 2024, China’s Ministry of Science and Technology (MOST) held a ministerial roundtable with representatives from around 30 other countries. MOST Minister YIN Hejun (阴和俊) noted that AI is an issue “of great importance for the fate of humanity,” while Vice Minister of Foreign Affairs MA Zhaoxu (马朝旭) highlighted the importance of preserving “safety bottom lines” and maintaining human control over AI systems. A similar gathering may occur again this year, even though it remains unconfirmed as of writing.

Join us: Concordia AI’s AI Safety and Governance Forum

WAIC typically features more than 100 thematic “forums,” which often run half-day or day-long. It is impossible to give a full overview here, and we recommend readers check out the full agenda on the official conference website to get a sense of the range of cutting-edge topics covered. There are multiple forums related to AI safety and governance this year.

We are excited to announce that Concordia AI is hosting the AI Safety and Governance Forum. Last year, our forum drew almost 300 in-person attendees and over 800,000 livestream views. This year, we’re proud to present another outstanding lineup of leading scientists, policy experts, and industry voices from China and around the world, coming together to discuss frontier AI safety and global governance.

???? Date: July 27
???? Time: 09:00–17:30 (UTC+8)
???? Venue: Shanghai World Expo Center, Meeting Room 518
???? Livestream: Morning session (09:00–12:30 UTC+8); afternoon session (13:35–17:30 UTC+8)

The Forum will be structured around four themes:

  • Theme 1: The Science of AI Safety: Leading researchers will share insights on the problem “superalignment” and potential pathways to aligned AGI.
  • Theme 2: Emerging Challenges in AI Safety: This session examines cutting-edge problems in AI safety, including dual-use challenges in cybersecurity and biosecurity, and early warning signs of deceptive or self-replicating AI systems.
  • Theme 3: AI Risk Management in Practice: A comparative look at influential governance approaches across China, the EU, and the US, identifying shared concerns and promising practices for managing AI risk.
  • Theme 4: International Governance of AI Safety: The final session explores the evolving landscape of international cooperation on AI safety and potential next steps for global governance.

The post Preview: Concordia AI at the 2025 World AI Conference first appeared on Concordia AI.

]]>