Opinion

Technical Acceleration Methods for AI Safety: Summary from October 2025 Symposium

​Published on October 22, 2025 9:33 PM GMTWith AI capabilities advancing in several domains from elementary-school level (GPT-3, 2020) to beyond PhD-level (2025) in just five years, the AI safety field may face a critical challenge: developing and deploying effective solutions fast enough to manage catastrophic and existential risks from beyond-human level AI systems that may emerge on timelines shorter than we hope. On October 10, 2025, I organized a hybrid symposium bringing together 52 researchers and founders (25 in-person in NYC) to explore technical methods for accelerating AI safety progress. The focus of the event was not to discuss any single research safety agenda, but how to reduce catastrophic/existential risk from near-term powerful AI faster, as a field. This post shares what we covered and learned in the talks of our five speakers. Testing a Symposium Format and Three Enablers for AI Safety AccelerationAcceleration of AI safety progress appears valuable in particular in the context of short timelines, to reduce catastrophic or existential risks from beyond-human level AI systems in the near-term.Excellent writing exists on this and related topics from established practitioners. This event aimed to test whether a hybrid symposium with structured presentations could complement the existing discourse and available online forums and facilitate researcher and founder connections and conversations. While there are many lenses and perspectives on achieving accelerated AI safety effectiveness for managing x-risk in short timeline scenarios, the event agenda was driven by three themes that may point to key acceleration enablers:Start with the right ideas: Can we focus on those safety interventions that reduce catastrophic/existential risk from beyond-human level AI the most, and how could we quantify and automate such a selection?Mature/validate ideas fast: Can we automate safety R&D work to mature the best safety interventions faster towards implementation readiness, given resource and talent constraints?Implement ideas in frontier AI: Can we achieve implementation of the best matured safety interventions in frontier model development pipelines, to realize actual reduction of catastrophic/existential risk from near-term AI?  Presentations and discussions of the symposium described in this post primarily covered the first two items. The third item remains critical but was largely outside of the focus on technical methods that this event applied. Presentation SummariesEach speaker at the symposium brought distinct perspectives on AI safety acceleration themes. Below are summaries and key take-aways, with links to full materials for more detailed information.1. Opening and Introduction on AI Safety Acceleration MethodsSpeaker: Martin Leitgab, Independent, martin.leitgab@gmail.comSlide link: HereSummary:How likely is near-term powerful AI, and what is its catastrophic/existential risk?AI capabilities in various domains have been advancing rapidly, from elementary school level (GPT-3) in 2020 to beyond PhD level in 2025- what about next five years?Very few systematic AI x-risk estimates exist, but indicate that AI x-risk may be distinctively higher than other x-risk (e.g. nuclear war x-risk)Is AI safety on track to limit x-risk from near-term powerful AI?Current research literature indicates that several AI safety approaches may not effectively limit x-risk beyond-human level AI, while  misaligned behaviors continue to emerge and scaleNew/breakthrough or other/neglected ideas may need to be pursued that scale to beyond-human level AIAI safety progress acceleration domains include:  1. Finding the most effective x-risk interventions for beyond-human level AI2. Maturing those fast to a deployable state3. Achieving their implementation in frontier model company development pipelines.  The first three of the following presentations covered technical acceleration topics on item/domain #1, on finding most effective safety interventions and related frameworks.2. AI Safety Research Futarchy: Using Prediction Markets to Choose Research Projects for MARSSpeaker: Jason Brown, Geodesic/University of Cambridge, jrb239@cam.ac.ukSlide link: Here, LessWrong post hereSummary:Using Prediction Markets to Prioritize Research ProjectsApplying futarchy to select which AI safety projects MARS mentors pursue for best impact, aggregating community knowledge across 10+ research ideas from mentor groupsPredictions on multiple success metrics: LessWrong upvotes, arXiv publications/citations, and top-tier conference acceptancesPrediction markets incentivize diligent analysis of participants to select most impactful research ideas- unlocking profit by trading on accurate predictions of research outcomesMarkets closed after the symposium on Monday 10/13/20253. Predicting Extinction: Overview on Tools and Methods to Forecast ASI RisksSpeaker: James Newport, Swift Centre, james@swiftcentre.orgSlide link: HereSummaryWhy Use Forecasting for AI RiskCreates structured processes for identifying problems/risks/opportunities while facilitating transparent reasoning and incentivizing error reductionAssists in understanding predictiveness of real-world events/actions for better prioritizationTechnical Tools AvailablePrediction markets/forecasting platforms and Bayesian networks: Aggregate decentralized forecasts into single probabilities; map causal dependencies and propagate uncertainty through probability distributionsSwift Centre application: Custom platform for rapidly eliciting, structuring, and visualizing group probability assessmentsDecision-Making FrameworksDynamic Adaptive Decision Pathways: Designs sequence of low-regret actions with future decisions contingent on monitored tipping points as uncertainty reduces over timeExample live forecast (open after event)4. Towards Predicting X-risk Reduction of AI Safety Solution Candidates through an AI Preferences ProxySpeaker: Martin Leitgab, Independent, martin.leitgab@gmail.comSlide link: HereSummary:Create Measurement Proxy for X-risk in Loss of Control ScenarioBased on few assumptions: AI scaling continues for 3-5 years; capabilities reach beyond-human levels and misalignment behaviors scale, leading to successful exfiltrationImplementing measurement proxy as benchmark of AI preference rankings across Instrumental/Convergent, Pro-Human, and Anti-Human goals, via contextualized binary dilemmasQuantify X-Risk Reduction Potential of AI Safety InterventionsInput: Extract logical paths how interventions address x-risks from AI safety literature (e.g. arXiv cs.AI papers, LessWrong articles)- Eleuther AI volunteer team built algorithm prototype on Alignment Research DatasetForecasting/Simulator LLM reasons through benchmark changes assuming AI safety intervention is implemented- comparing predicted changes in preferences allows to identify safety interventions with highest x-risk reduction potentialThe next two presentations cover technical acceleration topics on item/domain #2, automating the maturation of safety interventions through the safety R&D workflow.5. Automating the AI/ML Safety Research Workflow- Challenges and ApproachesSpeakers: Ronak Mehta and Jacques Thibodeau, Coordinal Research, contact@coordinal.orgSlide link: HereSummaryBenchmark Under-Elicitation Problem vs Real CapabilitiesMost AI R&D benchmarks (like METR) use single autonomous agent scaffolds, but not able to elicit and leverage available capabilitiesReal deployments in labs use high-performing multi-agent systems with e.g. type-safe outputs, clean context management, proper error recoveryAutomation Timelines Perception SkewedR&D automation progress moves much faster through multi-agent systems than single-agent metrics indicateAI safety automation can leverage improved performance but needs to be done with awareness of risksAutomated Alignment Risks & Defense-in-DepthCore issues and possible automation bottlenecks:Alignment work may be more difficult than capability work e.g. due to lack of verification ground truth and hard problems relating to valuesModel situational/evaluation awareness that may lead to scheming that is hard to detect (e.g. subtle research sabotage)Manage risks by e.g. Responsible Automation Policy with Defense-in-DepthConditional safety layers between automated always-on (e.g. deception probes) and triggered-upon-suspicion(e.g. resampling, debate) layers, and rare high-cost human oversight layerAI safety automation opportunities: Trusted codebases, mixed model deployment, strong-to-weak handoff, detailed automation inputs (specs, schedules), and AI dealmaking considerations6. Model Evaluation Automation- Technical Challenges and How to Make ProgressSpeaker: Justin Olive, Arcadia Impact, justin@arcadiaimpact.orgMaterial: HereSummary:High-Quality Evaluations Are Essential But Resource-ConstrainedAI evaluations inform decision-making, underpin governance frameworks (AI Act, RSPs), and support AI safety research (assessing risky capabilities/propensities, testing solution efficacy)Costly to develop and maintain; limited resources constrain achievable quality and coverage (see The Evals Gap – Apollo Research)Example: Inspect Evals aims to provide centralized library of 90+ open-source benchmarks; upholding quality, reliability, and configurability standards is very resource-intensiveAutomation Through Standardization as SolutionCan only high-quality evals with automated eval development and maintenance given resource constraintsFirst step is standardization applied across evals- standard serves as source of truth guiding coding agents and verifying their outputsIdeal outcome: Coding agents run continuously using standard as reference to prioritize tasks, execute them, and self-assess work quality7. Meeting CloseoutSpeaker: Martin Leitgab, Independent, martin.leitgab@gmail.comSlide link: HereSummary:Non-technical and technical methods are critical complementary components for successFunding, talent pipelines/field-building, and governance are needed to enable success of technical acceleration methodsBeyond The Talks- Discussion ThemesDuring Q&A and post-event conversations, several questions emerged that may be productive focal points for future events: Measurement problem A: How can we quantify x-risk? E.g. can we build top-level threat models/safety cases that can be calibrated with specific probability ranges for overall AI catastrophic or x-risk?Measurement problem B: How can we quantify x-risk reduction by safety interventions? How good are proxies like research taste or prediction markets for effective x-risk reduction, and what are their limitations?Automation trade-offs: How do we balance automation benefits against possible differential capability acceleration?Looking AheadThis event aimed to serve researchers and founders working on acceleration methods by providing a hybrid symposium venue to present their work and coordinate. I am thankful for the speakers who took time to share their work, and for everyone who attended the meeting in-person and virtually.Attendance levels >50 and attendee feedback after the event suggest interest in continuing this type of format. Building on this initial experiment, future events will aim to reach more researchers, founders, funders, and forward thinkers in this domain. The goal will remain the same- to provide a structured forum for coordinating the mitigation of catastrophic or existential risk from beyond-human level AI systems that may emerge in the near term.   Follow-on events are planned at this point around e.g. EAG Bay Area in February 2026, with focus on:Expanding reach for more diverse approaches and coordination of practitionersDeeper dives into next steps and challenges for implementing acceleration opportunitiesHow to get involved:If you work on technical acceleration methods and are interested in contributing or collaborating on future versions of this event, please connect at martin.leitgab@gmail.com .What did we miss? What would make future events more valuable? Please comment so this effort can serve the community better.Note: I organized this event independently. I will be joining a new employer later this month, however this work was done in my personal capacity.All errors, misquotes of speaker material, and similar are entirely my own. Please let me know if you see any so I can fix the issue!Discuss ​Read More

​Published on October 22, 2025 9:33 PM GMTWith AI capabilities advancing in several domains from elementary-school level (GPT-3, 2020) to beyond PhD-level (2025) in just five years, the AI safety field may face a critical challenge: developing and deploying effective solutions fast enough to manage catastrophic and existential risks from beyond-human level AI systems that may emerge on timelines shorter than we hope. On October 10, 2025, I organized a hybrid symposium bringing together 52 researchers and founders (25 in-person in NYC) to explore technical methods for accelerating AI safety progress. The focus of the event was not to discuss any single research safety agenda, but how to reduce catastrophic/existential risk from near-term powerful AI faster, as a field. This post shares what we covered and learned in the talks of our five speakers. Testing a Symposium Format and Three Enablers for AI Safety AccelerationAcceleration of AI safety progress appears valuable in particular in the context of short timelines, to reduce catastrophic or existential risks from beyond-human level AI systems in the near-term.Excellent writing exists on this and related topics from established practitioners. This event aimed to test whether a hybrid symposium with structured presentations could complement the existing discourse and available online forums and facilitate researcher and founder connections and conversations. While there are many lenses and perspectives on achieving accelerated AI safety effectiveness for managing x-risk in short timeline scenarios, the event agenda was driven by three themes that may point to key acceleration enablers:Start with the right ideas: Can we focus on those safety interventions that reduce catastrophic/existential risk from beyond-human level AI the most, and how could we quantify and automate such a selection?Mature/validate ideas fast: Can we automate safety R&D work to mature the best safety interventions faster towards implementation readiness, given resource and talent constraints?Implement ideas in frontier AI: Can we achieve implementation of the best matured safety interventions in frontier model development pipelines, to realize actual reduction of catastrophic/existential risk from near-term AI?  Presentations and discussions of the symposium described in this post primarily covered the first two items. The third item remains critical but was largely outside of the focus on technical methods that this event applied. Presentation SummariesEach speaker at the symposium brought distinct perspectives on AI safety acceleration themes. Below are summaries and key take-aways, with links to full materials for more detailed information.1. Opening and Introduction on AI Safety Acceleration MethodsSpeaker: Martin Leitgab, Independent, martin.leitgab@gmail.comSlide link: HereSummary:How likely is near-term powerful AI, and what is its catastrophic/existential risk?AI capabilities in various domains have been advancing rapidly, from elementary school level (GPT-3) in 2020 to beyond PhD level in 2025- what about next five years?Very few systematic AI x-risk estimates exist, but indicate that AI x-risk may be distinctively higher than other x-risk (e.g. nuclear war x-risk)Is AI safety on track to limit x-risk from near-term powerful AI?Current research literature indicates that several AI safety approaches may not effectively limit x-risk beyond-human level AI, while  misaligned behaviors continue to emerge and scaleNew/breakthrough or other/neglected ideas may need to be pursued that scale to beyond-human level AIAI safety progress acceleration domains include:  1. Finding the most effective x-risk interventions for beyond-human level AI2. Maturing those fast to a deployable state3. Achieving their implementation in frontier model company development pipelines.  The first three of the following presentations covered technical acceleration topics on item/domain #1, on finding most effective safety interventions and related frameworks.2. AI Safety Research Futarchy: Using Prediction Markets to Choose Research Projects for MARSSpeaker: Jason Brown, Geodesic/University of Cambridge, jrb239@cam.ac.ukSlide link: Here, LessWrong post hereSummary:Using Prediction Markets to Prioritize Research ProjectsApplying futarchy to select which AI safety projects MARS mentors pursue for best impact, aggregating community knowledge across 10+ research ideas from mentor groupsPredictions on multiple success metrics: LessWrong upvotes, arXiv publications/citations, and top-tier conference acceptancesPrediction markets incentivize diligent analysis of participants to select most impactful research ideas- unlocking profit by trading on accurate predictions of research outcomesMarkets closed after the symposium on Monday 10/13/20253. Predicting Extinction: Overview on Tools and Methods to Forecast ASI RisksSpeaker: James Newport, Swift Centre, james@swiftcentre.orgSlide link: HereSummaryWhy Use Forecasting for AI RiskCreates structured processes for identifying problems/risks/opportunities while facilitating transparent reasoning and incentivizing error reductionAssists in understanding predictiveness of real-world events/actions for better prioritizationTechnical Tools AvailablePrediction markets/forecasting platforms and Bayesian networks: Aggregate decentralized forecasts into single probabilities; map causal dependencies and propagate uncertainty through probability distributionsSwift Centre application: Custom platform for rapidly eliciting, structuring, and visualizing group probability assessmentsDecision-Making FrameworksDynamic Adaptive Decision Pathways: Designs sequence of low-regret actions with future decisions contingent on monitored tipping points as uncertainty reduces over timeExample live forecast (open after event)4. Towards Predicting X-risk Reduction of AI Safety Solution Candidates through an AI Preferences ProxySpeaker: Martin Leitgab, Independent, martin.leitgab@gmail.comSlide link: HereSummary:Create Measurement Proxy for X-risk in Loss of Control ScenarioBased on few assumptions: AI scaling continues for 3-5 years; capabilities reach beyond-human levels and misalignment behaviors scale, leading to successful exfiltrationImplementing measurement proxy as benchmark of AI preference rankings across Instrumental/Convergent, Pro-Human, and Anti-Human goals, via contextualized binary dilemmasQuantify X-Risk Reduction Potential of AI Safety InterventionsInput: Extract logical paths how interventions address x-risks from AI safety literature (e.g. arXiv cs.AI papers, LessWrong articles)- Eleuther AI volunteer team built algorithm prototype on Alignment Research DatasetForecasting/Simulator LLM reasons through benchmark changes assuming AI safety intervention is implemented- comparing predicted changes in preferences allows to identify safety interventions with highest x-risk reduction potentialThe next two presentations cover technical acceleration topics on item/domain #2, automating the maturation of safety interventions through the safety R&D workflow.5. Automating the AI/ML Safety Research Workflow- Challenges and ApproachesSpeakers: Ronak Mehta and Jacques Thibodeau, Coordinal Research, contact@coordinal.orgSlide link: HereSummaryBenchmark Under-Elicitation Problem vs Real CapabilitiesMost AI R&D benchmarks (like METR) use single autonomous agent scaffolds, but not able to elicit and leverage available capabilitiesReal deployments in labs use high-performing multi-agent systems with e.g. type-safe outputs, clean context management, proper error recoveryAutomation Timelines Perception SkewedR&D automation progress moves much faster through multi-agent systems than single-agent metrics indicateAI safety automation can leverage improved performance but needs to be done with awareness of risksAutomated Alignment Risks & Defense-in-DepthCore issues and possible automation bottlenecks:Alignment work may be more difficult than capability work e.g. due to lack of verification ground truth and hard problems relating to valuesModel situational/evaluation awareness that may lead to scheming that is hard to detect (e.g. subtle research sabotage)Manage risks by e.g. Responsible Automation Policy with Defense-in-DepthConditional safety layers between automated always-on (e.g. deception probes) and triggered-upon-suspicion(e.g. resampling, debate) layers, and rare high-cost human oversight layerAI safety automation opportunities: Trusted codebases, mixed model deployment, strong-to-weak handoff, detailed automation inputs (specs, schedules), and AI dealmaking considerations6. Model Evaluation Automation- Technical Challenges and How to Make ProgressSpeaker: Justin Olive, Arcadia Impact, justin@arcadiaimpact.orgMaterial: HereSummary:High-Quality Evaluations Are Essential But Resource-ConstrainedAI evaluations inform decision-making, underpin governance frameworks (AI Act, RSPs), and support AI safety research (assessing risky capabilities/propensities, testing solution efficacy)Costly to develop and maintain; limited resources constrain achievable quality and coverage (see The Evals Gap – Apollo Research)Example: Inspect Evals aims to provide centralized library of 90+ open-source benchmarks; upholding quality, reliability, and configurability standards is very resource-intensiveAutomation Through Standardization as SolutionCan only high-quality evals with automated eval development and maintenance given resource constraintsFirst step is standardization applied across evals- standard serves as source of truth guiding coding agents and verifying their outputsIdeal outcome: Coding agents run continuously using standard as reference to prioritize tasks, execute them, and self-assess work quality7. Meeting CloseoutSpeaker: Martin Leitgab, Independent, martin.leitgab@gmail.comSlide link: HereSummary:Non-technical and technical methods are critical complementary components for successFunding, talent pipelines/field-building, and governance are needed to enable success of technical acceleration methodsBeyond The Talks- Discussion ThemesDuring Q&A and post-event conversations, several questions emerged that may be productive focal points for future events: Measurement problem A: How can we quantify x-risk? E.g. can we build top-level threat models/safety cases that can be calibrated with specific probability ranges for overall AI catastrophic or x-risk?Measurement problem B: How can we quantify x-risk reduction by safety interventions? How good are proxies like research taste or prediction markets for effective x-risk reduction, and what are their limitations?Automation trade-offs: How do we balance automation benefits against possible differential capability acceleration?Looking AheadThis event aimed to serve researchers and founders working on acceleration methods by providing a hybrid symposium venue to present their work and coordinate. I am thankful for the speakers who took time to share their work, and for everyone who attended the meeting in-person and virtually.Attendance levels >50 and attendee feedback after the event suggest interest in continuing this type of format. Building on this initial experiment, future events will aim to reach more researchers, founders, funders, and forward thinkers in this domain. The goal will remain the same- to provide a structured forum for coordinating the mitigation of catastrophic or existential risk from beyond-human level AI systems that may emerge in the near term.   Follow-on events are planned at this point around e.g. EAG Bay Area in February 2026, with focus on:Expanding reach for more diverse approaches and coordination of practitionersDeeper dives into next steps and challenges for implementing acceleration opportunitiesHow to get involved:If you work on technical acceleration methods and are interested in contributing or collaborating on future versions of this event, please connect at martin.leitgab@gmail.com .What did we miss? What would make future events more valuable? Please comment so this effort can serve the community better.Note: I organized this event independently. I will be joining a new employer later this month, however this work was done in my personal capacity.All errors, misquotes of speaker material, and similar are entirely my own. Please let me know if you see any so I can fix the issue!Discuss ​Read More

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *