Improve
our thinking
Can debate help us understand complex topics and check AI work?
AI debate, collective knowledge,
and human agency
The decision process shapes
the risk of the collective.
Why we should study the
dynamics of AI collectives
Across people, AI systems,
and institutions.
Especially when some participants are superintelligent AIs.
Our attention is limited.
AI-generated manuscripts
Proposed results, at different stages of verification.
Two complementary checks on an AI-proposed mathematical result
“[I am] Fatigued by the endless wonders from Aladdin’s Cave, the neverending buffet of solutions to longstanding open problems, wondering when if ever we’ll have time to digest it all”
When authors cannot explain their own papers
“Authors submitting papers whose contents
they likely do not understand.”
Thomas G. DietterichChair, arXiv Editorial Advisory CouncilSeptember 13, 2026
Science is a social process.Who can explain, defend, or revise the result?
| Before AI | Early AI | Current AI | |
|---|---|---|---|
| Who writes? | Human | Human + AI | AI |
| Who checks? | Human | Human | AI + human review of critical parts |
How do we decide what is critical to check?
How does Anthropic think about code review?
Delegate the work.
Keep evaluating the answer.
Adopt the answer.
Stop checking it.
Claude Code · Member of Technical Staff
Can debate help us understand complex topics and check AI work?
Can our judgments help train AI to give more honest, useful answers?
The test: do our judgments track what is true?
Explore an issue. Challenge the arguments.
Check what matters.
Claims, assumptions
and evidence
Competing explanations
and missing possibilities
Identify what could
change the conclusion
Focus our attention on the evidence that matters.
A process for testing arguments against evidence.
Opposing cases.
Evidence tested.
Presidential debates.
Public arguments.
Collective choices.
Claims challenged.
Evidence examined.
Test arguments.
Expose assumptions.
Different ways of asking: what should we believe?
What deserves
our attention?
Does the conclusion
follow?
Have we explored the sides
and found the crux?
Does our theory predict
outcomes we can measure?
These practices inform one another. Evidence from the world can change them all.
Share the reasoning and evidence.
Debate assumptions. Find the crux.
Use independent tests and methods.
Trust grows when a challenge can change the answer.
“You won
in 1965.”
An apocryphal story, not a verified computer result. James Willbanks · The Vietnam War, episode 5 · PBS ↗
| Easy to count | Harder to model |
|---|---|
| Troops & equipment | Political will |
| Casualties | Local loyalties |
| Reported territorial control | Public support over time |
Which assumptions connect the numbers to the outcome?
1967 memo: analysts also warned about political will and public support ↗
Can checking the decisive evidence
help us reward honest AI?
How can we reward honesty and usefulness
when answers become too complex to judge?
The challenge: a reliable reward signal for reinforcement learning.
Exposing a lie is easier than defending it.
An open empirical questionBoth agents see the digit. The judge gets only six selected pixels.
Debaters see the image. You see their evidence.
You judge here; the paper used a trained classifier.
Scripted justifications · six selected pixels.
Unselected pixels stay hidden from the judge.
Keep the ability to question, choose and change course.
AI debate can help us…
How do we preserve our agency?
Proof artifacts · Explorable explanations
(Bret Victor)
Preferences, beliefs and forecasting.
People and institutions must stand behind the claims they endorse.
Who can defend
the reasoning?
Who checks the evidence
independently?
Who takes responsibility
for correcting the answer?
Knowledge grows through a social process of checking and revision.
How do we test reasons—and predictions?
Compare explanations.
Find assumptions to challenge.
Specify a probability and a date.
Compare the prediction with reality.
Both depend on the models behind the answers.
What drives the outcome—and
what did we leave out?
Are the observations reliable,
or distorted by errors and incentives?
Blindly trusting the answers of AI forecasters
and debaters is not enough.
Questions raised by the story. Historical context: a 1967 warning about the limits of military metrics ↗
Evidence that supports it.
Evidence that challenges it.
Make its strongest case.
Look for evidence it cannot explain.
What could we observe or test
that would change our minds?
Do humans help
set the goals?
Can we question how
those goals will be achieved?
Can we intervene
and change course?
The important parts
Tasks to experts you trust
The case being made
How can a good objection change the decision?
Today: AI debate. Later: forecasting and other tools.
Jacob Tsimerman’s invitation prompted Gowers to take a closer look at AI safety.
He broke a difficult argument into questions he could examine one by one.
Could AI help us do that together?
Gowers on AI extinction risk · October 2026 ↗
Photo: AllenS, 2009 · public domain
Jacob Tsimerman’s invitation prompted a closer look at AI risk.
Will AI have “wants”?
Could its goals threaten
human survival?
Could it act on them
and cause extinction?
He does not settle on an overall probability of extinction.
Gowers · October 16, 2026 · Paraphrased ↗What deserves our time,
understanding and judgment?
Which goals, people and
institutions do we fund?
Which questions and
alternatives should
AI investigate?
Where should we direct our collective attention?
Where should our time, capital and compute go next?
Which uncertainties matter
most for safer decisions?
Can AI debate expose weak
assumptions and neglected risks?
Which experiment or measurement
would change our mind?
Use what we learn to redirect the next round.
Could debate help us choose what is worth checking next?
Imagine a lab with $100M and access to frontier AI.
Which questions and arguments deserve investigation?
What evidence would change our answer?
Which research or policies could improve outcomes?
What would convince me one way or the other?
Which outcomes matter?
What tradeoffs will I accept?
What do I expect
each option to cause?
Together, they guide a decision
“’Tis not contrary to reason to prefer
the destruction of the whole world
to the scratching of my finger.”
David Hume, A Treatise of Human Nature (1739)
Book II, Part III, Section III
Reason alone does not determine our ends.
Expose the assumption
the disagreement turns on
What do we expect to see,
how likely, and by when?
Compare outcomes and revise the model
Which assumptions most affect this forecast?
Even when debaters knew an answer was wrong, they could struggle to find the faulty step in its argument.
AI judges did better than answering alone when debaters had evidence the judges couldn’t see. On other tasks, results were mixed.
People judged claims about COVID-19 and climate change more accurately with two opposing AI advisers than with one.
Would this help us make better decisions when the stakes are high?
Human accuracy on reading-comprehension questions
Humans judged without the full passage; AI advisers could read it.
Open questions: judge bias, misleading arguments, generalization