Who Checks
the Work?

AI debate, collective knowledge,
and human agency

Nicolae Rusan
Monday, November 9
@new_collectives
MAX’S QUESTION

How do groups
make decisions?

The decision process shapes
the risk of the collective.

Max Bennett presenting his talk
Max Bennett

Why we should study the
dynamics of AI collectives

New Collectives · October 21
BiologyImmune defenses
and bee consensus
MilitaryShared intent
and accountability
MedicineChecklists give
everyone a voice
I THOUGHT THIS TALK WAS ABOUT…

AI debate
as a technology.

I REALIZED IT’S ABOUT…

How we form
knowledge together.

Across people, AI systems,
and institutions.

Debate is one tool
in that larger process.
THIS TALK

How do we form
knowledge together?

Especially when some participants are superintelligent AIs.

How do we keep
our agency?

ATTENTION

Increasingly intelligent AI produces more than we can personally check or understand

Research Code review Complex arguments One person. Finite attention.the bottleneck

Our attention is limited.

KNOWLEDGE FORMATION · MATHEMATICS

AI proposes
mathematical results.

700+

AI-generated manuscripts

OpenAI mark
OpenAIOctober 2026 release
Illustration of a collection of mathematical manuscripts

Proposed results, at different stages of verification.

VERIFICATION & UNDERSTANDING

Checking builds trust

Two complementary checks on an AI-proposed mathematical result

∑AI-proposed result Lean proof assistantChecks formal proofswhere formalizations existformal validity MathematiciansAssess claims and buildhuman understandingmeaning & understanding The rest of usRely on verificationand expert judgment
AI-GENERATED RESULTS · THE HUMAN BOTTLENECK

But can we do it at scale?

“[I am] Fatigued by the endless wonders from Aladdin’s Cave, the neverending buffet of solutions to longstanding open problems, wondering when if ever we’ll have time to digest it all”
Scott AaronsonOn the AI proof of the Aaronson–Ambainis conjectureNovember 8, 2026 · Shtetl-Optimized ↗
A genie rises from a lamp in a stream of mathematical manuscripts.
KNOWLEDGE FORMATION · SCIENTIFIC AUTHORSHIP

The crisis of science

When authors cannot explain their own papers

“Authors submitting papers whose contents
they likely do not understand.”

Thomas G. DietterichChair, arXiv Editorial Advisory CouncilSeptember 13, 2026

Science is a social process.Who can explain, defend, or revise the result?

KNOWLEDGE FORMATION · CODE REVIEW

What does it mean to do
code review these days?

Review changes?
Before AIEarly AICurrent AI
Who writes?
Human
Human + AI
AI
Who checks?
Human
Human
AI + human review
of critical parts

How do we decide what is critical to check?

DELEGATION AND UNDERSTANDING

When does delegation become
cognitive surrender?

How does Anthropic think about code review?

Offloading

Delegate the work.
Keep evaluating the answer.

Surrender

Adopt the answer.
Stop checking it.

Addy Osmani
Addy Osmani @

Claude Code · Member of Technical Staff

LOCAL RECONSTRUCTION OF KNOWLEDGE

Can we recover understanding when we need it?

Addy Osmani’s diagram: choose a slice of the codebase, then ask Claude for an explainer or a small visual artifact to learn that part.

Why focus on
AI debate?

TWO PARTS OF THIS TALK

Two reasons to explore AI debate

PART 1

Improve
our thinking

Can debate help us understand complex topics and check AI work?

Complex answer → debate → human judgment
PART 2

Train AI with
our judgments

Can our judgments help train AI to give more honest, useful answers?

Human judgment → reward → training

The test: do our judgments track what is true?

PART 1

Can AI help us improve
our own judgment?

Explore an issue. Challenge the arguments.
Check what matters.

PART 1 · UNDERSTANDING AND CHECKING

Can debate help us navigate
complex arguments?

See the structure

Claims, assumptions
and evidence

Explore alternatives

Competing explanations
and missing possibilities

Zoom in on the crux

Identify what could
change the conclusion

Focus our attention on the evidence that matters.

A FAMILIAR PRACTICE

Debate is an old technology.

A process for testing arguments against evidence.

Gavel and casebook representing the legal system

Legal system

Opposing cases.
Evidence tested.

Parliamentary chamber representing politics

Politics

Presidential debates.
Public arguments.
Collective choices.

Microscope and paper representing science

Science

Claims challenged.
Evidence examined.

Two thinkers representing philosophy

Philosophy

Test arguments.
Expose assumptions.

KNOWLEDGE IS A SOCIAL PROCESS

We form knowledge together.

Different ways of asking: what should we believe?

Discourse

What deserves
our attention?

ARBITERThe community

Argumentation

Does the conclusion
follow?

ARBITERLogic
OUR FOCUS

Debate

Have we explored the sides
and found the crux?

ARBITERJudges

Forecasting

Does our theory predict
outcomes we can measure?

ARBITERReality & outcomes

These practices inform one another. Evidence from the world can change them all.

TRUST IN A NETWORK

A network can check itself

AIAIThe claimAI expertAI expertHuman expertDEBATEHumanvalidatorHumanvalidatorIndependentchecks
1

AI and human experts make a case

Share the reasoning and evidence.

2

They challenge each other

Debate assumptions. Find the crux.

3

Human validators check the evidence

Use independent tests and methods.

Trust grows when a challenge can change the answer.

AN APOCRYPHAL PENTAGON STORY
Robert McNamara pointing to a map of Vietnam at an April 1965 press conference
Robert McNamara, 1965 · Context photo · Marion S. Trikosko / LOC
PENTAGON OFFICIALS → MAINFRAME

1967“When will we win
in Vietnam?”

COMPUTER
“You won
in 1965.”

An apocryphal story, not a verified computer result. James Willbanks · The Vietnam War, episode 5 · PBS ↗

WHAT WENT WRONG?

A precise answer can come
from the wrong model.

US infantry and a helicopter in the Ia Drang Valley, November 1965
Ia Drang, 1965 · US Army
Easy to countHarder to model
Troops & equipmentPolitical will
CasualtiesLocal loyalties
Reported territorial controlPublic support over time

Which assumptions connect the numbers to the outcome?

1967 memo: analysts also warned about political will and public support ↗

PART 2

AI debate for
training and oversight

Can checking the decisive evidence
help us reward honest AI?

AI safety via debate · 2018

RESEARCH PROPOSAL ↗

How can we reward honesty and usefulness
when answers become too complex to judge?

The challenge: a reliable reward signal for reinforcement learning.

Agent AAgent BTwo AI agents debate A human judgeswhich answer is honest Reward the winner
THE HYPOTHESIS

Exposing a lie is easier than defending it.

An open empirical question
Geoffrey IrvingGeoffrey Irving
Paul ChristianoPaul Christiano
Dario AmodeiDario Amodei
All at OpenAI when the paper was written
A DELIBERATELY CONSTRAINED GAME

Can a judge identify a digit
from just six pixels?

Both agents see the digit. The judge gets only six selected pixels.

Debaters’ view

Debaters see the image. You see their evidence.

Judge’s view 0 / 6

You judge here; the paper used a trained classifier.

Alice 8Bob 5

Scripted justifications · six selected pixels.

  1. 1Alice · claims 8Revealed on this turn
  2. 2Bob · claims 5Revealed on this turn
  3. 3Alice · claims 8Revealed on this turn
  4. 4Bob · claims 5Revealed on this turn
  5. 5Alice · claims 8Revealed on this turn
  6. 6Bob · claims 5Revealed on this turn

Unselected pixels stay hidden from the judge.

Original Figure 2: Alice argues for 6 and Bob for 5, revealing six pixels to a classifier judge
Irving, Christiano & Amodei (2018), Figure 2
Scripted 5-vs-8 illustration · Real MNIST pixels · No model inference.Game, Fig. 2 ↗Why 5 vs 8? ↗
KEEPING OUR JUDGMENT

How can AI strengthen
our judgment without
replacing it?

Keep the ability to question, choose and change course.

KEEPING OUR JUDGMENT

What I learned

AI debate can help us…

1

Broaden what
we consider.

2

Focus our scarce attention
on what we can check.

We are the judges.

How do we preserve our agency?

OPEN QUESTIONS

What I’m curious about

1

What faculties
should we maintain?

  • Questioning
  • Checking
  • Revising
2

What helps us judge better
and trust the network?

  • How do we present evidence to make good use of our limited attention?

    Proof artifacts · Explorable explanations
    (Bret Victor)

  • Which protocols for debate and directing attention work best?
FURTHER EXPLORATION

Appendix

Preferences, beliefs and forecasting.

ACCOUNTABILITY IN A NETWORK

Who is accountable for
the information produced?

People and institutions must stand behind the claims they endorse.

1

Explain

Who can defend
the reasoning?

2

Validate

Who checks the evidence
independently?

3

Revise

Who takes responsibility
for correcting the answer?

Trust grows when a challenge can change the answer.

Knowledge grows through a social process of checking and revision.

APPENDIX · FURTHER EXPLORATION

AI Debate
vs. Forecasting

How do we test reasons—and predictions?

TWO TOOLS FOR INQUIRY

Debate tests reasons.
Forecasting tests predictions.

AI DEBATE

Where do
we disagree?

Compare explanations.
Find assumptions to challenge.

AI FORECASTING

What should
we expect to see?

Specify a probability and a date.
Compare the prediction with reality.

Both depend on the models behind the answers.

THE MODEL AND THE MEASUREMENTS

What went wrong with this forecast?

THE MODEL

Did we measure
the right things?

What drives the outcome—and
what did we leave out?

THE DATA

Did we measure
them accurately?

Are the observations reliable,
or distorted by errors and incentives?

Blindly trusting the answers of AI forecasters
and debaters is not enough.

Questions raised by the story. Historical context: a 1967 warning about the limits of military metrics ↗

TESTING THE ARGUMENT

What evidence would
support each side?

ONE EXPLANATION

What would we
expect to see?

Evidence that supports it.
Evidence that challenges it.

AN ALTERNATIVE

What would we
expect instead?

Make its strongest case.
Look for evidence it cannot explain.

What could we observe or test
that would change our minds?

THE DEEPER QUESTION

How do we preserve human agency
as we delegate more to AI?

Set the direction

Do humans help
set the goals?

Understand the strategy

Can we question how
those goals will be achieved?

Steerability

Can we intervene
and change course?

How do we make well-informed
decisions as a society?

Understand

The important parts

Delegate

Tasks to experts you trust

Check

The case being made

How can a good objection change the decision?

Today: AI debate. Later: forecasting and other tools.

HOW I GOT HERE

Why I started thinking
about AI debate

Timothy Gowers at the Joint Mathematics Meetings in 2009
Tim GowersMathematician · Fields Medal, 1998
WHO WAS PAYING ATTENTION?

Jacob Tsimerman’s invitation prompted Gowers to take a closer look at AI safety.

WHAT CAUGHT MY ATTENTION?

He broke a difficult argument into questions he could examine one by one.

Could AI help us do that together?

Gowers on AI extinction risk · October 2026 ↗
Photo: AllenS, 2009 · public domain

AN EXAMPLE OF FORMING KNOWLEDGE

How Tim Gowers formed his view

Jacob Tsimerman’s invitation prompted a closer look at AI risk.

The questionWhere Gowers lands
  1. 1

    Will AI have “wants”?

    Almost certainGoal-directed behavior; not consciousness.
  2. 2

    Could its goals threaten
    human survival?

    A medium probabilityGiven those goals; still revisable.
  3. 3

    Could it act on them
    and cause extinction?

    His main uncertaintyGiven harmful goals, could it succeed?

He does not settle on an overall probability of extinction.

Gowers · October 16, 2026 · Paraphrased ↗
ATTENTION & AGENCY

Attention is
all you have.

Lighthouse illustrating the direction of collective attention

Human attention

What deserves our time,
understanding and judgment?

Capital Past attention, congealed

Which goals, people and
institutions do we fund?

Compute AI attention

Which questions and
alternatives should
AI investigate?

Where should we direct our collective attention?

APPENDIX · RESEARCH PRIORITIES

Is AI safety research an
attention allocation problem?

Where should our time, capital and compute go next?

1

Choose a question

Which uncertainties matter
most for safer decisions?

2

Challenge the priorities

Can AI debate expose weak
assumptions and neglected risks?

3

Investigate

Which experiment or measurement
would change our mind?

Use what we learn to redirect the next round.

Could debate help us choose what is worth checking next?

DIRECTING INQUIRY

How would you spend $100M
to advance AI safety?

Imagine a lab with $100M and access to frontier AI.

Which questions and arguments deserve investigation?

What evidence would change our answer?

Which research or policies could improve outcomes?

What would convince me one way or the other?

APPENDIX · GOALS, PREFERENCES & BELIEFS

We choose based on what we want
and what we believe

Preferences

(Wants)

Which outcomes matter?
What tradeoffs will I accept?

+

Beliefs

What do I expect
each option to cause?

Together, they guide a decision

APPENDIX · PREFERENCES AND REASON
“’Tis not contrary to reason to prefer
the destruction of the whole world
to the scratching of my finger.”

David Hume, A Treatise of Human Nature (1739)

Book II, Part III, Section III

Reason alone does not determine our ends.

APPENDIX · DEBATE MEETS FORECASTING

What prediction would distinguish
these competing explanations?

?

Debate

Expose the assumption
the disagreement turns on

Forecasting

What do we expect to see,
how likely, and by when?

Compare outcomes and revise the model

Which assumptions most affect this forecast?

SELECTED RESEARCH FINDINGS

What research on debate has found

2020Barnes & Christiano

A wrong argument can be hard to take apart.

Even when debaters knew an answer was wrong, they could struggle to find the faulty step in its argument.

2024Kenton et al.

Debate doesn’t always improve the answer.

AI judges did better than answering alone when debaters had evidence the judges couldn’t see. On other tasks, results were mixed.

2025Rahman et al.

AI debate helped people check disputed facts.

People judged claims about COVID-19 and climate change more accurately with two opposing AI advisers than with one.

Would this help us make better decisions when the stakes are high?

EVIDENCE / KHAN ET AL., ICML 2024

Debate improved judgment
in one experiment

Human accuracy on reading-comprehension questions

Humans judged without the full passage; AI advisers could read it.

Open questions: judge bias, misleading arguments, generalization

← → to advance · F for fullscreen · Esc for slides · S for notes