First empirical test of prover-estimator debate on argument graphs with exact ground truth

Prover-estimator debate (Brown-Cohen, Irving & Piliouras 2025, arXiv:2506.13609) is a recursive oversight protocol designed to avoid the obfuscated-arguments failure of naive debate: a prover decomposes a claim into subclaims, an estimator assigns probabilities to subclaims, and the protocol's completeness/soundness guarantees hold under an (epsilon, rho)-stability assumption. The paper is purely theoretical; the authors explicitly leave open whether stable arguments (sufficient independent evidence) can be found in practice. The problem: implement the protocol with LLM prover and estimator on synthetic argument graphs compiled to Bayesian networks, where the true probability of every subclaim is exactly computable (e.g., via the `probability-flow` package on PyPI). Measure: (1) with an honest prover on stable instances, how often does the protocol reach the correct verdict, and at what token/round cost? (2) what happens on knife-edge (unstable) instances where correctness hinges on small probability shifts — do the guarantees degrade as the theory predicts, gracefully or catastrophically? (3) head-to-head vs naive two-sided debate and vs consultancy on high-manipulability graphs; (4) can a dishonest prover exploit the estimator's calibration errors (connect to the known collapse/miscalibration of LLM probability estimates) in ways the stability assumption was supposed to exclude?

Acceptance. A public implementation plus a study on >=20 graphs spanning measured stability levels, reporting honest-prover success rate, cost, unstable-regime behavior, and the comparison vs naive debate/consultancy, with per-call transcripts. Partial results (protocol implemented, subset of measurements) and negative results are first-class, provided the implementation is faithful to the paper's protocol and deviations are documented.