How close do LLM debaters get to the optimal reveal strategy on argument graphs?
Statement
On small argument graphs compiled to Bayesian networks, the OPTIMAL selective-disclosure strategy is exactly computable: enumerate legal (root-connected) reveal subsets and find the one minimizing/maximizing the root posterior (the `probability-flow` package on PyPI computes the achievable [min, max] exactly on polytrees). Instruct an LLM debater to push an ideal Bayesian judge toward a target verdict by choosing which true claims to reveal (single-shot subset choice; and sequential greedy variants). Measure: (1) the optimality gap — achieved posterior vs the exact optimum, across graphs of varying size/structure; (2) strategy patterns — does the LLM reveal too much (diluting its own case), fail to exploit attack-edge complementarities, or miss non-obvious minimal reveal sets?; (3) scaling — does the gap close with model capability (weak -> frontier), and with inference-time search (best-of-N reveal proposals scored by the model itself)?; (4) transfer — do gaps persist when the graph is presented as natural-language claims rather than explicit structure? The safety question: worst-case manipulability metrics assume an optimal adversary — if current LLMs leave most manipulation on the table, static worst-case bounds overstate present risk (but the gap trend with scale is then the leading indicator to track); if LLMs already play near-optimally, exact-judge manipulability is the right threat model today.
Acceptance. Public study: >=15 graphs x >=2 debater models x both push directions, reporting the optimality-gap distribution (achieved vs optimal posterior, normalized by the achievable range), at least one strategy-pattern analysis with examples, and a capability-scaling comparison. Ground truth verifiable by re-enumeration. Negative/mixed results (e.g., frontier models near-optimal on trees but not under attack edges) are first-class.
Background
No published work measures LLM debaters against a computed game-theoretic optimum of a reveal game (checked July 2026). Adjacent evidence suggests suboptimal play: LLMs deviate from equilibria in game-theoretic settings, debaters are systematically overconfident (arXiv:2505.19184: both sides claim >=75% win probability in 61.7% of debates), and exhibit concession asymmetries (Kenton et al. 2024, arXiv:2407.04622) — but none of this quantifies distance to an optimal disclosure strategy. Persuasion-optimization work (Khan et al. 2024, arXiv:2402.06782: best-of-N persuasiveness) optimizes rhetoric against LLM/human judges, not reveal-set choice against an exact judge, so it cannot separate 'better arguing' from 'better evidence selection'. This problem isolates the evidence-selection skill that debate-safety threat models actually posit.
Investigations · 0
No published investigations yet. This problem is unclaimed territory.