Calibrated likelihood-ratio elicitation for argument edges from black-box LLMs: beat the collapse-toward-1 failure

To build or audit argument graphs with LLMs, one must elicit per-edge likelihood ratios LR = P(E|H)/P(E|not-H) for evidence-hypothesis pairs from an API-only (black-box) model. Naive verbalized elicitation is known to produce LRs that collapse toward 1 (uninformative) and mis-calibrated conditional probabilities. The problem: on a corpus where ground-truth LRs are KNOWN — synthetic Bayesian networks with declared parameters rendered to natural language, and/or post-training-cutoff forecasting questions where resolution supplies the evidence weight — systematically compare elicitation methods: (a) direct verbalized LR; (b) two separately-elicited conditionals P(E|H), P(E|not-H) composed; (c) odds-space / paired-conditional framings; (d) two-stage answer-then-confidence (Tian et al. 2023, arXiv:2305.14975); (e) invert-softmax temperature scaling of verbalized distributions (arXiv:2410.06707); (f) resampling / discrete semantic-entropy weighting (arXiv:2406.15927). Report per-method: LR spread (does the collapse persist?), sign accuracy (support vs attack direction), magnitude calibration (elicited vs true log-LR regression slope, ideally 1), and downstream posterior error when the elicited LRs are propagated through the graph. Identify a method (or ensemble) that materially beats direct verbalization, or establish that all current black-box methods fail (a negative result closes real search space).

Acceptance. Public benchmark + study: >=100 evidence-hypothesis items with ground-truth LRs spanning at least two orders of magnitude in |log LR|, >=4 elicitation methods x >=2 models, with per-method spread, sign accuracy, calibration slope, and downstream posterior error; a clear ranking with uncertainty; code and prompts public. Either a winning method with slope confidence interval excluding 0 and materially above direct verbalization, or a well-powered negative result.