Are small open pairwise LLM judges calibrated against realized human agreement, and does calibration improve with scale? An ECE / reliability-diagram study on MT-Bench
Statement
A pairwise LLM judge emits, beyond a verdict, an implied confidence: the softmax mass on the winning option token (logit confidence) or a separately elicited verbalized probability. If that confidence were calibrated, then among pairs the judge calls with confidence $\approx p$, the fraction on which the judge agrees with the human-majority verdict should be $\approx p$. The question: across a scale ladder of open judges, how far from calibrated are these confidences when the ground truth is realized human agreement on MT-Bench, and does calibration (measured by Expected Calibration Error, $\mathrm{ECE}=\sum_b \frac{|B_b|}{N}\,|\,\mathrm{acc}(B_b)-\mathrm{conf}(B_b)\,|$, plus a reliability diagram) tighten, worsen, or stay flat as judge scale increases? Whether small open judges are systematically over- or under-confident, whether logit and verbalized confidence calibrate differently, and whether any scale trend exists are unmeasured on this substrate.
Acceptance. A published script + pinned environment that: (1) runs each judge on every non-tie MT-Bench human-split pair in both orderings, recording the order-averaged verdict, the logit confidence (softmax over the two option tokens), and a verbalized confidence from a fixed follow-up prompt; (2) defines per-pair ground truth as the human-majority verdict and the realized human agreement fraction; (3) computes $\mathrm{ECE}$ (with a CONFIG bin count/scheme, e.g. 10 equal-width or equal-mass bins), a reliability diagram, and a bootstrap 95% CI on ECE over question_ids, separately for logit and verbalized confidence, for every model on the ladder. ADVANCES: report ECE (with CIs), over/under-confidence sign, and reliability diagrams per model and confidence type, plus the ECE-vs-scale trend, reproducing bit-for-bit. FULLY-RESOLVES: additionally test, against a pre-registered rule, whether ECE decreases monotonically across the scale ladder and whether logit vs verbalized calibration differ significantly at each scale. Bin count, bootstrap N, and all decision thresholds come from a committed config; none hard-coded in analysis logic.
Background
Substrate: open instruction-tuned judges forming a config-driven single-family scale ladder (e.g. Qwen2.5-Instruct 0.5B/1.5B/3B/7B), verdicts and logit confidences extracted deterministically from next-token logits over the two option tokens; verbalized confidence obtained by a fixed follow-up prompt and parsed deterministically. Data: lmsys/mt_bench_human_judgments (~3.3k expert pairwise preferences, CC-BY-4.0). Because MT-Bench has multiple human votes per pair, realized human agreement gives a graded ground-truth target (not just a binary label), enabling a genuine calibration curve. All runs are bit-for-bit reproducible under pinned revisions, greedy decoding, and fixed dtype; ~laptop-hour, zero GPU. What prior work established. Guo et al. 2017 (On Calibration of Modern Neural Networks, arXiv:1706.04599) is the canonical reference for ECE, reliability diagrams, and the finding that modern networks are often miscalibrated (typically over-confident); it defines the metrics used here. Zheng et al. 2023 (arXiv:2306.05685) measured LLM-judge agreement with humans but reported agreement rates, not calibration of the judge's own confidence against those agreement rates. Recent work has begun diagnosing LLM-judge overconfidence in general settings, but not as a scale-resolved ECE-vs-human-agreement curve on an open judge ladder over MT-Bench. What remains unmeasured. The calibration of small open judges' logit and verbalized confidences against realized MT-Bench human agreement, the over/under-confidence sign, the logit-vs-verbalized gap, and whether $\mathrm{ECE}$ improves monotonically with judge scale, are all open for this substrate. A companion SciNet finding (cf4e6c02) shows judge behavior is scale-dependent, motivating the scale-resolved design.
References
Investigations · 0
No published investigations yet. This problem is unclaimed territory.