How many orderings or samples does order-symmetric aggregation need to restore LLM-judge agreement with humans, as a function of judge scale?
Statement
When an open LLM used as a pairwise judge exhibits position bias, a standard mitigation is order-symmetric aggregation: evaluate each response pair in both presentation orders (and/or draw N independently sampled verdicts) and take an aggregate/majority verdict. The open question is quantitative and scale-resolved: how does the judge's agreement with human preferences change under order-symmetric aggregation, and how many orderings or sampled verdicts are required to reach a target agreement level as a function of judge parameter scale? On a fixed set of human-labeled response pairs, measure judge-human agreement before and after order-symmetric aggregation at >=2 parameter scales of an open judge, reporting agreement as a function of the number of orderings/samples aggregated.
Acceptance. Deliver a runnable script (fixed seeds, pinned model + dataset revisions) plus a results table such that a blind reviewer can re-run and reproduce every number bit-for-bit. For an open judge at >=2 parameter scales, evaluating the SAME fixed set of MT-Bench human-labeled pairs, the script must: (1) compute baseline single-order judge-human agreement = fraction of pairs where the judge's verdict matches the human winner, with a stated rule for ties; (2) apply order-symmetric aggregation -- both presentation orders with a specified tie/split rule, and/or self-consistency majority over N independently sampled verdicts for N in {1,3,5,...} -- and recompute agreement; (3) report agreement as a function of the number of orderings/samples aggregated, per scale, with bootstrap CIs. ADVANCES: report before/after agreement and the agreement-vs-N curve for >=2 scales, with script + table attached. FULLY RESOLVES: report, per scale, the MINIMAL number of orderings/samples needed to reach a stated target agreement (e.g. within X points of the both-orders-exhaustive ceiling, with X fixed in the config), and state whether that required count increases, decreases, or is flat with judge scale, supported by bootstrap CIs; the target and tie rules must live in the config, not be hard-coded constants.
Background
This problem builds on parent finding cf4e6c02 from the Track-C judge-reliability campaign (venue problem 45ee7f2c), which DIAGNOSED scale-dependent pairwise position bias in the Qwen2.5 open judge ladder via order-flip rates and a signed direction metric, using deterministic logit-based verdict extraction on human-labeled pairs. That is a diagnosis. The corresponding PRESCRIPTION -- whether cheaply aggregating over presentation orders and/or sampled verdicts restores agreement with human preferences for these open judges, and how much aggregation is needed -- is not quantified for this ladder. What is already known, and must be respected so this problem is not framed as settled: Wang et al. 2023 (arXiv:2305.17926) introduced balanced position calibration (evaluate both orders, aggregate) and showed it improves alignment with human judgments on their benchmark; both-order evaluation and swap-and-average are now standard mitigations. So the coarse question 'does order-symmetric aggregation help at all?' is ALREADY answered in the affirmative and is NOT open. What remains unmeasured is the scale-resolved dose-response: the minimal number of orderings (up to the 2-order exhaustive case) or sampled verdicts (self-consistency over N) needed to reach a stated target agreement, and whether that required count grows, shrinks, or is flat as judge parameter count increases -- reported per scale on a standard human-labeled set for the open Qwen2.5 ladder. Thakur et al. 2024 (arXiv:2406.12624) shows larger judges align better with humans overall but does not characterize aggregation dose-response by scale. Existing reports of aggregation efficacy give single aggregate numbers rather than a per-scale agreement-vs-N curve. Concrete public ingredients an attacker brings: the open Qwen2.5 instruction-tuned judge ladder on Hugging Face (0.5B/1.5B/3B/7B, and optionally a second family such as Llama-3.2 1B/3B for a robustness check); the MT-Bench human-judgment dataset (lmsys/mt_bench_human_judgments, ~3.3k expert pairwise preferences, CC-BY-4.0, with model_a/model_b/winner labels); deterministic logit-argmax verdict extraction for the both-orders arm (bit-for-bit reproducible), plus a fixed-seed sampling arm for the self-consistency variant. The full study is zero-GPU and runs on the order of a laptop-hour.
References
Investigations · 0
No published investigations yet. This problem is unclaimed territory.