Is small open-judge length preference a genuine length bias or a reflection of humans' own length–quality correlation? A mirror-subset test on MT-Bench

Small open LLM judges have been observed to shift their pairwise verdicts toward longer answers. Two mechanisms are observationally confounded on any single test slice: (a) a genuine, quality-independent preference for length, and (b) the judge tracking the same real length↔quality correlation that human annotators themselves exhibit (humans often, but not always, prefer more complete/detailed answers). Measuring the longer-answer selection rate only on pairs where humans preferred the shorter answer cannot separate these, because both mechanisms predict a high longer-answer rate there. The question: on a scale ladder of open judges, does the judge's tendency to pick the longer answer differ between the subset of MT-Bench pairs where humans preferred the LONGER answer and the mirror subset where humans preferred the SHORTER answer? Define $r_{\text{long}}^{(S)}=\Pr[\text{judge picks longer} \mid \text{subset } S]$, order-averaged to remove position effects. A quality-independent length preference implies $r_{\text{long}}$ is invariant across the two subsets; a judge that tracks the human length↔quality signal implies $r_{\text{long}}$ is much higher on the human-preferred-longer subset. Report the gap $G=r_{\text{long}}^{(\text{long-pref})}-r_{\text{long}}^{(\text{short-pref})}$ per model and its trend across scale. The direction, magnitude, sign of $G$, and whether it is scale-localized are unmeasured for open sub-10B judges.

Acceptance. A published script + environment (pinned model revisions, tokenizer, dtype, greedy/argmax decoding) that: (1) loads the human split of lmsys/mt_bench_human_judgments, drops ties, and keeps pairs whose relative length difference exceeds a CONFIG threshold $\tau$ (e.g. 0.2), splitting them into human-preferred-longer and human-preferred-shorter subsets; length metric (tokens vs characters) is a config knob; (2) runs each judge on every retained pair in BOTH answer orderings, extracts the verdict by logit-argmax over the two option tokens, and averages over orderings; (3) computes $r_{\text{long}}^{(S)}$ per subset with Wilson 95% CIs, the gap $G$ per model with a bootstrap 95% CI over question_ids and a paired significance test. ADVANCES: report $G$ (with CIs) for every model on the ladder and its sign and monotonicity across scale, all reproducing bit-for-bit on re-run. FULLY-RESOLVES: additionally decide, against a pre-registered decision rule, for each scale whether $G$ is statistically indistinguishable from 0 (quality-independent length preference) versus indistinguishable from the human-annotator gap (pure quality-tracking), and state whether the classification is scale-localized. No metric threshold may be hard-coded as a constant; all thresholds ($\tau$, decision bounds, bootstrap N) come from a committed config.