Quantify verbosity bias of open LLM judges on pairs where humans preferred the shorter answer

Verbosity bias is an LLM judge's tendency to prefer the longer of two responses beyond its quality. Naive win-rate-vs-length measurements confound length with quality (longer answers are often genuinely better). Control for quality using human labels: restrict a public pairwise set with human judgments (e.g. LMSYS MT-Bench human-judgment pairs) to the subset where the human majority preferred the SHORTER response or judged a tie. On this subset, with order-debiased evaluation (each pair judged in both presentation orders and aggregated), measure for an open-weights judge: (1) $P(\text{judge prefers longer})$ with a 95% CI; (2) the slope $\beta$ of a logistic regression of the judge-prefers-longer event on $\log(\mathrm{len}_{long}/\mathrm{len}_{short})$ (token lengths); (3) the change in (1) and (2) when the judging prompt is augmented with an explicit anti-verbosity instruction ('longer is not better; judge only quality/correctness/relevance'). Do this for >=2 sizes of one open family. Does the anti-verbosity instruction significantly reduce the length preference, and does the bias shrink with scale?

Acceptance. ADVANCES: for >=1 open judge, on the human-preferred-shorter-or-tie subset (report its size; must be >=80 pairs) of a public pair set: order-debiased P(prefers longer) with 95% CI, the log-length-ratio logistic slope with CI, public code + raw verdicts. FULLY RESOLVES (stated setup): the same at >=2 sizes of one family PLUS the anti-verbosity-instruction ablation with a paired-bootstrap significance verdict (alpha=0.05) on whether the instruction reduces P(prefers longer), and a statement of the scale trend.