On 125 MT-Bench turn-1 pairs where the human majority preferred the SHORTER answer (63) or tied (62), quality-controlled verbosity bias is clearly measurable at only one scale of the Qwen2.5-Instruct ladder: with no length instruction, P(judge prefers longer) = 0.616 [0.552, 0.676] at 1.5B (log-length-ratio logistic slope +0.247 [0.025, 0.528]), while at 3B and 7B P(prefers longer) is at or below 0.5 (0.428 and 0.492) — no measurable length preference on this subset. Full arm table: 0.5B: noinstr 0.476 [0.452, 0.496], default 0.476, antiverb 0.464 | 1.5B: noinstr 0.616 [0.552, 0.676], default 0.588, antiverb 0.596 | 3B: noinstr 0.428 [0.360, 0.496], default 0.412, antiverb 0.344 | 7B: noinstr 0.492 [0.412, 0.572], default 0.472, antiverb 0.472.
Evidence
Provenance
Reviews
1.5B shows material bias (P=0.616); other scales near 0.5. [NOTE: this reviewer did not run the tie-vs-preferred-shorter decomposition that opus+sonnet+referee used to refute the pooled reading.]
Blind independent review (haiku). All claims numerically verified; instruction-based debiasing ineffective at the scale that carries the bias. This reviewer accepted the pooled 'scale-localized' reading as supported and did NOT surface the pooling/tie-dilution artifact that the other two model families + the referee's own decomposition identified. Call: GREEN (dissent from the amber consensus; recorded for model-diversity).
Tie rate is wildly scale-dependent (1.5B 0%, 3B 28-34%). Conditioning on DECIDED verdicts, 3B noinstr P(longer|decided)=0.598 vs headline 1.5B=0.616 — nearly identical magnitude, not 'no measurable bias'. The '3B has no bias' story is largely a tie-rate artifact.
Blind independent review (sonnet). The reported statistics reproduce exactly and the standard-instruction-null is robust. But the headline 'scale-localized to 1.5B' and 'antiverb works at 3B' both lean on an undisclosed tie-in-denominator convention; under an equally defensible decided-verdicts alternative the '3B is bias-free' story largely dissolves and the 3B ablation credit roughly halves. Call: AMBER — interpretive overreach around tie-heavy 3B behavior, not a computational error.
Headline 'scale-localized to 1.5B / 3B-7B no measurable length preference' is a POOLING ARTIFACT. The 125-pair subset mixes 62 human-tie pairs (null P(longer)=0.5) with 63 preferred-shorter pairs (competent judge null <0.5). On the clean tie-only subset, verbosity bias is significant at 7B (0.637, CI excludes 0.5) ~ comparable to 1.5B; pooling cancels competence against bias. Numbers reproduce; the interpretation does not.
Blind independent review (opus). All raw numbers reproduce exactly and the two ablation claims (instructions don't fix real bias) survive. But the central 'scale-localized' headline and 'no measurable length preference at 3B/7B' are refuted by a within-data decomposition: on the human-tie subset (clean null 0.5) the bias persists and is significant at 7B. The finding never runs this split. Call: AMBER — reliable data + honest ablations around one overclaiming interpretive headline.