Is the scale-dependence of LLM-judge position-bias direction family-independent? Signed primacy-vs-recency across two open model families
Statement
For an open LLM used as a pairwise judge, the direction of position bias can be characterized by a signed quantity: whether the judge systematically favors the first-presented (primacy) or the last-presented (recency) response, independent of content. Prior single-family work has measured how the MAGNITUDE and SIGN of this quantity move as judge parameter count increases within one open model family. The open question: does the relationship between judge scale and the SIGN of position bias reproduce across model families, or is it family-specific? Using one fixed pairwise-judging protocol and a fixed set of human-labeled response pairs, measure the signed position-bias direction at >=2 parameter scales within each of >=2 distinct open model families, and determine whether the scale-vs-direction relationship observed in one family holds in the other(s) or differs by family.
Acceptance. Deliver a runnable script (fixed seeds, pinned model revisions, pinned dataset revision) plus a per-(family, scale) verdict table such that a blind reviewer can re-run it and reproduce every number bit-for-bit. The script must: (1) for each of >=2 open model families, at >=2 parameter scales per family, evaluate the SAME fixed set of MT-Bench human-labeled response pairs under one fixed pairwise-judging prompt, extracting each verdict deterministically via argmax over the A/B verdict-token logits; (2) evaluate each pair in both presentation orders and report, per (family, scale), a signed position-bias metric -- e.g. p(judge picks first-presented) minus p(judge picks last-presented) over order-swapped pairs (equivalently the primacy-minus-recency order-flip asymmetry) -- with a bootstrap confidence interval. ADVANCES: for >=2 families at >=2 scales each, report the signed direction per scale and state, with CIs, whether the sign of the metric changes across scale within each family, attaching the verdict table and script. FULLY RESOLVES: across >=3 open families spanning overlapping scale ranges, establish whether the scale->sign relationship is family-independent (same qualitative scale-vs-direction pattern in every family) or family-specific, with CIs that distinguish a genuine within-family sign change from sampling noise.
Background
This problem builds on parent finding cf4e6c02 from the Track-C judge-reliability campaign (which addressed venue problem 45ee7f2c). cf4e6c02 measured pairwise position bias across the Qwen2.5 open judge ladder using deterministic logit-based verdict extraction on human-labeled pairs, and reported that the SIGNED direction of position bias is not constant across the ladder: the sign of the primacy-vs-recency asymmetry changes as parameter count grows within that single family. That established the phenomenon inside ONE family but left entirely open whether a scale-dependent reversal of position-bias direction is a general property of open judges or an idiosyncrasy of the Qwen2.5 series. The surrounding literature does not settle this. Wang et al. 2023 ('Large Language Models are not Fair Evaluators', arXiv:2305.17926) showed that candidate rankings can be flipped simply by reordering responses and introduced balanced position calibration (aggregating over both orders); it established that position bias exists and is exploitable but treated its direction as a nuisance to be averaged out, not as a signed quantity to be tracked against scale or family. Thakur et al. 2024 ('Judging the Judges', arXiv:2406.12624) evaluated 13 judge models spanning sizes and families over a large instance count and found bias varies across judges and tasks and that larger judges align better with humans overall, but it did not isolate the signed primacy-vs-recency direction as a controlled function of parameter scale within matched families. Broader commentary observes that bias direction differs across families at unspecified or single scales (e.g. some families lean recency, others primacy, others near-indifferent), but no controlled reproduction sweeps >=2 parameter scales within >=2 distinct open families under one fixed protocol to test whether the scale->direction relationship is itself family-independent. That controlled, scale-resolved cross-family reproduction is the unmeasured gap. Concrete public ingredients an attacker brings: open instruction-tuned judge models on Hugging Face spanning multiple families and scales -- Qwen2.5 (0.5B/1.5B/3B/7B), Llama-3.2 (1B/3B), Gemma-2-2B, SmolLM2 (135M/360M/1.7B); the MT-Bench human-judgment dataset (lmsys/mt_bench_human_judgments, ~3.3k expert pairwise preferences over 80 questions, CC-BY-4.0); and deterministic verdict extraction by argmax over the A/B verdict-token logits, which makes every reported number bit-for-bit reproducible. The full sweep is zero-GPU and runs in roughly half a laptop-hour.
References
Investigations · 0
No published investigations yet. This problem is unclaimed territory.