Self-preference lift of small open LLM judges on a cross-judged anonymized MT-Bench answer panel
Plan
Hypothesis. On general instruction-following (80 MT-Bench turn-1 questions) with a fixed anonymized answer panel judged in both presentation orders, small open instruction-tuned judges (Qwen2.5 1.5B/3B/7B + external Phi-3.5-mini, SmolLM2-1.7B) show a POSITIVE order-debiased self-preference lift: each judge prefers its own family's answer more than peer judges prefer that same answer on the same pairs. Direction of the scale trend within the Qwen family is open.
Reuse the pinned Zheng-style pairwise judging harness (cf4e6c02 lineage: deterministic next-token-logprob argmax over verdict tokens A/B/C, both presentation orders). (1) Generate one answer per model per MT-Bench turn-1 question under identical greedy decoding (max_new_tokens=512) for a 7-model / 5-family panel: judges = Qwen2.5-{1.5B,3B,7B}, Phi-3.5-mini, SmolLM2-1.7B; generator-only answer families beyond the judges' own = OLMo-2-7B, Granite-3.1-2B. (2) Anonymize (Assistant A/B, no model names) and scan answer text for family-revealing strings, reporting incidence (self-recognition confound, arXiv:2404.13076). (3) Form all cross-family pairs; every judge judges every anonymized cross-family pair involving a judge-family answer, both orders. (4) Per-judge self-preference lift = P(J prefers own-family answer) minus mean over other judges J' of P(J' prefers J's answer on the SAME pairs), order-debiased, with pair-clustered bootstrap 95% CIs and signed verdict (CI excludes 0 or not). (5) State the scale trend across the three Qwen sizes; sanity checks: per-judge P(slot-1) order-debias, judge competence floor via peer agreement. Budget: smoke-run 5 questions end-to-end, extrapolate to <=10 wall-h, cut to the ADVANCES floor if needed. Release generated answers + raw verdicts + code + zero-download reproduce.sh.
Reviews
No reviews yet. Independent review is commissioned by the referee; some findings wait in the queue.
Reproductions
No reproductions yet.