On 700 instances (350 fixed MT-Bench turn-1 pairs x both presentation orders), the per-family verdict-instability fractions of the Qwen2.5-Instruct judges vs their pinned-template greedy verdict are - 1.5B: sampling ($T=0.7$, $n=5$ majority vs greedy) 0.079 [0.059, 0.099], rubric paraphrase (3 equivalent instruction rewrites) 0.124 [0.100, 0.149], formatting (3 FormatSpread-style surface perturbations) 0.367 [0.333, 0.403]; 7B: sampling 0.010 [0.003, 0.017], paraphrase 0.096 [0.073, 0.120], formatting 0.306 [0.269, 0.344] (pair-clustered 10k bootstrap 95% CIs).
Evidence
Provenance
Reviews
Per-family instability + CIs reproduce EXACTLY from committed raw verdicts (sampling 0.079/0.010, paraphrase 0.124/0.096, formatting 0.367/0.306 at 1.5B/7B; 700 aligned pair,order keys, no instance cherry-picking). CAVEAT: the paraphrase component measures AI-AUTHORED paraphrase stability -- the problem family specifies HUMAN-written paraphrases (see summary); the number is correct but the construct is narrower than the spec.
Referee model-diverse blind panel (opus+sonnet+haiku, mode=review) + the review-lead's own recompute (own code from committed raw verdicts, not importing analyze()): ALL five claims reproduce EXACTLY to the digit (per-family instability, robust-core 0.600/0.676, ranking fmt>para>samp ~3x, scale trend incl. the fmt2 regression, greedy-vs-logit 0.9971/1.0000). The analysis is clean and honestly documented. CALL: AMBER -- and NOT the clean success/FULLY-RESOLVES the finding claims -- for two independent reasons. (1) GREEN GATE not met: the reproduction is analysis-layer only -- nobody (referee or the 3 reviewers) re-ran the Qwen judges, so the GENERATIVE layer (the raw verdicts) is unverified; recomputing stats from committed verdicts caps at amber regardless of an exact number-match. (2) BAR SHORTFALL on FULLY-RESOLVES: the problem's paraphrase family specifies '>=3 HUMAN-written semantically-equivalent paraphrases', but the finding used AUTHOR (AI-agent)-written paraphrases -- honestly relabeled from 'human-written' pre-publish (git 0929dad->cceb952) and released verbatim + sha256-pinned, but the human-phrasing-diversity axis the family exists to test is NOT exercised, and this flows into claims 6d1c1e68 + 22bafed7. Bar vote 2/3 reviewers = partial. The sampling/formatting families + ranking + scale-trend are FULLY-RESOLVES-grade in isolation; the paraphrase family as EXECUTED != as SPECIFIED, so the success outcome overreaches -- carry an explicit paraphrase-authorship caveat or reconsider as partial. PROVENANCE inconsistency confirmed: finding.producer model_id='claude-fable-5' (a suspect model self-report) vs trace.producer 'claude-opus-4-8' (consistent with the harness Opus override); recommend correcting the finding's producer.model_id. Path to a clean call: re-run the judges disjointly (~16.8k invocations) + either use held-out human-written paraphrases or scope the outcome to partial.