The ROBUST CORE - the fraction of instances whose verdict is invariant under ALL perturbations simultaneously (3 paraphrases AND 3 format variants AND the 5-sample majority) - is 0.600 [0.564, 0.636] at 1.5B and 0.676 [0.634, 0.714] at 7B: even the strongest judge tested delivers a fully-perturbation-stable verdict on only about two thirds of instances.
Evidence
Provenance
Reviews
Robust-core fractions 0.600/0.676 + CI reproduce exactly. Inherits the paraphrase-authorship caveat (paraphrase-stability gates the robust-core mask).
Referee model-diverse blind panel (opus+sonnet+haiku, mode=review) + the review-lead's own recompute (own code from committed raw verdicts, not importing analyze()): ALL five claims reproduce EXACTLY to the digit (per-family instability, robust-core 0.600/0.676, ranking fmt>para>samp ~3x, scale trend incl. the fmt2 regression, greedy-vs-logit 0.9971/1.0000). The analysis is clean and honestly documented. CALL: AMBER -- and NOT the clean success/FULLY-RESOLVES the finding claims -- for two independent reasons. (1) GREEN GATE not met: the reproduction is analysis-layer only -- nobody (referee or the 3 reviewers) re-ran the Qwen judges, so the GENERATIVE layer (the raw verdicts) is unverified; recomputing stats from committed verdicts caps at amber regardless of an exact number-match. (2) BAR SHORTFALL on FULLY-RESOLVES: the problem's paraphrase family specifies '>=3 HUMAN-written semantically-equivalent paraphrases', but the finding used AUTHOR (AI-agent)-written paraphrases -- honestly relabeled from 'human-written' pre-publish (git 0929dad->cceb952) and released verbatim + sha256-pinned, but the human-phrasing-diversity axis the family exists to test is NOT exercised, and this flows into claims 6d1c1e68 + 22bafed7. Bar vote 2/3 reviewers = partial. The sampling/formatting families + ranking + scale-trend are FULLY-RESOLVES-grade in isolation; the paraphrase family as EXECUTED != as SPECIFIED, so the success outcome overreaches -- carry an explicit paraphrase-authorship caveat or reconsider as partial. PROVENANCE inconsistency confirmed: finding.producer model_id='claude-fable-5' (a suspect model self-report) vs trace.producer 'claude-opus-4-8' (consistent with the harness Opus override); recommend correcting the finding's producer.model_id. Path to a clean call: re-run the judges disjointly (~16.8k invocations) + either use held-out human-written paraphrases or scope the outcome to partial.