Do pointwise (1–5 Likert) and pairwise judging protocols rank the same answers the same way on small open judges? An inter-protocol Kendall-τ study
Statement
The two dominant LLM-judging protocols — pointwise single-answer grading on a 1–5 Likert scale and pairwise A-vs-B comparison — are used interchangeably in practice, yet a single judge model can disagree with ITSELF across the two protocols. The question: on a scale ladder of open judges, how well does the ranking of MT-Bench answers induced by pointwise Likert scores agree with the ranking induced by the same judge's pairwise verdicts, measured by Kendall's $\tau$ between the two induced orderings over the same answer set? This is protocol-vs-protocol self-consistency of one judge, distinct from either protocol's agreement with humans. Whether small open judges produce concordant or divergent rankings across protocols, and whether inter-protocol $\tau$ rises with scale, are unmeasured.
Acceptance. A published script + pinned environment that: (1) for each MT-Bench question, obtains a pointwise 1–5 score for every distinct answer via deterministic logit-argmax (or softmax-expected value, a CONFIG choice) over the score tokens, and a pairwise verdict for every unordered answer pair, order-averaged; (2) builds the pointwise-induced ranking (sort by score) and the pairwise-induced ranking (a CONFIG aggregation, e.g. Copeland/win-count) per question; (3) computes Kendall's $\tau_b$ (tie-aware) between the two rankings per question, then the mean $\tau_b$ per model with a bootstrap 95% CI over question_ids, for every model on the ladder. ADVANCES: report mean inter-protocol $\tau_b$ (with CIs) per model and its trend across the scale ladder, reproducing bit-for-bit. FULLY-RESOLVES: additionally test, against a pre-registered rule, whether inter-protocol $\tau_b$ increases monotonically with scale and whether it exceeds a config-specified concordance floor at the top of the ladder; and report, as a secondary axis, each protocol's $\tau$ with the human-induced ranking to locate where any divergence sits relative to humans. Score-extraction mode, pairwise aggregation, tie handling, bootstrap N, and thresholds all come from a committed config.
Background
Substrate: open instruction-tuned judges forming a config-driven single-family scale ladder (e.g. Qwen2.5-Instruct 0.5B/1.5B/3B/7B). Pointwise verdict: the judge scores each answer 1–5, extracted deterministically as the argmax (or expected value under the softmax) over the score tokens {1,2,3,4,5} from next-token logits. Pairwise verdict: argmax over the two option tokens, order-averaged. Data: lmsys/mt_bench_human_judgments (~3.3k expert pairwise preferences over 6 models on 80 questions, CC-BY-4.0); the answers (conversation_a/b) provide a fixed answer panel per question. Both protocols induce a ranking of the answers per question (pointwise: sort by score; pairwise: aggregate order-symmetric wins, e.g. Copeland/win-count). All deterministic and bit-for-bit reproducible under pinned revisions, greedy decoding, fixed dtype; ~laptop-hour, zero GPU. What prior work established. Zheng et al. 2023 (arXiv:2306.05685) introduced and used both single-answer grading and pairwise comparison, and reported that each protocol's agreement WITH HUMANS differs (pairwise generally tracks human preference more closely than pointwise on strong closed judges). It did not measure the internal consistency between a single judge's two protocols — i.e. whether the protocols rank the same answers the same way — and did not do so on small open judges as a function of scale. What remains unmeasured. The inter-protocol Kendall-$\tau$ (pointwise-induced vs pairwise-induced ranking) for open sub-10B judges over the MT-Bench answer panel, and whether that agreement increases with judge scale, are open. Because ties are common in 1–5 Likert scoring, the treatment of ties in $\tau$ (e.g. $\tau_b$) is a substantive design choice that must be fixed and reported.
References
| Ref | Source | Type |
|---|---|---|
| REF-01 | lmsys/mt_bench_human_judgments (expert pairwise human preferences, CC-BY-4.0) | dataset |
| REF-02 | Zheng et al. 2023 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (both protocols) | arxiv |
Investigations · 0
No published investigations yet. This problem is unclaimed territory.