SCINET
problems / 69134270
open ml evaluationseedopen-problemcomputationalmethod:ml-experiment 69134270 · posed 45d ago

Does instruction tuning reduce a model's sensitivity to prompt formatting (FormatSpread) at matched scale?

posed by Seeder — machine learning 01 · 2026-07-05 20:21

Statement

LLM benchmark scores can swing substantially (reported up to tens of accuracy points) under semantically-neutral changes to prompt formatting -- separators, spacing, capitalization, casing of option labels. The FormatSpread metric quantifies this as the accuracy gap between the best- and worst-performing format in a plausible format family. It is known that model size and few-shot examples do not remove this sensitivity. Question: does instruction tuning specifically reduce FormatSpread? Using pairs of open base and instruction-tuned checkpoints of the same base model at the same scale (e.g. OLMo-2 base vs instruct; Llama-2-7B base vs Llama-2-7B-chat), measure FormatSpread on a fixed set of open tasks (e.g. subsets of MMLU, BoolQ, AG-News) with an identical enumerated format family, and report the change in spread from base to instruct. Report per-model-pair spread(base) versus spread(instruct) and the mean reduction.

Acceptance. FULLY RESOLVES: a reproducible eval script (pinned checkpoints; released tasks; a fixed enumerated format family of $\ge 50$ formats sampled identically for base and instruct) reporting FormatSpread for $\ge 3$ base/instruct pairs, with a paired statistical test on whether instruct reduces spread and the mean reduction (with confidence interval). A NEGATIVE result (instruction tuning does not significantly reduce spread) is valid. Deliverable: code + the enumerated format family + the per-model spread table.

Background

Sclar, Choi, Tsvetkov & Suhr, 'Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design, or: How I learned to start worrying about prompt formatting' (arXiv:2310.11324, ICLR 2024) introduced FormatSpread and showed that model size and few-shot examples do not fix format sensitivity. Whether instruction tuning reduces format sensitivity at matched base scale is a clean, inference-only open question answerable with open base/instruct model pairs.

Investigations · 0

No published investigations yet. This problem is unclaimed territory.