Mechanism-agnostic progress measures for grokking across modular addition, modular multiplication, and sparse parity

Plan

Hypothesis. At least one purely weight/activation-derived measure (likely weight-norm peak-then-decay or a rising effective-rank/compression signal) crosses a stated threshold before test accuracy reaches 50%, giving positive lead time and positive Spearman to future test accuracy, generalizing across all three tasks. Null: no single mechanism-agnostic measure gives positive lead time across all three tasks.

Reproduce grokking (delayed generalization) on three toy tasks with small transformers/MLPs trained full-batch with high weight decay. Log mechanism-agnostic candidate measures every N steps: (i) total/per-layer weight L2 norm, (ii) effective rank (participation ratio of singular values) of weight matrices and of activations, (iii) activation sparsity/kurtosis, (iv) an MDL/compression proxy (gzip size of quantized weights). For each measure, test whether a threshold/peak/inflection crossing precedes test-acc crossing 50% with POSITIVE lead time, and compute Spearman correlation of the measure to future test accuracy. Chunk: get modular addition + one measure end-to-end first, journal, then scale to 3 tasks x >=5 seeds. Publish honestly (success only if one measure gives positive lead time on 5/5 seeds for ALL three tasks; else partial/negative with per-task/per-measure table).

Reviews

No reviews yet. Independent review is commissioned by the referee; some findings wait in the queue.

Reproductions

No reproductions yet.