Do Matryoshka sparse autoencoders reduce feature absorption on Gemma-2-2B relative to standard SAEs? (SAEBench first-letter test)
Statement
Feature absorption is a failure mode of sparse autoencoders (SAEs) in which a latent that should represent a general concept (e.g. 'token starts with the letter L') systematically fails to fire on specific tokens because a more specialized latent has 'absorbed' that case, leaving holes in the general feature. Matryoshka SAEs (nested dictionaries trained to reconstruct at multiple prefix widths) were proposed to mitigate absorption by forcing coarse concepts into the earliest latents. Question: on a residual-stream layer of Gemma-2-2B (a layer with released Gemma Scope SAEs), does a Matryoshka SAE achieve a lower mean feature-absorption score than a standard JumpReLU/TopK SAE at matched sparsity $L_0$ (within $\pm 10\%$) and matched loss-recovered (within $\pm 1\%$), using the SAEBench first-letter absorption metric averaged over the 26 letters? Report the mean absorption fraction for each SAE type and their difference.
Acceptance. FULLY RESOLVES: a reproducible script (fixed seed; pinned Gemma Scope checkpoints and/or released Matryoshka-SAE training config) that computes the SAEBench mean first-letter absorption score for (a) a standard SAE and (b) a Matryoshka SAE at matched $L_0$ (within $\pm 10\%$) and matched CE-loss-recovered (within $\pm 1\%$) on the same Gemma-2-2B layer, and reports either a statistically significant reduction (paired test across the 26 letters, $p<0.05$) or a demonstrated null (equivalence within an absorption-fraction margin of 0.02). PARTIAL: the comparison at unmatched sparsity, or the same experiment on GPT-2-small with locally trained SAEs. Deliverable: code + SAE checkpoint IDs + the per-letter absorption table.
Background
Feature splitting and absorption were characterized by Chanin, Wilken-Smith, Dulka, Bhatnagar & Bloom, 'A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders' (arXiv:2409.14507, 2024). SAEBench (Karvonen et al., 'SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability', arXiv:2503.09532, 2025) operationalizes absorption via a first-letter classification probe plus k-sparse probing on SAE latents, and reports that JumpReLU/TopK SAEs at low $L_0$ exhibit substantial absorption. Sharkey et al., 'Open Problems in Mechanistic Interpretability' (arXiv:2501.16496, 2025), Section 2.1.2 (problems with sparse dictionary learning) lists robustness of the sparse decomposition as an open problem. Gemma Scope SAEs (Lieberum et al., arXiv:2408.05147, 2024) and Matryoshka SAE training code are openly released. Whether Matryoshka SAEs actually close the absorption gap at matched sparsity and fidelity on a standard open model is not settled.
Investigations · 0
No published investigations yet. This problem is unclaimed territory.