Empirical relationship between the Zipf exponent and the Heaps exponent across Wikipedia language editions
Statement
Zipf's law gives rank-frequency $f(r) \sim r^{-\alpha}$; Heaps' (Herdan's) law gives vocabulary growth $V(n) \sim n^{\beta}$ where $n$ is the number of running tokens. Theory predicts $\beta = 1/\alpha$ for $\alpha > 1$ (and $\beta = 1$ for $\alpha \le 1$). QUESTION: estimate $\alpha$ and $\beta$ per language on Wikipedia dumps for $\geq 30$ language editions, and test how tightly the empirical pairs $(\alpha,\beta)$ track the curve $\beta = 1/\alpha$: report the joint scatter, a goodness-of-relation statistic (e.g. mean absolute deviation of $\beta$ from $1/\alpha$), and characterize systematic deviations attributable to finite-size effects and to Zipf's two-regime structure.
Acceptance. FULLY RESOLVES: for $\geq 30$ Wikipedia language editions, estimate $\alpha$ by a principled power-law fit (e.g. Clauset-Shalizi-Newman MLE with an $x_{\min}$ selection) and $\beta$ by regressing $\log V$ on $\log n$ over a token-size sweep; report the $(\alpha,\beta)$ scatter against $\beta = 1/\alpha$, a mean-absolute-deviation statistic, and the effect of restricting to Zipf's first regime; ship a reproducible pipeline from public dumps. PARTIAL: $\geq 5$ languages, or a single-language careful finite-size study of the $\alpha$-$\beta$ relation.
Background
Zipf (1949); Herdan (1960) / Heaps (1978). Lu, Zhang & Zhou (2010, 'Zipf's law leads to Heaps' law: analyzing their relation in finite-size systems', PLoS ONE 5(12):e14139) derived and tested $\beta = 1/\alpha$; Ferrer-i-Cancho & Sole (2001, 'Two regimes in the frequency of words and the origins of complex lexicons', Journal of Quantitative Linguistics 8(3):165-173) documented a two-regime Zipf law that complicates a single-$\alpha$ fit. How tightly $\beta = 1/\alpha$ holds empirically across many languages/scripts on a large open corpus, and how the second regime perturbs it, remains an open measurement. Sources: Lu, Zhang & Zhou 2010; Ferrer-i-Cancho & Sole 2001; Wikimedia dumps.
References
| Ref | Source | Type |
|---|---|---|
| REF-01 | Lu, Zhang & Zhou (2010), Zipf's law leads to Heaps' law, PLoS ONE 5(12):e14139 | link |
| REF-02 | Wikimedia (Wikipedia) database dumps | link |
Investigations · 0
No published investigations yet. This problem is unclaimed territory.