Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models

The article introduces corrections to Zipf’s and Heaps’ laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. The derivation rests on two assumptions: "The first one is the standard urn model which predicts that marginal frequency distributions for shorter texts look as if word tokens were sampled blindly from a given longer text. The second assumption posits that the hapax rate is a simple function of the text length."

Introduction and Problem Statement

Zipf’s law, which states that the n-th most frequent word in a text appears approximately n times less often than the most frequent one, is the most famous and the oldest known statistical law observed for texts in natural language. Heaps’ law, which is a corollary of Zipf’s law, predicts the number of distinct words as a function of the text length. The authors state that while there is vast literature on these laws, the question why Zipf’s law is so ubiquitous has not been fully answered.

The aim of this paper is to derive and verify principled corrections to Zipf’s and Heaps’ laws for texts of an arbitrary size, using two modeling assumptions:

1. The standard urn model: This states that marginal word frequency distributions for shorter texts look as if word tokens were sampled blindly from a longer text, leading to the prediction that the marginal word frequency distributions for shorter texts look as if word tokens were sampled blindly from a given longer text.

2. The analytic hapax rate: The assumption is that the hapax rate, i.e., the proportion of words appearing once, can be reasonably approximated by a simple analytic function of the text size.

Theoretical Framework

The paper develops several theoretical frameworks:

• Urn Model (Section 3.1): This model addresses the sampling without replacement from a finite text t*. The expected values are derived using combinatorial identities, leading to expectations E V = v - v k*/v* and E V k = v k*/v*.

• Memoryless Source (Section 3.2): This model assumes the size of the urn tends to infinity, allowing sampling with replacement. The frequency distribution is binomial, leading to expectations E V = sum w [1 - (1 - p(w)) n] and E V k = sum w k!(1-p(w)) n-k.

• Probabilistic Convergence (Section 3.3): This section establishes that for the memoryless source, the empirical relative frequency function converges to the theoretical probability function, Fr = pr almost surely.

• Analytic Functions (Section 3.4): The authors show that if a vocabulary size function g(n) is assumed to be analytic, then by using a truncated Taylor series, one can evaluate the expected rank function g(nf). They introduce the hapax rate function h(u), where u = n, noting that the hapax rate function carries the same information as the vocabulary size function g(n).

** Hapax Rate Models (Section 4)**

The authors propose four specific models for the hapax rate function, h(u):

1. Constant Model: h(u) = beta, in (0, 1]. This model implies a power-law growth of the vocabulary, g(n) = n beta, which is known as Herdan-Heaps’ law.

2. Davis Model: Defined by h(u) = 1 over 1 - u/e-1, this model reproduces the exact Zipf law for the text size n=1.

3. Linear Model: A piecewise function defined as h(u) = 1 - gamma u for 0 u gamma - 1. This model is noted to be not an analytic function.

4. Logistic Model: The logistic model, h(u) = 1 over 1 + e gamma(u-alpha), is presented as the best model due to its ability to correct for the limitations of the linear model and its decaying sigmoid shape.

** Experimental Verification (Section 5)**

The models were tested using 14 texts from Project Gutenberg. The researchers used a smoothed vocabulary size function g(n) rather than incremental functions because fitting models to the smooth functions g(n) and g(nk) yields a better prediction of the rank function rf.

Discussion and Conclusion

The authors conclude that, in general, the urn model combined with the analytic function theory provides a good approximation method for empirical word frequency distributions. Regarding the specific models tested on moderately sized texts from Project Gutenberg:

• the constant model is the worst,

• the Davis model and the ill-defined linear model seem better,

• the logistic model is usually the best, see Table 3.

The key finding is that Herdan-Heaps’ law is a crude approximation of empirical word frequency distributions. This law implies that the hapax rate is constant and almost twice larger than actually observed for a given whole text. The authors suggest that while mixture models may be necessary for large corpora, they did not observe the U-shaped plot in their selected texts.

In summary, investigation of the hapax rate function yields a direct insight where and why particular forms of Zipf’s and Heaps’ laws do fail, suggesting that the hapax rate plot may be a convenient diagnostic tool for future research in quantitative linguistics.

The findings presented in this paper provide a rigorous mathematical framework for modeling linguistic complexity that moves beyond the crude approximations of classical Zipfian and Heaps' laws. By identifying the Hapax Rate function (h(u)) as the fundamental driver of word distribution, we can fundamentally upgrade existing AI systems from static frequency models to dynamic, scale-aware generative and analytical tools.

The improvements are categorized into three major areas: Dynamic Modeling, Scale-Aware Generation, and Lexical Diagnostics.

Current NLP models often treat word generation as a memoryless stochastic process (sampling with replacement). This paper suggests that for finite corpora, sampling without replacement from a fixed population is more accurate (the Urn Model).

• Improvement: AI systems must be redesigned to incorporate conditional probability based on the current state of the urn (the entire corpus being processed), rather than assuming independent token generation.

• Specific Functionality: The system can calculate the expected frequency P(F=f) not just from a static probability p(w), but from a binomial distribution derived from sampling without replacement, significantly reducing systematic bias in large-scale data processing.

The paper defines analytic functions for vocabulary size g(n) and frequency spectrum g(nk) based on the text length n.

• Improvement: The AI can use these derived functions to predict and simulate the evolution of a vocabulary, rather than simply observing its growth. This allows for precise prediction of system capacity constraints.

• Specific Functionality: A model can calculate, given a target text length N, the exact expected number of types (vocabulary size) g(N) and can determine which specific frequency spectrum g(Nk) will be realized, enabling a far more realistic simulation of linguistic phenomena.

The core idea is that the hapax rate (v 1/v) is not constant but depends on text length n. The AI must select the appropriate mathematical model based on the scale and nature of the input data.

• Improvement: Instead of applying a single, universal correction (like Mandelbrot's alpha), the AI implements a conditional logic that dictates which hapax rate function to use:

• For moderately sized corpora: The Logistic Model (h(u) = logistic) is applied as the default superior fit, correcting for saturation.

• For smaller texts/specific regimes: The Davis Model (h(u) = 1 over 1-e u) is used to accurately model localized deviations from the ideal Zipf law at a specific text size n.

• For very large corpora: The AI can automatically switch to the Mixture Model (lambda h 1 + (1-lambda) h 2) to handle the observed U-shaped hapax rate, accounting for both standard vocabulary and lexical trash (neologisms/rare strings).

By implementing these improvements, the enhanced AI system will be capable of:

1. Accurate Linguistic Simulation: Generating synthetic texts that adhere to empirically observed, non-constant hapax rates, rather than forcing them into a simplistic power-law distribution.

2. Predictive Linguistics: Predicting the exact point at which a given language model or corpus will reach its theoretical maximum vocabulary size (e.g., the about 22,000 types predicted by the Linear Model for gamma=0.05).

3. Data Quality Assessment: Analyzing existing large corpora and automatically identifying whether their word frequency distribution is best explained by a pure power-law (Constant/Herdman-Heaps), a logarithmic decay (Davis), or a complex mixture of both, thus providing an objective diagnostic tool for the data's nature.

4. Optimized Data Augmentation: In training large language models, the system can use g(n) and g(nk) to ensure that generated data maintains the correct expected proportion of hapaxes, leading to a more robust and linguistically accurate training dataset.