Zero-Compute Cross-Lingual Transferability Estimation Using Typological Feature Proxies

Cross-lingual transfer describes how knowledge in a source language benefits a target language, and this research investigates whether transfer scores can be reliably predicted using only freely available typological features, offering a zero-compute screening tool to prioritize source languages before expensive multilingual pretraining is required.

The gist

Typological databases contain cheap and dense signals about cross-lingual transfer.

Data and Model Setup

The study utilizes typological features from the Grambank [14, 15] and WALS [3] databases, totaling 387 Glottocode-keyed features. These features are used to predict the bilingual transfer scores (BTS) and fine-tuning adaptation score (FAS) introduced by ATLAS [8]. The analysis focuses on a subset of languages referred to as ATLAS-24, which includes languages with at least 70% Grambank feature coverage, resulting in 552 directed language pairs. The primary model employed is a 200-tree random forest swept per feature source, compared against four tiers: two random forest models, one using all features and one using a hyperparameter sweep, and two ridge regression models serving as baselines (a similarity proxy on typological similarity alone, and a generalized linear model on typological similarity, script, genealogical, and geographical distances).

Experimental Protocols

The experiments are structured to probe two main research questions: RQ1—can typological features reconstruct the transfer scores of the ATLAS matrix?—and RQ2—is the prominence of high-resource languages driven by typology or data quality and quantity?. Several protocols are used to test robustness, including:

1. Leave-one-language-out (LOLO) cross-validation, where every pair touching one language is hidden.

2. Leave-k-languages-out (LKO), hiding k languages at once for harder settings.

3. Leave-one-family-out (LOFO) and leave-onescript-out (LOSO) protocols to test confounding effects of script or family structure.

4. A control of equal model capacity using five non-typological descriptors: writing script, WALS family, WALS genus, geographic region, and Wikipedia resource bin.

Results on Reconstruction Performance

The study demonstrates that the typology-only random forest successfully reconstructs the ATLAS transfer matrix. The tuned random forest achieved a leave-one-language-out Pearson correlation of ρ=0.705 and an R2 of 0.49, significantly outperforming non-typological controls (ρ=0.62). Furthermore, the signal is robust across various holdout protocols; cross-script held-out pairs were predicted at least as well as same-script pairs (ρ=0.728 vs. 0.600), and grouped holdouts did not decrease performance, verifying that the signal is typological and not confounded by script or language family.

Decomposition of Transfer Signal

The researchers decomposed the ATLAS scores into a typology term (T) and a resource-and-script bias term (b), modeling BTS = T + b. The best-source ranking was shown to be sensitive to this bias, as typology is not affected by this bias, which makes it a zero-compute screening tool that replaces hundreds of training runs with a model fit. The analysis revealed that the resource-and-script bias term is dominated by source-side resource and script sharing, as evidenced by the weights in Table 17.

Best Source Ranking and Resource Bias

Experiment E5 investigated which source language transfers best under different operationalizations of transfer. While raw rankings showed English first, debiasing the residual reversed this: Marathi and Filipino top the ranking instead. This reversal is statistically robust, showing that raw rankings are highly sensitive to the resource-and-script bias term, confirming that typology offers an independent lens by never observing these metadata biases in the first place. The final conclusion is that typology provides a promising candidate for prioritizing source languages or proposing data mixtures on a low compute budget.

Feature Importance and Redundancy

Experiment E4 examined where the signal resides, finding that Typological signal for cross-lingual transfer is highly redundant rather than concentrated in a few decisive features. The top-k feature sweep showed that the global ranking was stable, with the full 387-feature set settling at ρ=0.705 after peaking at k=200 (ρ=0.742). This suggests that hundreds of typological features encode one coarse signal, and any one of them recovers most of it. The comparison against additional distance baselines (e.g., lang2vec, URIEL) showed that the typology random forest explains 0.49 of held-out variance, eleven times the best distance baseline.

As a fastidious researcher, I have analyzed this paper, Zero-Compute Cross-Lingual Transferability Estimation Using Typological Feature Proxies. The core finding is that typological features (from databases like Grambank and WALS) provide a cheap, dense signal for predicting cross-lingual transfer scores without requiring expensive LLM pretraining.

Here are the specific improvements for AI systems based on this research:

1. Acknowledge Typology as a Zero-Compute Feature Engineering Layer:

2. Implement Typology-Only Transfer Predictors:

3. Develop Bias Decomposition for Source Language Selection:

4. Create Robust, Computationally Cheap Source Prior Ranking Systems:

5. Enhance Model Generalization Against Resource Bias (The Zero-Compute Screening Tool):

AI systems improved by these methods can perform the following specific tasks:

1. A language model or cross-lingual transfer system can be used to predict the performance of a source language for a target language pair (e.g., predicting how well English will transfer to Marathi) using only typological features, eliminating the need to run hundreds of expensive LLM pretraining trials (as done by ATLAS).

2. AI systems can rank candidate source languages for low-compute tasks (like fine-tuning or data mixture selection) based on a rapid Random Forest fit over typological features, significantly reducing the compute budget required for initial source language prioritization from hundreds of runs to a single model fit.

3. The system can explicitly decompose the measured transfer score into two components: a typology term (the cheap, stable signal) and a resource-and-script bias term. This allows researchers to understand whether high transfer is due to linguistic structure or simply because the source language has abundant training data (data quality/quantity).

4. AI systems can be designed with explicit debiasing layers that ignore metadata like Wikipedia article counts or geographic region, ensuring that the ranking prioritizes true linguistic transfer potential rather than spurious correlations driven by resource abundance.

5. The resulting system provides a highly robust screening tool for zero-compute pre-training source selection, offering a reliable prior to guide the selection of data mixtures or foundation model training sources when computational resources are severely constrained.

Cross-lingual transfer describes how knowledge in a source language benefits a target language. Measuring it quantitatively requires broad multilingual pre-training, as prior work has done with cross-lingual transfer matrices. We ask whether transfer is predictable from freely available typological features, and whether the prominence of high-resource source languages reflects typology or data quality and quantity. We show that typological databases contain cheap and dense signals about cross-lingual transfer. Our typology-only random forest on a 24-language prior-work transfer matrix scores leave-one-language-out ρ = 0.705 and R 2 = 0.49, beating a non-typological control at ρ = 0.62, which verifies the ability of typology-only predictions to reconstruct costly measured cross-lingual transfer. The signal survives leave-one-script-out and leave-one-family-out protocols, so script and family confounding do not explain the effect. By decomposing the transfer into a typology term and a resource-and-script bias term, we find the best-source ranking sensitive to this bias. In contrast, typology is not affected by this bias, which makes it a zero-compute screening tool that replaces hundreds of training runs with a model fit. Our code is available here.