Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders

Multilingual encoder-based language models are widely adopted for code-mixed analysis tasks, yet we know surprisingly little about how they represent code-mixed inputs internally — or whether those representations meaningfully connect to the constituent languages being mixed.

The gist: Code-mixed adaptation of multilingual encoders improves EN–CM alignment at the cost of EN–HI, reorienting the anchor choice for cross-lingual alignment and leading to better cross-lingual performance on downstream tasks.

Dataset Construction

The study constructed a unified trilingual corpus by merging three publicly available datasets: CM-En Parallel (Dhar et al., 2018), PHINC (Srivastava and Singh, 2020), and the LinCE 2021 multiview Hinglish dataset (Aguilar et al., 2020). This corpus comprises parallel English, Hindi (Devanagari script), and Hindi–English code-mixed sentences. The final dataset contains 21,139 aligned sentence triples. Since the first two datasets did not provide Hindi translations, missing sentences were generated using the Google Translate API and verified using the IndicTrans2 model to ensure consistency.

Cross-Lingual Representation Analysis

The researchers examined how multilingual encoders represent code-mixed text relative to their monolingual counterparts using interpretability tools such as Centered Kernel Alignment (CKA), token-level saliency, and entropy-based uncertainty analysis. Key observations revealed that code-mixed representations remain weakly anchored to both languages under standard pretraining. Specifically, code-mixed representations are largely explained by the English representation space, while native-script Hindi provides complementary signals that reduce representational uncertainty.

Impact of Code-Mixed Adaptation

Continued pretraining on code-mixed data was investigated to understand its effects on alignment. The findings indicated a systematic trade-off: continued pretraining on code-mixed data improves English–code-mixed alignment at the cost of English–Hindi alignment. After adaptation, the "alignment ordering inverts to: EN ↔ CM > EN ↔ HI > HI ↔ CM, suggesting that codemixed data provides alignment signals primarily between English and code-mixed settings. Furthermore, script choice was found to be significant: When Hindi is Romanized during pretraining, HI↔CM alignment degrades, whereas preserving Hindi in Devanagari improves alignment."

Proposed Trilingual Post-Training Alignment Stage

Motivated by these dynamics, the authors introduced a trilingual post-training alignment stage. This stage adapts pretrained multilingual encoders using the objective:

L = Lbase + λ · Lalign (3)

where Lalign encourages semantically equivalent sentences across EN, HI, and CM to occupy nearby regions in the shared embedding space. The Cross-Lingual Alignment Score (CLAS) was proposed as a metric to measure balanced cross-lingual alignment across language pairs. Experiments showed that incorporating this trilingual supervision leads to more balanced cross-lingual alignment across language pairs while largely preserving strong monolingual consistency.

Downstream Task Validation

The effectiveness of the proposed stage was evaluated on two code-mixed classification tasks: sentiment analysis and hate speech detection. The results demonstrated that trilingual alignment encourages stable multilingual representations and improves cross-lingual robustness, leading to improved cross-lingual prediction consistency across CM, EN, and HI inputs for both tasks. For instance, in sentiment analysis, the trilingual alignment applied to Hing-mBERT improved the consistency score from 0.3283 to 0.4464 when training on code-mixed data. Similarly, for hate speech detection with Hing-RoBERTa, it improved the consistency score from 0.6124 to 0.6720 when training on Hindi, suggesting that trilingual alignment is more effective when the finetuning language is Hindi.

Conclusion

The study concludes that while code-mixed pretraining improves EN-CM alignment, the explicit trilingual alignment objective provides consistent and additive benefits over code-mixed pretraining alone, particularly in achieving a more balanced cross-lingual generalization across English, Hindi, and code-mixed representations. This approach not only yields better cross-lingual performance on downstream tasks but also mitigates the degradation of EN–HI alignment.

Limitations

The study notes that the analysis focuses on Hindi–English code-mixing as a representative case study, though the mechanisms are expected to generalize. Future work is suggested to extend the analysis to additional languages and scale the alignment stage with larger and more diverse code-mixed corpora. Additionally, limitations include reliance on automatic translations for corpus construction and focusing primarily on encoder-based models rather than decoder-based LLMs.

Ethical Considerations

The authors noted that the study involves social media text potentially containing offensive or harmful language, which was included solely for research purposes to study linguistic phenomena in code-mixed settings.

Here are the specific improvements that can be made to AI systems, based on the findings of this research, and what those improved systems could achieve:

)

1. Improvements in Multilingual Encoder Pre-training Objectives:

2. Incorporation of a Trilingual Post-Training Alignment Stage (Trilingual Post-Training Alignment Objective):

3. Enhanced Cross-Lingual Representation Consistency Scoring (Cross-Lingual Alignment Score - CLAS):

4. Improved Robustness and Stability in Code-Mixed Classification Tasks:

5. The AI system will be trained using a modified pre-training objective that explicitly incorporates parallel sentence supervision across English, Hindi (Devanagari), and Romanized code-mixed text. The objective function will be:

L = Lbase + λ · Lalign

2. This improved system can achieve:

2.1. More balanced cross-lingual alignment across all language pairs (EN–HI, EN–CM, HI–CM).

2.2. Mitigation of the degradation in the English–Hindi alignment that often occurs when adapting models to code-mixed data (a key trade-off identified in Section 4).

2. The system will utilize a new metric, the Cross-Lingual Alignment Score (CLAS), to evaluate its internal representation geometry, moving beyond simple task performance metrics.

3. The AI system can achieve:

3.1. A quantifiable measure of how balanced the model's shared embedding space is across its constituent languages.

3.2. Identification of which specific training regimes (e.g., pre-training on CM data vs. post-training alignment) yield the most robust, bidirectional cross-lingual performance, guiding optimal fine-tuning strategies for multilingual models.

4. The system will leverage interpretability techniques (CKA and Entropy-Based Uncertainty Reduction) during development to understand internal representation dynamics.

5. This capability allows researchers to:

5.1. Determine the semantic subspace through which code-mixed text is processed, confirming if the model relies excessively on an English-dominant space or if native script signals (like Devanagari Hindi) are effectively providing complementary, uncertainty-reducing information.

6. The system will demonstrate improved cross-lingual consistency on downstream tasks (Sentiment Analysis and Hate Speech Detection).

7. Specifically, the AI system will exhibit:

7.1. Higher Macro-F1 scores and superior performance stability when processing code-mixed inputs compared to standard models, especially in the more complex Hindi–English setting.

7.2. Enhanced robustness in hate speech detection by ensuring that classification consistency is maintained across different language training conditions (CM, EN, HI), leading to more reliable safety evaluations.

Multilingual encoder-based language models are widely used for code-mixed analysis, yet their internal representations of code-mixed inputs -- and their relationship to the constituent languages -- remain poorly understood. Using Hindi-English as a case study, we construct a unified trilingual corpus of parallel English, Hindi (Devanagari), and Romanized code-mixed sentences. We then probe cross-lingual representation alignment in standard multilingual encoders and their code-mix-adapted variants using CKA, token-level saliency, and entropy-based uncertainty analysis. We find that while standard models align English and Hindi well, code-mixed inputs remain loosely connected to either language -- and that continued pre-training on code-mixed data improves English-code-mixed alignment at the cost of English-Hindi alignment. Interpretability analyses further reveal a clear asymmetry: models process code-mixed text through an English-dominant semantic subspace, while native-script Hindi provides complementary signals that reduce representational uncertainty. Motivated by these findings, we introduce a trilingual post-training alignment objective that brings code-mixed representations closer to both constituent languages simultaneously, yielding more balanced cross-lingual alignment and downstream gains on sentiment analysis and hate speech detection -- showing that grounding code-mixed representations in their constituent languages meaningfully helps cross-lingual understanding. Code is available at: https://github.com/debajyotimaz/tri align EMNLP 2026.