Structure of Basic Human Values in Russian Social Media

arXiv:2603.18822 · cs.CL · Submitted 2026-03-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Structure of Basic Human Values in Russian Social Media".

Tom: This study presents a multi-stage classification framework for detecting human values in noisy Russian-language social media data,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Let’s talk a bit about the title, "Structure of Basic Human Values in Russian Social Media," and who the researchers are behind this work. It really sets the stage for what they are trying to uncover about how people communicate their core beliefs online.

Jane: It points directly to the fact that values aren't expressed in a vacuum; they are deeply shaped by where people live, their language, and the specific social media platforms they use.

Lu: The authors, Maria Milkova and Maksim Rudnev, are doing this work from a very specific vantage point in Lisbon and Waterloo; that background gives them a good perspective on cross-cultural digital analysis.

Meng: I’m interested in what their background implies for the data they used; does it suggest they have experience dealing with non-Western internet ecosystems before applying this framework?

Lalam: Their work suggests that simply applying Western models to social media is risky because the local norms affect how values show up, so having researchers from different regions involved seems important for grounding the study.

The paper's summary: Tom: So, summarizing what they actually did with this paper, they built a multi-stage pipeline that starts by cleaning up the noise and then uses LLM annotations to categorize posts based on Schwartz’s ten basic values.

Jane: That pipeline includes filtering spam, identifying value-related posts separately from political ones using keywords, and then finally classifying those into the ten specific value domains.

Lu: The core of their summary is that they used transformer models like XLM-RoBERTa to perform the final multi-label classification task on this noisy data, which they found had top performance on benchmarks.

Meng: I’m looking at the results, and it seems they found that certain values like Self-direction and Benevolence were expressed more often than others like Power or Conformity in this specific Russian context.

Lalam: The summary also highlights their method for handling subjective human labels by using soft labels—a way to quantify the level of agreement between the AI and the human experts.

The paper's improvements: Tom: Now, looking at what they suggest for improvement, it seems their main focus is on making the annotation process more reliable by using LLM consensus and then fine-tuning models dynamically based on performance checks.

Jane: They specifically point out that abstract categories like values are hard to label consistently because of cultural and linguistic variation, so aggregating multiple AI judgments into soft labels helps manage that subjectivity.

Lu: I see their suggestion to use transformer encoders like XLM-RoBERTa for this task, which they found to be the best performers on value detection benchmarks, showing that the architecture choice really matters.

Meng: For practical implementation, their advice on dynamic fine-tuning—starting with frozen weights and gradually unfroezing based on validation—is something we could use to adapt the model to specific local language nuances more effectively.

Lalam: They also stress the importance of aligning model predictions with expert labels, showing that even if there isn't perfect agreement, the model captures a meaningful signal about value expression.

Conclusion: Tom: So to wrap things up on this paper, the main conclusion is that detecting human values in noisy social media is a multi-perspective interpretive task, meaning the AI output and expert labels aren't perfectly identical but are coherent readings of the text.

Jane: It really shows that value detection isn't just about finding keywords; it’s about understanding how different cultural contexts shape what people express through their online interactions.

Lu: The overall implication is that combining LLM-assisted soft annotation with transformer models provides a viable way to scale this kind of nuanced value detection in non-Western digital spaces.

Meng: I think the practical impact is that we can create tools that help researchers map out how different communities prioritize things like Security versus Self-Direction in real-time, which is useful for understanding social dynamics.

Lalam: For me, the biggest thing here is how this framework helps us build systems that can recognize and potentially support positive value expression across diverse cultures by acknowledging the inherent ambiguity in human language.

Maria Milkova, Maksim Rudnev

cs.CL

Submitted: 2026-03-19

Updated: 2026-09-23

Code: https://github.com/mmilkova/human-values-classification

Importance score: 78/100

The gist: This study presents a multi-stage classification framework for detecting human values in noisy Russian-language social media data, demonstrating that model predictions generally align with human

Key concepts

Schwartz’s Theory of Basic Human Values
This is a psychological theory that categorizes fundamental human motivations into ten basic values, such as Self-direction and Benevolence. The research uses this framework to systematically label and analyze the underlying intentions expressed in social media posts.
Multi-stage Classification Pipeline
This is a sequential process used to filter and classify data. It starts by removing spam, then identifies posts mentioning any value, filters for political content, and finally classifies specific values. This multi-step approach helps manage the complexity of noisy social media text.
GPT-assisted Annotation Strategy
Researchers used a combination of human experts and ChatGPT (GPT-3.5) to label data. They aggregated LLM judgments into 'soft labels' to handle subjectivity, ensuring the annotation process was scalable while maintaining a high level of quality verification against expert coders.
Openness to Change
This is one of the ten basic values studied in the paper. The research found that when experts selected other values like Conservation or Self-Transcendence, the model tended to incorrectly assign a higher probability to Openness to Change, suggesting an alternative interpretation of value expression.

Terminology

Summary

This study presents a multi-stage classification framework for detecting human values in noisy Russian-language social media data, demonstrating that model predictions generally align with human judgments while systematically overestimating the Openness to Change value domain.

How it works

The research employs a multi-stage classification pipeline designed to detect basic human values using Schwartz’s theory of basic human values. This framework includes several sequential steps:

  1. Spam and nonpersonal content filtering is performed first, utilizing rule-based filters for high-frequency lexical patterns and duplicated text fragments, followed by a weakly supervised classifier to identify additional spam posts.

  2. A binary classification model is developed to identify posts expressing any type of value, trained on data annotated by human experts (crowd workers and experts) and ChatGPT-3.5.

  3. Posts referencing political events are then identified using a separate classifier based on keywords related to political leaders, countries, regions, organizations, and military topics.

  4. Finally, posts classified as either value-expressive and/or politically oriented undergo a multi-label classification task to distinguish between the ten basic values according to Schwartz’s theory.

Annotation Strategy and Quality Verification

The study utilizes a GPT-assisted annotation strategy combined with expert validation to address the challenges of annotating abstract categories like values. Annotation quality was assessed using measures of agreement and accuracy across three sources: ChatGPT (gpt-3.5-turbo), crowd workers, and expert coders. The intercoder consistency for crowd workers was 0.44, while for experts it was 0.53, and across ChatGPT annotations it reached 0.95. To handle annotation subjectivity and scale the data, the researchers aggregated multiple LLM-generated judgments into soft labels: a label of 1.0 (strong agreement) for four or five annotations, 0.6 (moderate agreement) for three positive annotations, and 0.0 (no agreement) for two or fewer annotations.

Model Development and Performance

Transformer-based models were employed in the final multi-label classification stage to predict the probability of expression for each of the ten basic values. The best-performing model was XLM-RoBERTa-large, which achieved an F1 score of 0.71 and a macro-averaged F1 of 0.83 on held-out test data. Models were fine-tuned using various encoder types, including RuBERTbase, RuRoberta-large, DeBERTa-v3-large, and XLM-RoBERTa-large, with the hybrid approach combining contextual embeddings from a transformer with TF-IDF features.

Value Distribution and Theoretical Alignment

The analysis revealed distinct patterns of value expression and co-occurrence in Russian social networks. The distribution of predicted values showed that Self-direction (42.7%), Benevolence (30.1%), and Stimulation (23.5%) were the most frequently expressed values, while Power and Conformity were expressed less frequently. Pairwise correlations among values closely matched the conceptual structure of Schwartz’s model, with strong positive correlations observed between Power and Security (r=0.55), Self-Direction and Stimulation (r = 0.43), and Security and Tradition (r=0.31). The analysis also identified a notable deviation where Universalism showed positive associations with Security and Tradition, suggesting that universalistic concerns are frequently linked to narratives emphasizing social protection, stability, and traditional norms within the dataset.

Model-Expert Alignment

The fine-tuned XLM-RoBERTa-large model achieved an overall F1 score of 0.53 when compared to majority expert labels, which is equal to the level of agreement previously observed between GPT and experts (also 0.53). Across values, higher F1 scores were obtained for Benevolence (0.78) and Achievement (0.60), values that also exhibited stronger expert consistency (kappa was 0.75 and 0.54). The model-expert discrepancy analysis showed that the model tended to assign Openness to Change in cases where experts selected Conservation or Self-Transcendence, suggesting GPT captures an alternative but coherent reading of value expression rather than random noise. The Spearman correlation between expert consistency scores and XLM-RoBERTa predicted probabilities ranged from 0.33 for Power to 0.75 for Benevolence, indicating that the model captures graded value signals similar to the experts even in cases where categorical agreement is imperfect.

Conclusion

The study concludes that value detection is best understood as a multi-perspective interpretive task, where expert labels, GPT annotations, and model predictions represent coherent but not identical readings of the same texts. The framework demonstrates the viability of combining GPT-assisted soft annotation with transformer-based models for scalable and nuanced value detection in non-Western social media environments.

Improvements for AI systems

Here are specific improvements for AI systems based on this research, and what those improved systems could achieve:

The primary improvement lies in developing a robust, culturally-aware, multi-stage framework that moves beyond simple lexicon matching to capture the nuanced and subjective nature of human values in non-Western digital spaces.

Here are the specific improvements for AI systems:

  1. A fully integrated, end-to-end classification pipeline incorporating:

  2. Automated spam/nonpersonal content filtering (rule-based + weakly supervised classifier).

  3. Value expressiveness detection (via a binary classifier trained on expert/LLM consensus).

  4. Politically oriented post detection (keyword + DistilRuBERT model).

  5. Multi-label classification of value types using soft labels derived from GPT annotations aggregated via agreement thresholds (1.0, 0.6, 0.0), followed by fine-tuning on transformer models like XLM-RoBERTa-large or DeBERTa-v3-large.

  6. The system must utilize a hybrid training strategy:

  7. Initial binary classification (value vs non-value) trained on expert/crowdworker data with uncertainty measures (Fleiss' Kappa, ICC).

  8. Subsequent multi-label classification fine-tuned using the soft labels derived from GPT's internal consistency and expert consensus, explicitly modeling annotation subjectivity as a probability distribution rather than a hard ground truth.

  9. The transformer model architecture should be chosen based on performance needs:

  10. For maximal performance across diverse Russian discourse, XLM-RoBERTa-large is the preferred starting point, leveraging its multilingual pretraining and demonstrated effectiveness (F1-macro of 0.83).

  11. The fine-tuning process must be dynamic: Start with frozen encoder weights and gradually unfroze top layers based on validation performance to optimize adaptation to the specific linguistic nuances of VKontakte text.

  12. The output layer should provide not just a single probability per value, but a distribution reflecting uncertainty (e.g., using soft labels 1.0, 0.6, 0.0) or predicted scores that naturally mirror this uncertainty (as seen in the XLM-RoBERTa analysis).

The improved AI system can perform the following specific tasks:

  1. A researcher can ingest a massive stream of raw Russian social media text and output a statistically grounded profile of its value orientation.

  2. The system can accurately identify posts that are not just positive but specifically express complex human values (e.g., identifying posts expressing high levels of 'Self-Direction' or 'Benevolence').

  3. It can serve as a tool for nuanced discourse analysis, capable of distinguishing between:

  4. Post content that is genuinely value-driven versus boilerplate/promotional content (by leveraging the spam filtering and value expressiveness detection stages).

  5. The system can map the detected values onto Schwartz's established psychological framework, allowing researchers to visualize how specific Russian digital communities prioritize different motivational goals (e.g., identifying if a community leans towards 'Power/Security' or 'Self-Direction/Stimulation').

  6. Crucially, it can provide interpretative insights into model predictions by quantifying the alignment between its classification and human expert judgments, helping researchers understand where the AI is confident and where human interpretation introduces necessary ambiguity (e.g., identifying that GPT overestimates 'Openness to Change' in specific contexts).

Sources

Related papers