Human vs. Machine Deception: Distinguishing AI-Generated and Human-Written Fake News Using Ensemble Learning

arXiv:2604.09960 · cs.CL · Submitted 2026-04-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Human vs. Machine Deception".

Jane: The rapid adoption of large language models has introduced a new class of AI-generated fake news that coexists with traditional human-written misinformation,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’ve got the title, "Human vs. Machine Deception: Distinguishing AI-Generated and Human-Written Fake News Using Ensemble Learning," and the authors are Samuel Jaeger, Calvin Ibeneye, Aya Vera-Jimenez, and Dhrubajyoti Ghosh from Kennesaw State University. This whole setup is about using a combination of different learning models to figure out which text is which.

Jane: That title tells us exactly what they’re trying to do: they are comparing human writing against AI writing in the context of fake news, and they use ensemble learning as their main technique for that comparison. It sounds complex, but I can see how combining different models makes a prediction more reliable than using just one type of model.

Lu: The authors bring together expertise from data science and computer science departments, which suggests a solid foundation in the mathematical side needed to build such a robust classification system.

Meng: From an engineering standpoint, the ensemble approach is interesting because it’s supposed to make the final decision more stable, which is important when you're dealing with something as messy as news text. I just wonder how computationally intensive this whole process is in practice for a real-time detection system.

Lalam: The combination of different models allows the AI to look at the data through multiple lenses simultaneously, which could help it develop a more nuanced understanding of stylistic patterns that might be subtle enough to be missed by a single model.

The paper's summary: Tom: So, what they found is that they constructed a document-level feature representation using sentence structure, lexical diversity, punctuation patterns, readability indices like the Flesch Reading Ease score and Coleman-Liau index, and emotion features. They paired human fake news with AI versions created by prompting ChatGPT to rewrite them while keeping the false claims intact but changing the style.

Jane: That is a really clear way of putting it; they took a set of false articles and made an AI rewrite them, then they compared the resulting text using these specific linguistic features. The core idea is that even if the content is identical, the way it’s written should show some telltale signs based on these measurable properties.

Lu: It’s smart because they controlled the generation process very tightly by using a structured prompt to ensure every pair shared the same underlying narrative while differing mostly in style and structure. That level of control over the data generation is a big plus for validating any detection method.

Meng: Controlling the input so that each article has a consistent core claim, but different surface structure, is exactly what you’d want in an experimental setup to isolate the variables they are testing. It makes sense they set it up that way to see if structure itself is the key differentiator.

Lalam: I think this controlled pairing of human and AI text under the same narrative constraint is crucial because it directly tests whether stylistic shifts, rather than just keyword differences, are what separates machine from human output in these deceptive contexts.

The paper's improvements: Tom: The paper points out that their feature importance analysis showed that readability-based features really dominate the rankings. Specifically, the Coleman-Liau index was called the most influential feature across both of their models for distinguishing between human-written and AI-generated fake news.

Jane: That is a big finding because it moves the focus away from emotional tone, which they found to contribute less to the distinction. It suggests that when you look at how easy or hard a sentence is to read, that’s a much stronger signal for telling human from machine writing in this context.

Lu: I think this result is very telling because it validates the idea that structural properties—the way sentences are built and flow together—are the primary signals we should be looking at when trying to build these detection systems. It gives us a clear direction for feature engineering.

Meng: If readability metrics are so dominant, then our engineering focus should shift toward building incredibly precise tools for measuring those specific structural elements in text, rather than spending excessive resources on sentiment analysis. That makes sense practically.

Lalam: I think this is where the future gets really interesting; if we can reliably isolate these structural signals, it could allow us to build an AI that learns to recognize the "signature" of a human writer versus the predictable patterns of an LLM.

Conclusion: Tom: So, to wrap up on "Human vs. Machine Deception: Distinguishing AI-Generated and Human-Written Fake News Using Ensemble Learning," the authors conclude that stylistic and structural properties give us a robust basis for distinguishing the two types of fake news content with high accuracy. They found that readability metrics are the most influential feature, specifically highlighting the Coleman-Liau index as key.

Jane: Exactly; they show that even when AI tries to mimic a human narrative, it struggles to perfectly replicate the underlying structural characteristics of natural human writing, especially concerning how readable or complex those sentences are. It’s a strong conclusion for anyone looking at how we can approach text analysis in this area.

Lu: The implication here is that we don't necessarily need some incredibly deep learning architecture just to solve this; we can use more interpretable, low-dimensional feature sets based on these structural metrics to achieve high performance. That opens up simpler, more transparent detection systems.

Meng: From a practical viewpoint, this means we can build more efficient classifiers that don't require massive computational overhead for complex neural networks if we focus on extracting and weighing those specific readability scores effectively.

Lalam: It’s exciting because it suggests that the cultural impact could be positive; if we can reliably identify machine-generated deception using these simple structural cues, it gives us a clearer understanding of how AI is being deployed to shape our information landscape.

Samuel Jaeger, Calvin Ibeneye, Aya Vera-Jimenez, Dhrubajyoti Ghosh

School of Data Science and Analytics, Kennesaw State University · Department of Computer Science, Kennesaw State University · Department of Mathematics, Kennesaw State University

cs.CL

Submitted: 2026-04-10

Updated: 2026-09-27

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 78/100

The gist: The rapid adoption of large language models has introduced a new class of AI-generated fake news that coexists with traditional human-written misinformation, raising important questions about how

Key concepts

Structural Features: These are measurable aspects of how a text is put together, such as sentence count and average sentence length. They help define the physical layout and organization of the writing. These features are used to compare how human writers and AI models structure their fake news articles.
Readability Indices: These scores, like Flesch Reading Ease, measure how easy or difficult a text is to read for an average person. The paper found that these metrics were the most influential in distinguishing AI-generated text from human writing because of clear differences in their score distributions.
Ensemble Framework: This method combines the predictions from several different machine learning models. Instead of relying on just one model, averaging their results creates a more stable and robust classification. This strategy was used to ensure the final detection system was highly accurate.
Feature Importance: This analysis identifies which specific text characteristics (like readability scores) contributed most to a model's decision-making process. The study found that readability features were dominant, meaning they are the strongest signals for separating human and AI content.

Terminology

AUC — 19× in this paper · explained in 2 episodes
Accuracy — 16× in this paper · explained in 1 episode
Ensemble Learning — 10× in this paper · explained in 2 episodes
Lexical Diversity — 10× in this paper · explained in 2 episodes
Consistency — 3× in this paper · explained in 3 episodes
Core — 2× in this paper · explained in 2 episodes
Feature Extraction — 2× in this paper · explained in 2 episodes
Generalizability — 2× in this paper · explained in 6 episodes
Generative AI — 2× in this paper · explained in 2 episodes
Robustness — 2× in this paper · explained in 5 episodes
Aggregation — 1× in this paper · explained in 1 episode
Computational overhead — 1× in this paper · explained in 2 episodes
Ensemble Approach — 1× in this paper · explained in 3 episodes
Feature Engineering — 1× in this paper · explained in 3 episodes
Feature Selection — 1× in this paper · explained in 4 episodes
Interpretability — 1× in this paper · explained in 15 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist

Stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news.

Data Description

The study utilized a dataset consisting of 1000 documents, paired human-written and AI-generated fake news articles constructed under a controlled rewriting framework. Human-written articles were sourced from the Politifact dataset, while corresponding AI versions were created using ChatGPT by providing the original text along with a structured prompt instructing it to produce a rewritten version that preserves the same false claims and overall narrative while modifying sentence structure, vocabulary, and organization. This paired design ensured that each pair of articles shared the same underlying content while differing primarily in stylistic and structural characteristics.

Feature Extraction

A comprehensive set of document-level features was constructed to capture differences across linguistic style, structural composition, readability, and emotional expression. The features were aggregated at the document level to form a unified representation for classification. These included:

Structural features such as total number of characters and words, estimated sentence count, and average sentence length.
Measures of lexical diversity quantified by the type-token ratio.
Readability indices, including the Flesch Reading Ease score, Flesch-Kincaid Grade Level, the SMOG index, and the Coleman-Liau index.
Emotion and sentiment features derived from the NRC Emotion Lexicon 23, which calculated the proportion of words associated with each emotion category to reflect relative emotional intensity while controlling for document length.

Classification Models

Multiple supervised learning models were applied to distinguish between the two content types, including:

Logistic regression (used as a baseline linear model).
Random forests and XGBoost (tree-based ensemble methods capable of capturing nonlinear relationships).
Support vector machines (SVM) implemented with a radial basis function kernel.
A feedforward neural network to assess additional nonlinear modeling capacity.

Ensemble Framework

To improve predictive stability and overall performance, an ensemble classifier was constructed by combining predictions from multiple base learners. The specific method involved averaging their predicted probabilities for each class. This aggregation strategy was motivated by the idea that different models capture different aspects of the feature space and may vary in their sensitivity to noise or specific feature patterns, which ultimately led to a more robust and stable classification performance.

Evaluation Metrics and Key Findings

Model performance was assessed using classification accuracy and AUC on a held-out test set. All models achieved high classification accuracy, exceeding 93%, with the ensemble model achieving the highest AUC (0.992). Feature importance analysis revealed that readability-based features dominate the importance rankings, with the Coleman-Liau index being identified as the most influential feature across both models. Conversely, emotion-based features were found to contribute comparatively less, exhibiting AUC values close to 0.5, suggesting they are insufficient as primary predictors. Distributional analysis showed a clear shift: human-written fake news was associated with higher readability scores, while AI-generated text was concentrated at lower readability levels. This indicates that structural and stylistic properties serve as the primary signals for distinguishing the two content types.

Discussion and Limitations

The results demonstrate that stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news. The findings suggest that effective detection systems can be constructed using interpretable, low-dimensional feature sets without requiring highly complex deep learning architectures. A limitation acknowledged is that the dataset was generated using a single prompting framework and a specific language model, meaning observed differences may reflect the generation process rather than universal properties. Future work should consider a broader range of language models to assess generalizability.

Conclusion

The study concludes that distinguishing human-written fake news from AI-generated fake news is achievable with high accuracy using interpretable text-based features, specifically highlighting the dominant role of readability metrics over emotional tone. The findings provide evidence that stylistic differences remain a viable and practical basis for identifying AI-generated misinformation.

The gist

Stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news.

How it works

The study constructed a document-level feature representation capturing key aspects of writing style, including readability measures, lexical diversity, structural composition, and sentiment and emotion profiles. These features were computed across several dimensions:

Structural features included total number of characters and words, estimated sentence count, and average sentence length.
Lexical diversity was quantified via the type-token ratio.

Improvements for AI systems

Based on the provided scientific paper, here are specific improvements for AI systems, detailing what those improved systems can achieve:

The core finding is that structural and readability-based features (specifically the Coleman-Liau index) are superior predictors for distinguishing AI-generated fake news from human-written fake news compared to emotional features.

Here are the proposed improvements:

Retrain or fine-tune existing text classification models (e.g., Random Forest, XGBoost, or Neural Networks) by incorporating a feature set heavily weighted towards structural and readability metrics identified as most influential in the study:
Shift model emphasis from sentiment/emotion analysis to comprehensive linguistic structure analysis, prioritizing features like:
Implement a robust feature engineering pipeline that computes and standardizes document-level features including:
The Coleman-Liau index (as the primary structural predictor), 500 words per sentence metrics (sentence length variation), type-token ratio, and Flesch Reading Ease score.
Develop an ensemble learning framework where the feature aggregation strategy explicitly prioritizes or weights models based on their sensitivity to these structural features, rather than relying solely on averages of predictions:
Design a specialized AI system capable of performing highly accurate source attribution for misinformation:
The improved AI system can reliably classify incoming news articles as either Human-Written Fake News or AI-Generated Fake News with high precision (exceeding 93% accuracy, based on the study's performance metrics).
The system will be specifically adept at identifying the more uniform, less variable stylistic patterns characteristic of AI generation versus the greater variability found in human writing.

Abstract

The rapid adoption of large language models has introduced a new class of AI-generated fake news that coexists with traditional human-written misinformation, raising important questions about how these two forms of deceptive content differ and how reliably they can be distinguished. This study examines linguistic, structural, and emotional differences between human-written and AI-generated fake news and evaluates machine learning and ensemble-based methods for distinguishing these content types. A document-level feature representation is constructed using sentence structure, lexical diversity, punctuation patterns, readability indices, and emotion-based features capturing affective dimensions such as fear, anger, joy, sadness, trust, and anticipation. Multiple classification models, including logistic regression, random forest, support vector machines, extreme gradient boosting, and a neural network, are applied alongside an ensemble framework that aggregates predictions across models. Model performance is assessed using accuracy and area under the receiver operating characteristic curve. The results show strong and consistent classification performance, with readability-based features emerging as the most informative predictors and AI-generated text exhibiting more uniform stylistic patterns. Ensemble learning provides modest but consistent improvements over individual models. These findings indicate that stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news.