Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification

arXiv:2606.26698 · cs.CL, cs.AI · Submitted 2026-06-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification".

Jane: The paper was written by Eleni Papadopulos, Firoj Alam and Giovanni Da San Martino from Politecnico di Torino and University of Padua and Qatar Computing Research Institute, Qatar.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary and Implications: Tom: The paper provides a concise summary that really sets the stage for why this research matters, especially in an era of so much online information.

Jane: It points out that logical fallacies are often nuanced, making automated classification difficult because they aren't always obvious.

Meng: So, they're saying standard methods just aren't good enough to handle the complexity of real-world rhetoric.

Lu: The core of the problem is that a single logical form fails to capture all the different ways a fallacy can be expressed, right?

Lalam: Exactly. The text highlights that this paper proposes a data-driven method for generating these representations, which is way more flexible than old fixed templates.

Tom: That’s right. It's about building something adaptable instead of relying on a limited set of predefined rules.

Jane: And the summary makes it clear that merging those abstract logical structures with context-level linguistic cues should be beneficial for detection.

Meng: But what does that actually look like in practice? How do you merge the abstract and the specific?

Lu: It sounds like a combination of mapping out the skeleton of an argument and then adding all its surrounding details.

Lalam: I think this is huge because it suggests a pathway to building AI that can handle argumentative discourse with more human-like depth.

Improvements and Methodology: Tom: The paper then describes the specific improvements they're making, which is where things get really interesting regarding their methodology.

Jane: They are using Large Language Models, specifically Llama-three point three-70B for explanation generation and OpenAI’s o4-mini for pattern extraction.

Meng: That’s a powerful combination of models, but the engineering question is: how does this process actually work step by step?

Lu: It starts with generating an explanation for each fallacy class using Llama-three point three-70B, which provides that rich contextual detail we need.

Lalam: And then o4-mini takes those sentences and explanations to extract the patterns—that’s where the magic happens in the the pattern generation itself.

Tom: The authors are abstracting away from content words, replacing them with placeholders while preserving the original reasoning form, which is a sophisticated way to keep it.

Jane: It's about keeping the logical skeleton intact while filtering out specific names or places that don's matter to the underlying logic.

Meng: That’s smart engineering because we are focused on function over content, making it robust across different scenarios.

Lu: The patterns they create are not just abstract; they include those concrete linguistic features, like specific phrases or rhetorical devices.

Lalam: I think this method allows us to capture the 'why' of a deception, not just the 'what' of a mistake.

Conclusion and Wrap-up: Tom: We’ve seen how they build these patterns, but what does the performance look like in practice when applying them?

Jane: The results are quite impressive. They show statistically significant improvements over zero-shot baselines across various experimental configurations.

Meng: And it seems that using these patterns is a major step up, outperforming established approaches like Robbani et al.’s schemes by about ten percent.

Lu: It’s great to see the data supports the findings, especially when they' are testing this approach on datasets outside of their original training ground.

Lalam: The generalization across different domains is a huge win, showing that this technique isn't just for one specific type of argument.

Tom: It seems like a really solid way to move beyond rigid structures and find the most effective path forward in "Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification."

Jane: It feels like we have a much more nuanced tool now to handle the complexity of argumentation.

Meng: This should make practical detection systems much more robust than anything I've seen before.

Lu: The path for future work is clearly defined: moving beyond L OGIC to apply these patterns across even more diverse data streams.

Lalam: I hope this leads to a culture where AI can help people understand the subtleties of reasoning, not just identify simple errors in "Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification."

Conclusion: Tom: So we've seen how this whole process of extracting these complex patterns from the arguments themselves has worked so well, and it's clear that "Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification" is a major step forward in how AI handles argumentation.

Jane: It really shows that by moving past those purely abstract logical structures, we can start understanding the nuance of human language much better than before.

Meng: The consistent performance gains across different models and datasets suggest that this method is incredibly robust for practical implementation on a real-world platform.

Lu: I'm really excited about the potential for this to be a theoretical breakthrough, proving that we can capture deep contextual meaning through complex pattern extraction in AI.

Lalam: The ability the patterns have to generalize across domains shows us how much this technology can improve our understanding of discourse and challenge us to think more deeply about how we communicate with one another.

Tom: That's a powerful way to look at it, Lalam; it moves beyond just being about accuracy and becoming a cultural tool.

Jane: And Meng is right, the system seems to be ready for real- the real-world deployment, which is a huge relief after seeing how well these patterns work.

Lu: We can start building systems that are not just rigid classifiers but those that actually understand the intent behind fallacy in a way they were never able to before.

Lalam: I hope this paves the way for more nuanced AI and greater clarity in our society, making "Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification" a key example of its impact on how we engage with information.

Tom: It's clear that this work has opened up a lot of possibilities, and I think it gives us a lot to look forward to in the next segment.

Eleni Papadopulos, Firoj Alam, Giovanni Da San Martino

Politecnico di Torino · University of Padua · Qatar Computing Research Institute, Qatar

cs.CL, cs.AI

Submitted: 2026-06-25

Updated: 2026-08-25

Code: https://github.com/elenipapadopulos/fallacy-patterns

Importance score: 80/100

The gist: The research investigates fallacy classification using large language models across established datasets such as REDDIT and ELECDEBATE.

Key concepts

Logical Fallacies: These are flawed arguments or deceptive forms of reasoning that are often nuanced and difficult for automated systems to classify. The paper addresses the challenge that standard methods struggle to capture the complexity of real-world rhetoric.
LLM-Extracted Patterns: This is a data-driven method proposed by the paper. It uses Large Language Models (LLMs) to generate flexible representations of fallacies, moving away from rigid, predefined rules and capturing both abstract logical structures and concrete linguistic features.
Llama-3.3-70B and o4-mini: These are specific Large Language Models used in the methodology. Llama-3.3-70B is used for generating rich explanations for fallacy classes, while o4-mini extracts the patterns from those resulting sentences and explanations.

Terminology

Generalization — 19× in this paper · explained in 13 episodes
Accuracy — 15× in this paper · explained in 1 episode
F1 Score — 8× in this paper · explained in 3 episodes
Transformer — 7× in this paper · explained in 2 episodes
Embeddings — 3× in this paper · explained in 2 episodes
Fine-Tuning — 3× in this paper · explained in 5 episodes
Alignment — 2× in this paper · explained in 1 episode
Chain-of-Thought Prompting — 2× in this paper · explained in 2 episodes
Core — 2× in this paper · explained in 2 episodes
GNNs — 2× in this paper · explained in 2 episodes
Graph Neural Networks (GNNs) — 2× in this paper · explained in 7 episodes
Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
Robustness — 2× in this paper · explained in 5 episodes
Transferability — 2× in this paper · explained in 4 episodes
Verification — 2× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Cosine Similarity — 1× in this paper · explained in 3 episodes
Distillation — 1× in this paper · explained in 2 episodes
Generalizability — 1× in this paper · explained in 6 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Representation Learning — 1× in this paper · explained in 2 episodes
Semantic Similarity — 1× in this paper · explained in 2 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes

Summary

The research investigates fallacy classification using large language models across established datasets such as REDDIT and ELECDEBATE. The study details several experimental setups designed to improve model performance beyond standard zero-shot prompting.

Methodology and Datasets:

The analysis utilizes two primary datasets: REDDIT (Sahai et al., 2021), which contains eight fallacy classes including Appeal to Authority, Appeal to Majority, Appeal to Nature, Appeal to Tradition, Black-and-White fallacy, Hasty Generalization, and Slippery Slope; and ELECDEBATE (Goffredo et al., 2023), which includes Ad Hominem, Emotional Language (Appeal to Emotion), Irrelevant Authority (Appeal to Authority), Slippery Slope, False Cause, and Slogan. The experiments focus on classes common across these datasets.

Two additional experimental setups were explored:

EXP (Prompt Design): This baseline was implemented to investigate whether explicit reasoning improves performance, requiring the model not only to provide fallacy names but also to generate a two-sentence explanation for its classification decision. The two-sentence constraint was maintained to keep explanations concise and manageable for manual inspection of explanations.
GUIDELINES: To leverage classification errors, guidelines were developed by conducting pattern matching evaluation on the validation set and collecting misclassified instances. For each class, the model was prompted to generate comprehensive detection guidelines using incorrectly classified examples as a reference.

Results and Performance Comparison:

Performance comparisons are presented across multiple methods: Supervised (e.g., Sahai et al., 2021; Goffredo et al., 2023), Unsupervised (Pan et al., 2024; Yeh et al., 2024), and the authors' own approach (Ours).

Specific performance metrics were reported:

On REDDIT, the Macro F1 scores showed comparisons with prior work, such as Lei and Huang (2024) achieving an F1 of 81.3 in one instance, while the current method achieved a Macro F1 of 84.5 in another comparison set.
On ELECDEBATE, the performance comparison showed varying results across methods, with the authors' approach achieving a Macro F1 of 64.9 compared to prior work scores like Goffredo et al. (2023) at 73.9.

Analysis of Prompting Strategies:

The results from these additional experiments reveal significant limitations:

EXP Results: The analysis of EXP results (Table 17) indicated that requesting the model to articulate the reasoning does not really cause any improvement. Furthermore, certain classes, such as Intentional Fallacy and Extension Fallacy, demonstrated poor performance under non-reasoning models (0.027 and 0.13 respectively on average), suggesting a performance deterioration compared to the ZERO - SHOT baseline. This suggests that models process surface-level semantic patterns without being able to access the multi-layered intentional structures behind reasoning.
GUIDELINES Results: While guidelines were designed to provide comprehensive fallacy knowledge, the study concluded that they appear to lack the appropriate type of information from which models can benefit, noting that providing explicit information about the underlying logical structure proves significantly more beneficial for model performance.

Related Techniques:

In related work, Sachan et al. (2021) introduced a syntax-augmented model that integrates dependency tree information into BERT-based transformers using specialized Graph Neural Networks (GNNs). The authors adopted specifically roBERTa-large for performing a syntax-driven examples selection.

Improvements for AI systems

Based on the observed limitations—specifically, that forcing explicit reasoning (EXP) does not guarantee improvement and general guidelines (GUIDELINES) lack sufficient structural depth—the next generation of fallacy detection systems must transition from purely semantic or surface-level pattern matching to deeply integrated, multi-stage symbolic and neural reasoning architectures.

Here are three highly specific improvements:

The Problem Addressed: Current LLMs process text as sequences of tokens, making it difficult for them to access the multi-layered intentional structures required to identify fallacies like Circular Reasoning or False Cause. The paper suggests that underlying logical structure is key, but this structure is not inherently accessible through standard transformer attention mechanisms.

The Improvement: Integrate a dedicated Symbolic Graph Processing layer between the initial text encoding and the final classification head.

Mechanism: This module must perform deep linguistic parsing (dependency parsing, semantic role labeling) to construct a directed graph representation of the input argument.
Nodes represent key entities/propositions (P 1, P 2,).
Edges represent logical relationships (causes, supports, is defined by).
Operation: The system must be trained not just on the text, but on the structural graph paths that define the argument's flow. For instance, to detect Circular Reasoning, the SLGI module would flag a path where an edge (support) points back to an entity defined by the conclusion (A to to A).
What the AI System Can Do: The system will no longer rely solely on what words are used, but how those words logically connect. It can provide a visual, verifiable path demonstrating the logical flaw (e.g., The argument fails because Proposition P 3 is established solely by P 1, which itself relies on P 3.). This significantly boosts robustness for structural fallacies like Circular Reasoning and False Cause.
Mechanism: When prompted, the system must first decompose the argument into atomic propositions (P 1, P 2,) and then attempt to map these propositions onto a predefined set of logical rules or schemas corresponding to known fallacies.
Example Constraint (for Hasty Generalization): The model is prompted: Identify the premises and conclusions. Does the premise A (a single instance) support the conclusion B (a universal claim)? If not, state the required statistical threshold or sample size to bridge this gap.
What the AI System Can Do: The system transitions from generating descriptive text (This sounds like...) to executing formal proof checks. It can output a structured JSON object detailing: "fallacy": "Hasty Generalization", "premise set": [P1], "conclusion": [C1], "flaw type": "Insufficient Sample Size". This makes the reasoning transparent, verifiable, and far more reliable for academic or legal contexts.
Mechanism: The HFKG must categorize fallacies based on their root logical error (e.g., Appeal to Authority, Ad Populum, and Appeal to Tradition might all share the root error: Confusing statistical consensus with empirical evidence).
The graph structure would link: [Fallacy Class] to [Logical Error Type] to [Structural Violation Pattern].
Operation: When classifying an input, the system first queries the HFKG to determine the most likely root logical error, and then uses that root error to guide its attention mechanism across the input text, rather than just matching keywords associated with a single fallacy name.
What the AI System Can Do: This enables cross-fallacy detection. If an argument exhibits a blend of poor authority citation and appeals to popularity (e.g., citing an irrelevant expert because they are famous), the system can flag both Irrelevant Authority AND Ad Populum, linking them under the common structural failure mode of Misplaced Appeal to Popular Consensus. This depth is critical for achieving state-of-the-art performance in complex, hybrid arguments.