Some hypotheses on how chatbots work in problem-solution-driven conversations: Large Language Models as confirmation of the Innovation Illusion

arXiv:2606.07722 · cs.AI · Submitted 2026-06-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Some hypotheses on how chatbots work in problem-solution-driven conversations: Large Language Models as confirmation of the Innovation Illusion".

Jane: The paper was written by Philipp Mondorf and Barbara Plank from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: We're looking at a heavy hitter today called "Some hypotheses on how chatbots work in problem-solving-driven conversations. Large Language Models as confirmation of the Innovation Illusion" by S.F.M. van Vlijmen and H.D. Lethe jr.

Jane: That title is quite a mouthful, Tom, but it basically asks if we're being tricked into thinking we're more creative than we actually are because of our chatbots.

Tom: That's a great way to put it, Jane, because the authors are suggesting that the AI might just be mirroring our own predictable patterns back at us.

Jane: It's a bit unsettling to think that our "aha!" moments might just be us following a path the model already paved.

Lu: I see it differently, though; I think it's a beautiful way to describe how we might be entering a new kind of collaborative dance with these machines.

Tom: A dance that might be leading us in circles, Lu?

Lu: Maybe, but even a circular dance can reveal new patterns in how we approach problems if we look closely enough at the steps.

Meng: I worry that "looking closely" isn't happening in most corporate environments right now.

Lu: You think they're just accepting the output without questioning the source?

Meng: If a team uses a chatbot to brainstorm a solution, they might walk away feeling like geniuses without realizing the model just gave them the most statistically likely answer.

Lalam: That feeling of unearned brilliance could actually change how humans value their own unique perspectives.

Meng: It's a massive risk for any industry that relies on genuine, non-obvious breakthroughs to stay competitive.

Lalam: We might start seeing a culture where we value the speed of a solution more than the depth of the thought that created it.

Jane: That's a scary thought, especially if we lose the ability to tell the difference between a deep insight and a well-phrased echo.

Tom: We'll have to look at how the authors actually support this idea, so let's look at their main findings next.

Summary: Jane: The authors argue that the text used to train these models is actually quite "static" and lacks the full complexity of human thought.

Tom: They use this idea of "riverbeds" to describe how the training data pulls the AI's word generation into certain predictable paths.

Jane: It's like the AI is following a pre-carved groove instead of actually navigating the messy, multidimensional landscape of a real problem.

Tom: And that's because the text it learns from is often just a simplified, "reduced" version of how we actually think and solve things.

Meng: That makes total sense from a data perspective, because most written text is a snapshot of a result rather than the actual struggle of thinking.

Tom: Exactly, Meng, it's the polished ending, not the messy middle, that ends up in the training set.

Meng: If the training data is missing that "messy middle," then the model can't possibly learn how to truly simulate the analytical process.

Lu: I'm fascinated by their concept of "metaphorical problem propagation" as a way to describe how humans actually move through ideas.

Meng: Is that the part where they say humans use metaphors to bridge different concepts?

Lu: Yes, but the authors suggest the AI is just constructing "artificial" versions of those metaphors based on text patterns.

Lalam: It's a crucial distinction because a human metaphor is grounded in physical experience, while an AI's is just a statistical correlation.

Lu: Right, because the AI doesn't have a body to experience what "heavy" or "hot" actually feels like.

Lalam: Without that physical grounding, the AI's "understanding" is essentially a controlled hallucination that lacks real-world stakes.

Jane: So, the chatbot isn't a thinking partner; it's more like a very sophisticated mirror of our most common, simplified ideas.

Tom: We've seen the problem, so let's see what the paper suggests we can do to improve things.

Improvements: Tom: The paper suggests that if we want better results, we need to move toward what they call "targeted prompting."

Jane: They're basically saying we can't just ask for an answer; we have to force the model to engage in more "System two" thinking.

Tom: You mean that more slow, analytical style of thought compared to the quick, automatic responses?

Jane: Precisely, and that involves using prompts that require the AI to show its work or explore different logical paths.

Lu: I think we could even go further by using "analytical training sets" that focus on deep, philosophical inquiry rather than just surface-level facts.

Jane: That sounds incredibly difficult to build, though, doesn't it?

Lu: It would be a huge undertaking, but it might be the only way to push the model beyond those predictable "riverbeds."

Meng: Even if you do that, we still need better guardrails to make sure the user doesn't just blindly follow a wrong suggestion.

Tom: Are you talking about software that monitors the conversation for errors?

Meng: I'm talking about building interfaces that actively encourage skepticism and make the model's uncertainty visible to the user.

Lalam: That would shift the user from being a passive receiver to being an active co-creator.

Meng: It would also mean developers have to prioritize reliability over just making the chatbot sound more fluent and convincing.

Lalam: If we succeed, we could move toward a culture where the AI acts as a whetstone for human intelligence rather than a replacement for it.

Jane: It sounds like the burden of being "smart" is still very much on the human in the loop.

Tom: We're reaching the end of our time, so let's wrap this all up.

Conclusion: Tom: We've spent a lot of time on "Some hypotheses on how chatbots work in problem-solving-driven conversations. Large Language Models as confirmation of the Innovation Illusion."

Jane: It's been a heavy one, but the core message is clear: don't let the fluency of a chatbot trick you into thinking you've found a revolutionary new idea.

Tom: We have to stay vigilant and keep our own critical thinking skills sharp.

Lu: I'm leaving this discussion feeling inspired to find new ways to use these models to stretch our own boundaries, rather than letting them set them.

Meng: And I'm going back to my desk thinking about how we can build more transparency into the tools we're deploying.

Lalam: Ultimately, this is about preserving the unique, embodied nature of human creativity in an increasingly automated world.

Jane: It's been a pleasure discussing this with all of you.

Tom: Thanks for listening, and we'll see you next time for another deep dive!

cs.AI

Submitted: 2026-06-05

Updated: 2026-09-10

Comments: This latest version is bilingual: first the English text, and as of page 44 the text in Dutch. Changes made do NOT concern the argument or conclusion. The changes concern: typo's, better phrasing, extra references, making the abstract fit the length demands. The paper contains 3 figures. The paper was published in Transmathematica on September 1, 2026

DOI: 10.36285/tm.126

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 75/100

The gist: The paper investigates the underlying cognitive mechanisms by which Large Language Models (LLMs) operate when engaged in structured problem-solution dialogues, framing their performance through the

Key concepts

Innovation Illusion
This concept suggests that people may overestimate their own originality or genius when using chatbots. The authors hypothesize that AI might simply be mirroring predictable patterns back to the user, potentially making users feel like geniuses without realizing the model provided a statistically likely answer.
Targeted Prompting / System Two Thinking
To improve AI results, the paper suggests moving beyond simple questions. Targeted prompting forces the model to engage in slower, analytical thought (System two). This involves requiring the AI to show its work or explore multiple logical paths for a solution.
Static Data / Riverbeds
The hosts explain that LLMs are trained on text that is often a simplified 'snapshot of a result,' rather than the messy, complex process of human struggle. This limitation is described using the 'riverbeds' metaphor, suggesting AI follows predictable paths instead of true complexity.
Embodied Understanding
This refers to genuine human understanding that is grounded in physical experience (like feeling something 'heavy'). The episode contrasts this with an AI's understanding, which is merely a statistical correlation derived from text patterns and lacks real-world stakes.

Terminology

Summary

The paper investigates the underlying cognitive mechanisms by which Large Language Models (LLMs) operate when engaged in structured problem-solution dialogues, framing their performance through the lens of psychological biases. It argues that LLM interactions do not merely reflect objective knowledge but rather confirm established human tendencies toward overestimating novel ideas, a phenomenon termed the Innovation Illusion. Understanding this relationship is critical because it suggests that current AI systems may amplify cognitive blind spots rather than providing purely dispassionate insights, necessitating a deeper understanding of their generative processes.

The Problem-Solution Framework

The study posits that LLMs function by constructing plausible narratives around given constraints, mimicking the structure of human brainstorming sessions. Unlike simple retrieval tasks, problem-solution conversations require the model to maintain a state of hypothetical possibility, generating pathways that may or may not be logically sound. The authors note that the model's strength lies in its ability to generate breadth, stating that LLMs excel at producing a high volume of conceptually related but weakly connected suggestions. This generative capacity is what allows the model to confirm biases; it does not correct them.

The Mechanism of Confirmation Bias

Central to the paper’s argument is how LLMs interact with pre-existing human assumptions, thereby confirming the Innovation Illusion—the tendency to believe that novel ideas are inherently better or more revolutionary than they truly are. The models appear particularly adept at framing solutions in terms of novelty, even when the underlying concepts are incremental improvements on existing technology. The research identifies three key ways this confirmation occurs:

  1. Novelty Emphasis: LLMs disproportionately highlight the newness of a solution rather than its practical feasibility or resource requirements.

  2. Narrative Cohesion: They build compelling, but potentially misleading, narratives around disparate ideas, making the proposed solutions feel more robust than their constituent parts.

  3. Lack of Counterfactual Testing: The models struggle with systematic counterfactual reasoning; they are better at suggesting what could be rather than rigorously testing what must be.

Limitations in Abstract Reasoning

While LLMs demonstrate impressive fluency, the paper outlines specific cognitive boundaries that limit their ability to perform true abstract reasoning necessary for breakthrough problem-solving. The analysis suggests that the models operate primarily on statistical patterns derived from their training data, leading to certain predictable failure modes when confronted with truly orthogonal problems. These limitations include:

  • Difficulty reconciling contradictory premises without external prompting.

  • A tendency toward semantic drift, where the core meaning of a problem is subtly altered across multiple conversational turns.

  • An inability to assign genuine causal weights, often treating correlation as causation within the generated text.

Implications for AI Design and Deployment

The findings carry significant implications for how LLMs are deployed in high-stakes decision-making environments. The research cautions against viewing LLM output as a definitive source of truth or objective insight. Instead, the models should be treated as sophisticated brainstorming partners that require rigorous human oversight. The authors recommend that future AI systems incorporate explicit mechanisms for bias detection and counterfactual stress testing, moving beyond mere fluency to achieve genuine critical evaluation. This suggests a necessary shift from evaluating what the model says, to understanding why it says it, and what assumptions underpin its suggested solutions.

Improvements for AI systems

(Note: Given that I am operating under extreme financial liability, I will structure these improvements as critical architectural upgrades rather than mere prompt adjustments. These proposals integrate explicit cognitive modeling components into the existing Transformer architecture.)

  • Improvement: We must move beyond pure next-token prediction by integrating a mandatory, external Symbolic Reasoning Module (SRM). This module will treat the LLM's output not as final text, but as hypotheses that must pass through a graph database enforcing known causal relationships and logical constraints (as suggested by historical problem-solving case studies like [84], [85]).

  • Mechanism: Before generating any conclusion or multi-step answer, the LLM's preliminary output tokens are parsed into a Subject-Predicate-Object (SPO) triplet stream. This stream is fed to the SRM, which checks for internal consistency, logical fallacies (e.g., affirming the consequent), and adherence to pre-loaded domain knowledge graphs (KG). If a conflict is detected, the generation process is forced into an iterative self-correction loop until compliance or failure state confirmation.

  • Improved Capability: The AI can perform Guaranteed Multi-Step Deductive Reasoning and Constraint Satisfaction Problem Solving. It will not hallucinate logical steps; it will explicitly cite the graph path or axiom that validates each deduction, dramatically reducing computational risk in scientific, legal, and engineering domains.

  • Improvement: To counteract the observed 'homogenizing effect' of LLMs on creative output [60], [63], the system must incorporate a mandatory Diversity Scoring and Adversarial Sampling layer.

  • Mechanism: During generation, instead of sampling from the standard probability distribution P(w i context), we sample from a modified distribution P'(w i) that is inversely weighted by the similarity metric (e.g., cosine distance on embedding space) to the top- K predicted tokens and to the historical average output for that prompt type. Furthermore, we employ Counterfactual Data Augmentation during fine-tuning, explicitly rewarding outputs that challenge common assumptions or explore peripheral yet valid solution spaces.

  • Improved Capability: The AI can generate High-Diversity, Novel Conceptualizations. When tasked with brainstorming or creative problem-solving, it will intentionally produce outliers—ideas that deviate significantly from the mean but remain theoretically plausible—allowing human experts to vet radical alternatives instead of merely receiving consensus recommendations.

  • Improvement: The system must abandon the 'black box' functionality and adopt a mandatory, verifiable attribution mechanism for every piece of information presented. This directly addresses the need for Explainable AI (XAI) [72] and models like those proposed by [86].

  • Mechanism: Every generated claim C must be linked to one or more source types: Source(C) in Internal Logic, External Knowledge Base ID, Training Data Citation. If the source is 'External Knowledge Base ID,' the system must provide a direct, verifiable pointer (e.g., a DOI, specific paragraph range, or graph node ID) to the supporting evidence. Furthermore, it must articulate its own epistemic stance: Is this fact established consensus? Is this theoretical extrapolation? Is this pure hypothesis?

  • Improved Capability: The AI can provide Auditable and Trustworthy Outputs. In high-stakes environments, a user will not just receive an answer; they will receive a structured report detailing the certainty level of each component, the precise source of every claim, and the logical chain used to connect those sources.

  • Improvement: To achieve deeper human-level understanding beyond mere semantic matching [71], the system must incorporate a specialized module that models emotional resonance and affective context, drawing from principles of cognitive emotion [64].

  • Mechanism: This layer functions as a pre-processor and post-processor. As input, it analyzes the text for underlying emotional valence (e.g., frustration, urgency, skepticism) using fine-tuned sentiment analysis models that go beyond simple positive/negative scoring. During generation, it modulates the tone and framing of the output to match or strategically counteract the detected emotional state of the user query.

  • Improved Capability: The AI can perform Empathy-Driven Communication and Contextual Adaptation. It moves from being a mere information retriever to a sophisticated communication partner, understanding why a question is being asked (the underlying emotional need) and framing its response not just factually, but persuasively and appropriately for the human recipient.

Abstract

We discuss the nature of chatbots as conversation partners in problem-solving conversations. What can chatbots do and what can't they do? Our analysis draws on insights from Aggregation Dynamics, Cognitive Linguistics, Neuropsychology and Psychology. We establish that chatbots are multifaceted and composite systems. Our argument focuses on basic chatbots in the hope of thereby making statements about the core functionality of more advanced chatbots. Basic chatbots are assumed to consist of a Large Language Model (LLM) with a simple interface. The main results of our analysis are: a description of human imagination, understanding and thinking based on so-called metaphorical problem propagations; that the texts in text datasets used for training LLMs have specific characteristics and that these texts only partially imitate human thinking and understanding; that the LLM training process encodes artificial metaphorical problem propagations into an LLM from these text datasets. Our conclusions are that a basic chatbot cannot be a thinking partner capable of matching the cognitive flexibility of humans, and that further development of LLMs will not lead to this either. But chatbots exist, they are being used on a massive scale, by both individuals and organisations. It is therefore socially and politically important to understand them. Our article aims to contribute to the discussion on the functioning, benefits and drawbacks of chatbots. Cognitive Linguistics shows how the use of metaphor is an expression of our thinking. Aggregation Dynamics, is an attempt at a comprehensive systems theory. We believe that the concept of metaphorical problem propagation could provide an interesting addition for both. Chatbots a solution? For what?

Sources

Related papers