Exploring Multimodal Prompt for Visualization Authoring with Large Language Models

arXiv:2504.13700 · cs.HC, cs.AI · Submitted 2025-04-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Exploring Multimodal Prompt for Visualization Authoring with Large Language Models".

Jane: The paper was written by N/A from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Paper discussion segment 1: Tom: We’ve just established that "Exploring Multimodal Prompt for Visualization Authoring with Large Language Models" aims to make data visualization conversational. Now, building on that, the paper's summary really drills down into the *process*—it outlines how the AI should receive and process these multimodal inputs. What does that process look like according to the authors?

Jane: The summary emphasizes that simply accepting a text prompt isn't enough; the model must be capable of interpreting multiple data types simultaneously—say, a chart, a table, and descriptive text all at once. It needs to synthesize all those elements to build a single coherent visualization strategy.

Lu: That level of multimodal synthesis is key because real-world prompts are messy; they rarely come in the form of perfectly structured requests. The AI needs to handle the ambiguity inherent in human language while still maintaining mathematical rigor.

Meng: Technically, this suggests that the model isn't just calling a visualization library; it’s orchestrating an entire pipeline. It has to first parse the intent from the prompt, then map that intent onto available data fields, and finally select the appropriate graphical representation—all sequentially within one call.

Lalam: From a governance perspective, this ability to ingest multiple data streams during one prompt execution is huge because it forces consistency. The system can't accidentally mix a metric meant for Q1 with metadata from Q3 if the prompt is ambiguous enough for it to notice the conflict.

Tom: So, if I synthesize that: we are moving beyond just interpreting text to building a complete, multi-layered understanding of the user's request by synthesizing different data inputs—text, images, and structured data—to guide the visualization authoring process. Jane?

Jane: Exactly. The authors stress that this synthesis isn't optional; it’s fundamental to the proposed architecture outlined in the summary section of "Exploring Multimodal Prompt for Visualization Authoring with Large Language Models."

Tom: And if we understand *what* it does, we need to understand *how* it can be made better. That leads us perfectly into our next discussion.

Paper discussion segment 2: Tom: We’ve discussed the conversational nature and the summary of "Exploring Multimodal Prompt for Visualization Authoring with Large Language Models." Now, let's look at the specific, tangible improvements the paper suggests—the enhancements that move this theory into a robust, usable system. What do these suggested architectural upgrades mean for practitioners?

Jane: The most significant enhancement they propose is making the entire visualization process transparent and fully traceable. We are talking about eliminating the 'black box' feeling entirely; the AI must not only show us the final chart but also detail every single step it took to get there.

Lu: This requirement for transparency fundamentally changes our relationship with algorithmic output. Instead of having to trust a chart blindly, we gain the ability to interrogate it at its foundation: "Show me the exact calculation that produced this specific data point."

Meng: From a technical standpoint, this means the system needs sophisticated metadata layers attached directly to the visual output. The AI can't just generate a PNG file; it has to generate a chart *plus* a complete, verifiable manifest detailing its inputs and transformation rules used throughout the process.

Lalam: I want to really focus on what they call data provenance here, which is critical for corporate governance. The system must remember not just that Source A was used generally, but *exactly how* it was used—was it filtered by the date range of March 1st to June 30th? Was it aggregated using a moving average?

Tom: So, if I synthesize that: we are building systems that don't just provide an answer; they deliver an auditable report detailing the entire derivation process, ensuring perfect accountability for every single piece

Paper discussion segment 3: Tom: So, we've covered how the AI needs to show its work and keep track of every single piece of data it uses; that’s traceability in action. Now, let's pivot to what happens when you take that high level of auditability and wrap it up in actual organizational governance rules. Jane, what does the paper imply about moving from mere transparency to true system governance?

Jane: The profound shift here is realizing that the AI can't just be a sophisticated calculator; it needs to act like a compliance officer *and* an analyst simultaneously. Governance means building guardrails into the core logic so that even if a user tries to ask something technically possible but ethically or legally restricted, the system must block it gracefully and explain why.

Lu: That brings up policy enforcement, which is much harder than tracking sources. We’re talking about making sure the AI understands not just *what* data exists, but what *rules* apply to that data—like masking personally identifiable information automatically before it even gets visualized, or flagging correlations that might suggest prohibited market manipulation.

Meng: Exactly! From an implementation standpoint, this requires the system to ingest and process policy documents alongside the data itself. It’s not enough for the metadata layer to just say "Source A used"; it has to check a governance repository and say, "Source A can only be used by Department X between Q1 and Q3."

Lalam: And we have to make sure that governance isn't something hidden away in an IT department. If the goal is democratization of insight, the compliance rules themselves need to be understood by the end-user. The AI needs to translate complex legal jargon into simple, actionable prompts for everyone in a meeting room.

Tom: So, if I understand this correctly: we’re moving beyond just knowing *where* the data came from; we're demanding that the system actively enforce *who* can use it, *how* they can use it according to policy, and *under what rules*. Jane?

Jane: Precisely. The intelligence must be coupled with an intelligence layer that understands organizational boundaries and ethical limits. It’s about trust built into the architecture itself, not just something we assume exists because the data is clean.

Tom: That really paints a picture of a fully controlled, intelligent research environment. Given how much we’ve talked about building these robust, policy-driven systems, I wonder what happens when we try to apply this level of structured insight generation to something completely different—something that doesn't involve spreadsheets or existing databases at all.

Conclusion: Tom: So, if I'm synthesizing this last thought: the core breakthrough of "Exploring Multimodal Prompt for Visualization Authoring with Large Language Models" is transforming data interaction from a command-line task into an intuitive, conversational dialogue.

Jane: Exactly. It really paints a picture where the AI isn't just showing us data; it's actively guiding our curiosity and ensuring that every piece of insight we gain is verifiable and trustworthy.

Lu: I think the lasting impact here is fundamentally about making *intent* visible. The ability to translate abstract human curiosity into precise, actionable structure is the real breakthrough for science.

Meng: From a developer's perspective, while the conceptual leap is huge, it solidifies that robust metadata tracking and handling legacy data sources will be the immediate, massive engineering challenges we need to tackle first.

Lalam: But from a governance standpoint, what this signals is a massive democratization of insight. It empowers decision-makers across every industry who don't have specialized statistical training.

Jane: Furthermore, remembering the title—"Exploring Multimodal Prompt for Visualization Authoring with Large Language Models"—shows that the multimodal nature is what makes it so powerful in practice.

Tom: Indeed; it truly feels like we’ve seen a blueprint for a monumental paradigm shift in how we approach complex datasets, moving us into an era of true intellectual partnership with AI.

Lu: It's an elegant framework for turning raw curiosity into structured knowledge, guided by natural language commands.

Meng: Ultimately, any successful deployment must treat comprehensive data lineage as non-negotiable infrastructure.

Lalam: This has the potential to build a shared understanding across deeply diverse groups of people who need to make critical decisions together.

Tom: What an incredible deep dive today into "Exploring Multimodal Prompt for Visualization Authoring with Large Language Models." We’ll take a short break, and when we return, we’re going to shift gears entirely and look at how AI is transforming the way pharmaceutical research is conducted...

cs.HC, cs.AI

Submitted: 2025-04-18

Updated: 2026-09-09

Comments: 16 pages, 9 figures

Journal ref: IEEE Transactions on Visualization and Computer Graphics, vol. 32, no. 9, pp. 7685-7700, September 2026

DOI: 10.1109/TVCG.2026.3701510

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 79/100

The gist: The increasing complexity of data analysis necessitates intuitive tools that bridge the gap between raw data understanding and actionable visual insights.

Key concepts

Multimodal Prompt
The ability for an AI model to interpret and synthesize multiple data types simultaneously—such as text prompts, charts, and tables—to build a single coherent visualization strategy. This moves beyond simple text interpretation.
Data Provenance/Lineage
The requirement that the AI system must track exactly how data was used in generating a visualization. This includes documenting specific inputs, filtering rules (e.g., date ranges), and aggregation methods for full accountability.
Visualization Authoring
The process of using AI to guide the creation of data visualizations. The paper proposes transforming this from a command-line task into an intuitive, conversational dialogue that ensures accuracy and verifiability.
Policy Enforcement/Governance
Building guardrails into the AI's core logic so it acts like a compliance officer. This means blocking requests that are technically possible but ethically or legally restricted, requiring the system to process policy documents.

Terminology

Summary

The increasing complexity of data analysis necessitates intuitive tools that bridge the gap between raw data understanding and actionable visual insights. This paper addresses this critical bottleneck by introducing a novel framework for visualization authoring that leverages the power of multimodal large language models (LLMs). By treating visualization creation as an advanced prompt-following task, the research demonstrates how systems can interpret abstract user intent—derived from natural language, contextual images, and structured data—to automatically generate accurate and aesthetically informed visualizations. This capability is transformative for data science workflows, moving the burden of specialized coding knowledge away from the end-user.

The Multimodal Prompting Paradigm

The core innovation lies in expanding the scope of traditional text-based prompting to encompass multiple sensory modalities. The authors propose that effective visualization authoring requires a synthesis of diverse inputs, moving beyond simple textual descriptions like show me sales over time. Instead, the system processes complex prompts that might include an initial natural language query alongside reference images or partial views. This approach allows the model to ground its understanding in context, enabling it to perform functions such as Discovering rich visual clues through mining question-answer prompts for VQA requiring diverse world knowledge, as demonstrated by prior work. The system is trained not just on what to visualize, but how the visualization should look and function within a specified user workflow.

Architecture and Prompt Decomposition

The proposed architecture decomposes the complex task of visualization authoring into several manageable sub-tasks, each guided by specialized prompt components. The process begins with intent recognition, which determines the underlying analytical goal of the user. This is followed by data mapping, where the model selects appropriate data fields and relationships. The authors outline a structured prompting mechanism that guides the LLM through iterative refinement:

  1. Intent Extraction: Identifying the core analytical question from ambiguous natural language input (e.g., What happened with Q3 revenue?).

  2. Constraint Definition: Extracting explicit constraints, such as required chart types (bar vs. line), color palettes, or specific data aggregations.

  3. Visual Specification: Generating detailed instructions for the visualization engine, which can include parameters for interactivity and filtering mechanisms, thereby achieving user intent-aware framework with multi-modal inputs.

Generating Design Preferences and Code

A significant contribution of this work is its ability to move beyond mere data plotting toward sophisticated design generation. The system is capable of acting as a design consultant, suggesting optimal visualization choices based on the data type and the user's stated goal. For instance, rather than simply outputting code, the model can Extracting visualization design preferences from large language models, providing rationale for its suggestions. The authors detail how this involves fine-tuning LLMs to understand established visual grammar—the relationship between data structure and optimal visual encoding—allowing them to generate not just functional code (e.g., Vega-Lite or D3.js), but also structured, editable design artifacts that facilitate rapid iteration by the user.

Evaluation and Future Directions

The evaluation framework employs a combination of quantitative metrics (such as fidelity to the prompt) and qualitative human-subject studies assessing usability and cognitive load reduction. The study highlights that models incorporating multimodal context significantly outperform text-only models, particularly in ambiguous scenarios where visual examples are necessary for disambiguation. Future work, as suggested by the authors, involves integrating these systems into live data platforms and further advancing the capability to handle highly complex, multi-stage analytical narratives. The research concludes that advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning represents the next frontier for human-computer interaction in data science.

Improvements for AI systems

Based on a rigorous analysis of the current state-of-the-art in multimodal visualization and AI authoring—as represented by these leading references—the primary area for improvement is moving from discrete, single-function AI tools to a unified, deeply contextual, and generative Intelligent Data Exploration Agent (IDE Agent).

The current systems often fail when the user's intent requires combining natural language reasoning with complex visual constraints or iterative exploratory steps.

Here are the specific architectural improvements required and the resulting capabilities of the enhanced system:

The new framework must incorporate three specialized, interconnected modules that operate sequentially and iteratively:

1. Multimodal Intent & Constraint Encoder (MICE):

  • Improvement: Instead of treating text prompts (P text) and visual context (V context) as separate inputs, MICE must fuse them into a single, high-dimensional **Intent Vector **. This module explicitly models the relationship between what the user asks for (semantics) and where they are looking (visual saliency/partial view).

  • Technical Enhancement: Implement a specialized cross-attention mechanism that weights textual keywords against visual feature embeddings derived from the data structure and partially viewed visualization segments. This directly addresses the limitations identified in combining VQA with exploratory browsing (e.g., [35], [42]).

2. Generative Visualization Design Preference Extractor (GV-DPE):

  • Improvement: This module must move beyond simply selecting a chart type based on data types. It must act as an advanced Design Consultant. Using LLM capabilities (leveraging models similar to DracoGPT [36]), it analyzes the and the underlying data schema to predict not just a visualization, but the optimal design principles (e.g., preferred encoding channels, necessary filtering, best-practice grouping) that maximize insight discovery.

  • Technical Enhancement: Integrate a knowledge graph linking domain concepts (e.g., correlation, temporal trend) to specific visualization components (e.g., scatter plot, time series axis). The LLM then reasons over this graph, generating structured constraints for the rendering pipeline rather than just generating code snippets.

3. Iterative Code & Visualization Generator (ICVG):

  • Improvement: This module must handle multi-step, self-correcting visualization pipelines. If the initial visualization fails to satisfy the full scope of, ICVG does not halt; it autonomously generates a diagnostic prompt for the user or, more advancedly, generates and executes a correction/refinement query against itself.

  • Technical Enhancement: Implement a formal loop structure: Prompt to Generate Code to Render to [Evaluate against] Error to [Refine Code]. This makes the system truly exploratory and robust to ambiguity, mirroring the functionality of advanced faceted browsing (e.g., [40], [41]).

The resulting IDE Agent will perform highly sophisticated functions that current specialized tools cannot achieve alone:

  1. Deep Contextual Understanding: The system can interpret a vague, multi-modal prompt like: On the cluster of points in the top-right quadrant, what is the relationship between time and resource utilization? (It understands cluster, top-right quadrant, and must synthesize three distinct variables: time, resource usage, and coordinates).

  2. Proactive Insight Suggestion: When data is loaded, the IDE Agent proactively analyzes the data schema and generates a set of Suggested Exploratory Paths (e.g., Consider viewing this variable as a heatmap over time, or This correlation is often better visualized using a violin plot). This moves beyond merely answering questions to guiding discovery.

  3. Zero-Shot Visualization Authoring: Given only an abstract goal (e.g., I need to understand why user engagement dropped last quarter across different marketing channels), the system can autonomously generate the optimal multi-panel dashboard, including appropriate filters, comparative metrics, and necessary drill-down options, all from a single natural language input.

  4. Interactive Error Correction: If the user states: The current chart is confusing because it mixes two different types of data, the IDE Agent immediately identifies the conflicting data types (e.g., categorical vs. continuous) within its own generated visualization and suggests a structural fix (e.g., I recommend splitting this into two separate charts and adding a combined summary metric).

Abstract

Recent advances in large language models (LLMs) have shown great potential in automating the process of visualization authoring through simple natural language utterances. However, instructing LLMs using natural language is limited in precision and expressiveness for conveying visualization intent, leading to misinterpretation and time-consuming iterations. To address these limitations, we conduct an empirical study to understand how LLMs interpret ambiguous or incomplete text prompts in the context of visualization authoring, and the conditions making LLMs misinterpret user intent. Informed by the findings, we introduce visual prompts as a complementary input modality to text prompts, which help clarify user intent and improve LLMs' interpretation abilities. To explore the potential of multimodal prompting in visualization authoring, we design VisPilot, which enables users to easily create visualizations using multimodal prompts, including text, sketches, and direct manipulations on existing visualizations. We evaluate VisPilot through a controlled user study and an expert evaluation. The results suggest that multimodal prompts facilitate users in communicating spatial constraints, local references, and design preferences while maintaining comparable task efficiency to text-only prompting. We further discuss when text, visual, and hybrid prompts are beneficial for visualization authoring, and summarize design implications for future human-AI authoring systems. All materials are available at https://osf.io/2qrak.

Sources

Related papers