Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

arXiv:2606.15932 · cs.CL · Submitted 2026-06-14 · Read on arXiv

cs.CL

Submitted: 2026-06-14

Updated: 2026-08-30

Comments: Accepted by TMLR 2026

Code: https://github.com/lukas-blecher/LaTeX-OCR

License: http://creativecommons.org/licenses/by/4.0/

The gist: While Large Language Models (LLMs) have substantially advanced text-to-code generation, many real programming tasks specify intent through visual artifacts such as screenshots, charts, and videos.

Terminology

Abstract

While Large Language Models (LLMs) have substantially advanced text-to-code generation, many real programming tasks specify intent through visual artifacts such as screenshots, charts, and videos. These tasks require models to connect visual perception to executable programs, as correctness depends not only on syntax but also on layout, data semantics, and domain-specific constraints that apply after execution. This survey reviews Multimodal Code Intelligence, covering systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable structure, an intermediate reasoning trace, or an executable tool interface. Then we organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects artifact-generation problems to agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Across the literature, we argue that reliable evaluation requires evidence about semantics and interaction beyond visual fidelity. Looking ahead, future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move this field from single-output imitation toward evidence-grounded executable systems. An ongoing project and resources are available on GitHub.

Sources

Related papers