Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

summary

Video file (mp4)

The gist

The gist Vision2Code introduces a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation, which matters because it provides a reproducible testbed for

In short

Vision2Code introduces a reference-code-free benchmark to test how well vision-language models can write executable code to recreate structured images across diverse domains like charts, geometry, and 3D scenes. The evaluation uses dataset-specific rubrics applied after code execution, showing that this method yields scores highly correlated with human judgments compared to generic metrics.

Key concepts

Vision2Code Benchmark
This is a comprehensive testbed containing 2,169 examples from 15 datasets covering six visual domains: charts/plots, geometry, graphs, scientific imagery, documents, and 3D scenes. It allows researchers to measure image-to-code generation performance without needing paired source code for evaluation.
Dataset-Specific Rubrics
Instead of generic scoring rules, these rubrics are tailored to each dataset's requirements. For example, a chart rubric might heavily weight 'Numeric information' and 'Structure and layout,' while a spatial scene rubric focuses on 'Depth ordering.' These tailored categories enforce strict penalties for domain-specific failures.
Rendered-Code Evaluation
The model generates code, which is then actually executed by the system (defaulting to Python/Matplotlib). The resulting rendered image is then compared against the original source image using the dataset-specific rubric. This process tests whether the generated code successfully produces a visually accurate output.
Correlation with Human Ratings
The paper demonstrates that scores derived from this rigorous, domain-specific evaluation method correlate much more strongly with human ratings than generic metrics like simple pixel similarity or embedding-based scores. This validates the framework as a reliable way to assess VLM performance.

Terminology used across episodes

This episode discusses

The paper

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation · Read on arXiv

Ajay Vikram Periasami, Junlin Wang, Bhuwan Dhingra

Duke University

We study whether vision-language models (VLMs) can reconstruct structured, graphics-based images by writing executable code to generate them. Existing benchmarks on image-to-code generation are generally limited in the domains they cover, rely on reference source code for evaluation, and do not provide metrics to quantify domain-specific issues. To address this, we introduce Vision2Code, a multi-domain, reference-code-free benchmark containing 539 test examples from 15 different datasets, spanning charts and plots, geometry, graphs, scientific diagrams, documents and 3D spatial scenes. Evaluation is done by executing the generated programs and comparing their rendered outputs to the source images using dataset- and domain-specific rubrics for a VLM judge. We find that the final scores obtained in this manner have substantially higher correlation with human ratings than those obtained via generic VLM rubrics or standard pixel-based, embedding-based or perceptual similarity metrics. We further demonstrate the reliability of our evaluation under repeated sampling from the VLM judges, varying rubric weights, and in the presence of controlled errors. Our results highlight the variability across domains of existing image-to-code generation models, with accuracy on charts and graphs being significantly higher than other domains. Lastly, we demonstrate the utility of our scoring method for rejection sampling of generated programs for fine-tuning, improving Qwen3.5-9B by 16% and Phi-4-MM by 28% compared to their base checkpoints. Our code and data are publicly available at https://image2code.github.io/vision2code/.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation".

Jane: The gist Vision2Code introduces a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation, which matters because it provides a reproducible testbed for measuring, diagnosing,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, wrapping up this discussion on Vision2Code, Tom, what's the main message we should take away about this work?

Tom: The main thing is that by moving to a multi-domain benchmark with dataset-specific rubrics, they are creating a much more accurate way to measure how well vision language models can reconstruct structure into executable code.

Lu: It moves us away from the narrow visual scopes of older benchmarks and tackles things like dense text layout in documents or precise depth ordering in three dee scenes simultaneously <ref:2605.11307#pg1>.

Meng: For practical application, having this evaluation framework means we have a way to systematically diagnose where an AI is failing when it tries to turn a picture into functioning code.

Tom: And the authors suggest that this approach is more aligned with human judgment because it focuses on the specific visual information that must be preserved for each task.

Jane: It really shows that relying on generic similarity metrics doesn't capture the nuanced semantic requirements of structured image reconstruction.

Lu: They are testing how models handle things like label-to-object binding in geometry or edge directionality in graphs, which are specific structural challenges.

Tom: So, Vision2Code gives us a way to push these models toward better understanding the underlying structure rather than just generating a plausible-looking image.

Jane: It's an important step for making sure that when we evaluate these vision language models, we're measuring what actually matters in terms of code generation quality.

Tom: That’s it for this look at Vision2Code: a multi-domain benchmark for evaluating image-to-code generation.

Conclusion: Tom: So we’ve seen how Vision2Code is setting up this whole new system for testing image-to-code models that you can actually trust to work across different visual tasks.

Jane: It really boils down to them creating a benchmark where you don't need some pre-written code as a reference anymore, right?

Lu: Exactly. They’re looking at fifteen totally different types of images—everything from charts and graphs to complex three dee scenes—and they want to see if the model can turn those visuals into working programs.

Meng: The point is that these old benchmarks were too narrow, so this new one tries to prove the AI can do it in many places, not just one or two specific areas.

Tom: And what they found is that when you use these specific rubrics for each type of image, the final scores actually match human ratings way better than just looking at how similar the generated code looks to the original picture.

Jane: That’s a big deal because it tells us that just having a visually okay output isn't enough; it has to be structurally and semantically correct for what you asked of it.

Lu: They even showed that for things like charts and graphs, the model is significantly better than in other domains when they use this detailed scoring method.

Meng: From an engineering side, knowing where the weaknesses are across all these domains helps us know exactly what we need to train next to make those models more reliable.

Tom: It makes you wonder what this means for how we actually use AI to build things that rely on visual understanding.

Jane: It suggests that for any complex visual task, you need a way to measure the specific details being reconstructed, not just a general similarity score.

Lu: So it’s not just about making pretty pictures anymore; it’s about ensuring the underlying data structure is preserved correctly when generating code. **(Sound of traffic fades out)**

More episodes

← Home