NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation
cs.AI, cs.CL
Submitted: 2026-09-15
Updated: 2026-09-15
Code: https://github.com/NVIDIA-NeMo/DataDesigner
License: http://creativecommons.org/licenses/by/4.0/
The gist: We present NeMo Data Designer (NDD), an open-source, general-purpose framework for multi-modal synthetic data generation (SDG).
Terminology
Abstract
We present NeMo Data Designer (NDD), an open-source, general-purpose framework for multi-modal synthetic data generation (SDG). Designed to be intuitive to use, NDD provides a declarative configuration format in which human and/or agent users define each dataset column, with column types spanning text, code, structured outputs, images, embeddings, and statistical samplers that are explicitly configured to steer dataset diversity. Additional column types and functionality can be introduced using the framework's flexible plugin system. NDD's configuration is an inspectable artifact, supporting workflow sharing and reproducibility. SDG is an inherently iterative process. NDD therefore builds a preview-and-revision loop into its core workflow, allowing users to generate and inspect a small number of records, refine the specification, and rerun generation at full scale. At runtime, NDD resolves dependencies, schedules calls to user-provided model endpoints, and retries failed requests. We describe NDD's architecture and programming model and present case studies spanning structured, agentic, multimodal, and domain-specialized tasks, including datasets used in Nemotron model development and in production enterprise deployments.
Sources
- AlpaGasus: Training A Better Alpaca with Fewer Data
- AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations
- Strong Model Collapse
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
- Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
- Textbooks Are All You Need
- ProactBench: Beyond What The User Asked For
- Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
- PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency
- Kimi K2: Open Agentic Intelligence
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
- Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
- WebGPT: Browser-assisted question-answering with human feedback
- Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence
- Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
- Olmo 3
- Will we run out of data? Limits of LLM scaling based on human-generated data
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection