WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models

Contrastive vision-language models have achieved remarkable progress through largescale pretraining, but this work investigates whether global pretraining alone is sufficient for culture-specific understanding or if further adaptation with natively sourced data can boost performance. The study introduces WAON, a massive native Japanese image-text dataset, and demonstrates that fine-tuning on this data consistently achieves stronger performance on Japanese cultural benchmarks than fine-tuning on English-to-Japanese translated data.

The gist

Fine-tuning on WAON consistently achieves the best performance across all baselines when evaluating models on Japanese cultural benchmarks, while fine-tuning on ReLAION (en→ja translation) consistently degrades Japanese cultural understanding.

Dataset Construction and Scale

The paper introduces WAON, described as the largest publicly available native Japanese image-text dataset constructed from native Japanese web content in Common Crawl, containing approximately 155 million examples. The construction pipeline involves several cascaded steps to ensure high quality, including:

1. Extracting Japanese HTML documents from WARC files.

2. Performing text-based deduplication before downloading images, followed by image-based filtering and deduplication (using pHash).

3. Applying image quality filtering, such as discarding images whose width or height is below 150 pixels or whose aspect ratio falls outside [0.5, 2.0], and applying NSFW classification to remove unsafe content.

4. Filtering misaligned pairs using cosine similarity between image and text embeddings computed by SigLIP2-base-patch16-256, setting a threshold of 0.1 for semantic alignment filtering.

The final result after cross-snapshot deduplication yields approximately 155M high-quality image-text pairs.

Benchmark Development

To rigorously test the dataset's utility, the authors introduce WAONBench, a manually curated Japanese cultural benchmark designed to address limitations in prior resources. This benchmark comprises 374 classes spanning eight diverse categories: animal, building, event, everyday, food, nature, scenery, and tradition. It contains 161 classes and 7,654 examples in the Recruit dataset (Honda and Arai, 2024), which was limited to only food categories. WAONBench significantly expands this coverage by providing more than twice as many cultural concepts with 5 images per class for a total of 1,870 images. The visual diversity of WAONBench is quantified, achieving an average pairwise cosine distance of 0.495 compared to the Recruit dataset's 0.425.

Experimental Setup and Results

The experiments fine-tune the siglip2-base-patch16-256 model on three datasets: WAON (155M pairs), ReLAION (ja subset, 120M examples), and ReLAION (en→ja translation, 1,452M pairs). Evaluation is conducted across four benchmarks: WAONBench, Recruit, ImageNet (Western culture), and XM3600. The results show that fine-tuning on WAON consistently achieve[s] the best performance across all baselines. Specifically, WAON achieves a top-1 accuracy of 95.1% on WAONBench and 83.3% on Recruit, outperforming both baseline models and those fine-tuned on ReLAION variants. Conversely, fine-tuning on ReLAION (en→ja translation) consistently degrades Japanese cultural understanding, with performance dropping from 87.8% to 76.8% on WAONBench.

Conclusion and Implications

The study concludes that data origin is a critical factor for cultural adaptation in contrastive vision-language models. The findings suggest that while global pretraining provides a strong foundation, native Japanese data is crucial for culture-specific fine-tuning, as translation alone cannot compensate for the absence of natively sourced data. WAON serves as the largest publicly available natively sourced Japanese image-text dataset, and WAONBench provides a comprehensive benchmark to validate its effectiveness. The authors release WAON and code under the Apache 2.0 License, emphasizing ethical considerations regarding web data usage and content filtering.

Limitations

The study acknowledges several limitations, including taking Japanese as a single case study, fine-tuning only one model architecture (siglip2-base-patch16-256), and lacking a quantitative analysis of each pipeline step's contribution to dataset quality. Ethical considerations address the use of web data from Common Crawl, confirming that no image files are redistributed, and outlining mechanisms for reporting problematic samples. The authors also note that some harmful or biased content may remain despite filtering.

Acknowledgements

The research utilized the “mdx: a platform for building data-empowered society” and ABCI 3.

Here are specific improvements for AI systems based on the findings presented in the paper:

1. Improve cross-cultural performance in vision-language models (VLMs) by fine-tuning them with natively sourced, culturally specific data (like WAON) rather than relying solely on globally pretraining or translated data. This allows models to achieve state-of-the-art accuracy on culture-specific benchmarks (e.g., achieving 95.1% top-1 accuracy on WAONBench).

2. Develop a robust, high-quality dataset construction pipeline for low-resource languages by applying cascaded filtering and deduplication steps across multiple modalities (text, image quality, semantic alignment) to filter out noisy web data effectively.

3. Enhance the diversity and robustness of cultural understanding in models by creating manually curated benchmarks that significantly increase class coverage beyond existing narrow datasets (like WAONBench expanding from 161 classes to 374 classes across 8 diverse categories).

4. Improve visual diversity in fine-tuning data for culturally specific tasks by curating benchmark images to ensure wider dispersal in the embedding space, leading to more robust and less category-biased model representations (WAONBench achieving a higher average pairwise cosine distance than the Recruit dataset).

5. Build specialized Japanese vision-language models that exhibit superior cultural adaptation capabilities by leveraging large-scale global pretraining backbones (like SigLIP2) followed by fine-tuning on natively sourced data, rather than just relying on general multilingual models or translated data.

The improved AI systems can:

1. Accurately classify images into nuanced Japanese cultural categories (e.g., distinguishing between specific traditional festivals, regional scenery, and everyday objects) with high precision, even when tested on unseen Japanese cultural benchmarks.

2. Perform better at zero-shot classification tasks involving Japanese cultural concepts because the training data reflects native visual and textual contexts rather than biased translations or Western imagery.

3. Be more resilient to English-to-Japanese translation artifacts, as their performance will not degrade significantly when fine-tuned on the high-quality WAON dataset, unlike models fine-tuned on translated data.

4. Generate more diverse and contextually accurate visual representations of Japanese culture by being trained on a benchmark that covers a broad spectrum of cultural domains (animal, building, event, food, etc.), leading to better generalization across different aspects of Japanese identity.

Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing English-only caption filters and pretraining on global data is effective for improving multicultural performance. We study whether such global pretraining is sufficient for culture-specific understanding, or whether further adaptation with natively sourced data can boost performance beyond what global pretraining alone achieves. To enable this investigation, we present WAON, the largest publicly available native Japanese image-text dataset constructed from native Japanese web content in Common Crawl, containing approximately 155 million examples. We also introduce WAON-Bench, a manually curated Japanese cultural benchmark spanning 374 classes. Through comparative fine-tuning experiments on multiple Japanese image-text datasets, we observe that models fine-tuned on WAON consistently achieve stronger performance on Japanese cultural benchmarks than those fine-tuned on English-to-Japanese translated data. Controlled experiments at matched scale, filtering, and training budget across two model families further indicate that native web origin is the primary driver of this gain. We release our dataset, benchmark, model, and code.