BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models

summary

Video file (mp4)

The gist

The gist The authors introduce BEAR-Bench, a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents.

In short

BEAR-Bench is a complex, bilingual benchmark with 1000 questions testing multimodal models' reasoning over text-heavy business and scientific documents in English and Russian. It evaluates how well models can perform multi-step logical reasoning tasks across different document types, such as interpreting charts or formulas from financial reports and academic papers.

Key concepts

BEAR-Bench
A self-contained benchmark consisting of 1000 human-annotated questions based on text from business and scientific documents in English and Russian. It tests models' ability to perform complex, multi-step reasoning tasks rather than simple information extraction.
Domain Taxonomy
The benchmark is divided into two main domains: Business (financial reports, presentations) and Science (mathematical formulae, scientific plots). These are further broken down into specific sub-categories like tabular reasoning or symbol recognition to test diverse model capabilities.
Human Annotation Pipeline
Domain experts created 1000 question-answer pairs. Annotators were required to create multi-step questions requiring compositional reasoning and detailed answers, ensuring the benchmark tests deep understanding rather than surface-level retrieval.
Hallucination Detection Methods
Various methods—including token uncertainty scores, representation detectors, supervised hidden-state probes (SUQ), and LLM-as-a-judge—were compared to identify failures. The study found that SUQ achieved the highest balanced accuracy across proprietary models for detecting errors.

Terminology used across episodes

This episode discusses

The paper

BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models · Read on arXiv

Liubov Chubarova, Alexandra Kuleshova, *Daniil Volkov*, *Kirill Sultanov*, Alexey Zaytsev

Yandex Applied AI Institute

While Multimodal Large Language Models (MLLMs) have made significant strides in visual comprehension, their ability to reason about text-dense, professional documents remains incompletely evaluated. Existing benchmarks emphasize information extraction, require external domain knowledge, or cover professional documents only as one of many settings. They are also largely English- or Chinese-centric, leaving other languages and Russian, in particular, substantially underrepresented. To address these limitations, we introduce BEAR-Bench (Bilingual Enterprise and Academic Reasoning), a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents. We evaluate 16 proprietary and open-weight MLLMs, including Gemini 3.1 Pro and Qwen3.5-397B, on BEAR-Bench and observe clear headroom even for the strongest systems. Finally, we use the resulting model outputs to compare existing hallucination detection methods, evaluating not only how often models fail on BEAR-Bench but also how reliably those failures can be identified.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models".

Jane: The gist The authors introduce BEAR-Bench, a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We're looking at the title again, "BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models." It really tells you that the focus is on reasoning skills within professional contexts, not just simple vision tasks.

Jane: Exactly. The benchmark is designed to test how multimodal models perform when they have to understand complex business charts or scientific figures in both languages without needing any external help.

Lu: They are specifically evaluating document-grounded reasoning rather than just factual recall or shallow extraction, which is a key distinction for testing the actual intelligence of the model.

Meng: I wonder if that makes it easier to use for real deployment scenarios, because if we can't reason across these documents accurately, then deploying an AI assistant for a company’s reports is risky.

Lalam: The paper contributes by proposing this bilingual benchmark and then using it to compare different methods for detecting when models actually make mistakes when reasoning over professional text.

The paper's summary: Tom: So the core of BEAR-Bench is that they created this self-contained, complex set of questions targeting text-dense enterprise and scientific documents in both English and Russian.

Jane: It covers a lot of ground, testing things like interpreting academic figures, transcribing mathematical formulas, and analyzing data from plots in the science section.

Lu: In the business section, they have tasks like crossreferencing data in financial reports or doing logical reasoning over business charts to understand ownership structures.

Meng: The researchers used a pretty detailed collection process; for instance, they pulled English documents from the SEC system and Russian ones from platforms like e-disclosure.ru and moex.com to make sure it covered real-world complexity.

Lalam: They also have a very specific annotation pipeline involving thirteen domain experts who wrote exactly one multi-step, multi-hop question for every image they annotated.

The paper's improvements: Tom: One of the main things they suggest is using this benchmark not just to test accuracy, but also to check if failures can be detected reliably by different methods.

Jane: That’s a huge point, because deploying AI in professional settings requires knowing not only how often it fails but whether those failures can be detected reliably.

Lu: They address this gap by comparing token-level uncertainty scores, representation-based detectors, supervised hidden-state probes, and MLLM-as-a-judge methods across both proprietary and openweight models.

Meng: That comparison is really practical for us because it gives us a way to see which detection method works best when we're dealing with different access levels—like having direct access to internal model signals versus using proxy methods for proprietary ones.

Lalam: They found that the supervised hidden-state probe achieved the highest balanced accuracy on all eight proprietary models, with a score between zero point six seven and zero point seven four, while the VLM-as-judge baseline got a high balanced accuracy of zero.

Conclusion: Tom: So to wrap up, BEAR-Bench provides a bilingual test for multimodal reasoning in professional settings, and they show that it’s important to know how to spot model failures when dealing with dense text.

Jane: The implications are that if we want models to be useful in complex fields like finance or science, we need benchmarks like this that aren't just testing simple extraction but deep comprehension.

Lu: The work also highlights the persistent linguistic gap; even the strongest proprietary models show a drop in accuracy when moving from English to Russian, which confirms that language barriers still exist for these systems.

Meng: From an engineering standpoint, it shows us where we need to focus our improvements—specifically targeting spatial misgrounding and counting errors, because those seem to be the biggest stumbling blocks right now.

Lalam: Overall, this BEAR-Bench is a solid resource because it gives us a clear framework for evaluating reasoning over text in these challenging domains, and it’s going to help us build more robust systems.

More episodes

← Home