TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation

arXiv:2601.04202 · cs.CL, cs.AI, cs.LG · Submitted 2025-12-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation".

Jane: The paper was written by Anas Ezzakri, Nicola Piovesan, Mohamed Sana, Antonio De Domenico, Fadhel Ayed et al. from Huawei Technologies, Paris Research Center, Boulogne-Billancourt, France and Huawei Technologies, Shanghai Research Center, Shanghai, China.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title and Authors: Jane: The authors have identified a significant gap in LLM performance related to telecommunications standards, specifically focusing on the structure of the paper called TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation.

Tom: It seems that while AI is being used everywhere, this specific domain—the highly regulated telecom sector—is largely overlooked in many pretraining data sets.

Lu: When you look at the researchers who, they are trying to push the boundaries of understanding how these models handle data that isn't just natural language but complex technical layouts.

Meng: From an engineering standpoint, we often deal with 3GPP specifications, and if a benchmark like this is needed, it suggests that current deployment readiness in the telecom field is not yet achieved.

Lalam: This work elevates the status of the engineer's expertise; it forces us to acknowledge that simple text retrieval isn't enough when tables are fundamental to accurate system design.

Tom: It’s about ensuring that we aren't just looking at high-level concepts but at the actual, granular data inside those technical specifications, right?

Jane: Exactly. The paper is calling attention to the fact that these models fail when they lack a specific ability to interpret structured data within this specialized industry.

Lu: I think it shows how AI can be used not just as a quick answer machine but as a structural analyst for complex technical specifications.

Meng: And we need to see if these systems can actually integrate into our existing workflow, which requires understanding the format and structure of the documents they are reading.

Lalam: This pushes us toward a culture where AI is respected not just for its speed, but for its ability to grasp technical nuance within complex structured data.

Summary and Implications: Tom: Moving into the summary, TeleTables is built to test both the implicit knowledge and the explicit interpretation skills of LLMs in telecom.

Jane: The paper explains that it's not just about general knowledge; it’s about whether a model can accurately extract and reason over tabular data within those 3GPP documents.

Lu: I find the focus on implicit versus explicit ability fascinating because models are trying to do two things here: recall facts and perform complex data analysis.

Meng: The dataset consists of five hundred human-verified question-answer pairs, which is a lot of detail for a single benchmark, giving us concrete examples to work with.

Lalam: It's an important step toward improving the reliability of AI in critical infrastructure by demanding that the LLMs understand the data structure, not just surface-level facts.

Tom: The results show that smaller models under ten billion parameters struggle significantly with both recalling knowledge and interpreting those complex tables.

Jane: That reinforces a pattern; smaller models seem to have less exposure to this highly specialized content during their initial training phases.

Lu: And we can see that the larger, more powerful models, while showing better reasoning, are still having issues with the performance of these structured data tasks.

Meng: The engineering implication is clear: if we're building systems based on LLMs for specs like this, the size and type of the model are critical design factors.

Lalam: We must ensure our AI architecture is designed to handle not just text, but the structural integrity of technical information that defines our digital infrastructure.

Improvements and Methodology: Tom: So how did they build this impressive dataset? The methodology section really shows the ingenuity of the team in creating TeleTables.

Jane: They created a detailed four-stage pipeline, starting with extracting all those tables from thirteen distinct 3GPP technical specifications.

Lu: The use of multimodal LLMs in the refinement stage is where things get really creative—using Qwen2 point 5-VL-72B to transcribe non-textual elements like diagrams and formulas into text.

Meng: From an implementation perspective, I'm interested in how they convert the tables into multiple formats, including HTML, JSON, and also high-resolution PNG images to preserve the layout.

Lalam: This approach ensures that we aren't just getting a flat list of numbers but that the visual relationships and hierarchy embedded in the original documents are preserved by our AI.

Tom: The methodology really highlights that using different formats is key, which leads us to some very interesting findings about performance based on input formats.

Jane: The results indicate that HTML generally gives the strongest performance for most models, which is a big deal because of its ability to represent complex structures well.

Lu: It’s fascinating how the behavior differs; the way they are seeing better results in different formats suggests that each model has a different inductive bias based on its training data.

Meng: We need to decide if we prioritize efficiency, like using Markdown which is very compact, or accuracy, which means we might need that full HTML representation.

Lalam: This helps us refine our AI deployment strategy by choosing the format that best supports the structural understanding required for reliable engineering decisions.

Conclusion and Wrap-up: Tom: We’ve covered a lot of ground today, from the initial idea of TeleTables to how it’s built.

Jane: It's clear that this benchmark is helping us identify exactly where current LLMs fail when handling real-world technical data.

Lu: The insights here are so valuable because they show the precise gap between a model's knowledge and its capacity to navigate complex, structured information in a domain like telecom.

Meng: I think the most important takeaway is that we can't just rely on general-purpose AI for these critical tasks; specialization is necessary.

Lalam: The ultimate goal is to build an AI that understands the cultural context of technical standards, not just a statistical correlation between words and data points.

Tom: We have to emphasize that the entire process of TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation shows that we are on the cusp of a major shift.

Jane: It’s a clear indication that general training isn' insufficient, and we' are moving towards models that can reliably interpret and reason over complex technical standards.

Lu: I think this opens up huge possibilities for automating high-level engineering tasks in ways we haven't even started to imagine yet.

Meng: We need to start integrating these findings into the practical design of AI agents that interact with 3GPP documents, ensuring they are robust enough to handle the complexity.

Lalam: This technology promises a future where our digital infrastructure is managed by systems that truly understand its foundational data and structure.

Huawei Technologies, Paris Research Center, Boulogne-Billancourt, France · Huawei Technologies, Shanghai Research Center, Shanghai, China

cs.CL, cs.AI, cs.LG

Submitted: 2025-12-05

Updated: 2026-09-04

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The paper introduces TeleTables, a novel benchmark designed to rigorously evaluate Large Language Models (LLMs) in their ability to interpret and reason over complex tabular data found within

Key concepts

TeleTables: A benchmark designed to test LLMs in interpreting complex technical layouts found in telecommunications standards. It evaluates a model's ability to both recall facts and perform complex data analysis on structured information.
3GPP Specifications: Technical documents used in the telecom industry that are highly regulated. The paper focuses on extracting and reasoning over the tabular data within these specific, complex technical specifications.
Structured Data Interpretation: The ability to accurately extract and understand information presented in a non-natural format, such as tables or diagrams. This goes beyond simple text retrieval and is crucial for accurate system design.

Terminology

Benchmark — 29× in this paper · explained in 2 episodes
Accuracy — 16× in this paper · explained in 1 episode
3GPP Specifications — 7× in this paper · explained in 1 episode
Fine-Tuning — 4× in this paper · explained in 5 episodes
Large Language Models (LLMs) — 4× in this paper · explained in 6 episodes
RAG — 4× in this paper · explained in 3 episodes
Aggregation — 3× in this paper · explained in 1 episode
Alignment — 2× in this paper · explained in 1 episode
Artificial Intelligence — 2× in this paper
Fidelity — 2× in this paper · explained in 2 episodes
AI Agents — 1× in this paper · explained in 2 episodes
Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 3 episodes
Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Human-in-the-Loop — 1× in this paper · explained in 3 episodes
Positional Bias — 1× in this paper · explained in 2 episodes
Retrieval Augmented Generation — 1× in this paper
Retrieval Augmented Generation (RAG) — 1× in this paper · explained in 4 episodes
Structural Integrity — 1× in this paper · explained in 3 episodes
Structured Reasoning — 1× in this paper · explained in 3 episodes

Summary

The paper introduces TeleTables, a novel benchmark designed to rigorously evaluate Large Language Models (LLMs) in their ability to interpret and reason over complex tabular data found within telecommunication standards. The research addresses a significant gap in current AI capabilities, noting that LLMs perform poorly on telecom standards, particularly those published by the Third Generation Partnership Project (3GPP). By providing a framework to assess both the implicit knowledge and the explicit interpretation ability of these models, TeleTables aims to highlight why domain-specialized fine-tuning is necessary for reliable LLM performance in this technical sector.

The Problem with Current LLMs

The complexity inherent in 3GPP specifications arises from their extensive use of tables, which encode crucial information such as configuration parameters and operational procedures. Existing benchmarks, while useful for general table reasoning, primarily focus on text-based representations and fail to examine the specific challenges posed by the telecom domain. This study confirms that LLMs struggle with these dense, intricate tables. The evaluation reveals that smaller models (under 10B parameters) struggle both to recall 3GPP knowledge and to interpret tables, indicating a lack of exposure in their pretraining data and insufficient inductive biases for navigating complex technical material.

Data Extraction and Refinement Pipeline

The foundation of TeleTables is built upon a curated collection of 13 distinct 3GPP technical specifications, covering various series from Release 18 and 19. A total of 2,220 tables were systematically extracted using a four-stage pipeline:

Table Localization: Identifying elements preceded by standard table titles.
Table Extraction: Converting the localized tables into multiple formats, including HTML, JSON, and Markdown.
Metadata Collection: Enriching the data with contextual information from the the source document pages.
LLM-based Refinement: Utilizing a multimodal LLM (Qwen2.5-VL-72B) to transcribe non-textual elements (like formulas) and verify/correct structure against the image fidelity.

Multistage MCQ Generation Process

To ensure the benchmark captures both simple retrieval and complex logic, a two-stage pipeline is employed:

Basic MCQ Generation: A generator agent, utilizing Qwen2.5-VL-72B, produces a large volume of basic questions based on the all four table representations (Image, HTML, JSON, and Markdown). These are validated by a second agent to ensure correctness.
Difficult MCQ Generation: The process is escalated by feeding the generator five randomly selected basic MCQs alongside all table formats. It then synthesizes concepts from these simpler questions to construct more complex MCQs that require multi-step reasoning. This difficult set is filtered using a simplicity filter and subjected to human validation.

Evaluation Methodology and Input Formats

The benchmark assesses LLM performance in two critical settings: without access to the associated tables (relying only on textual reference) and with perfect retrieval (the table provided alongside the MCQ). The study found that the choice of representation format significantly impacts accuracy:

HTML: Generally yields the strongest performance for most models.
Markdown/JSON: These formats are preferred by certain architectures, such as Llama models, due to their lightweight nature.
Image Representations: Performance drops substantially across all visual models when compared to text-based formats.

Key Performance Findings

The results demonstrate a clear hierarchy of performance based on model capability:

Non-Reasoning Models: Showed limited ability, with smaller models failing to exceed 75% in cons@16.
Multimodal Models: Demonstrated superior capacity for structure interpretation, benefiting from cross-modal pretraining.
Reasoning Models: Consistently achieved the highest performance across all tested categories, maintaining high accuracy even on questions that require multi-step relational or arithmetic reasoning. The best-performing model (Qwen3-32B) achieved 91.18% pass@1 and 92.60% cons@16 overall. Furthermore, the analysis of table complexity shows a consistent decline in performance as the number of tokens in HTML representations increases, confirming that processing large or hierarchically structured tables remains a significant challenge for LLMs.

Improvements for AI systems

Based on a rigorous analysis of the findings in the paper TeleTables, I have identified several critical, actionable improvements necessary to elevate current AI systems from mere knowledge recall mechanisms to reliable technical interpreters.

Improvement: Implement a heterogeneous, multi-format input strategy for all structured data presented to the LLM. This involves simultaneously presenting the table in multiple encodings (e.g., HTML, JSON, and Markdown) within the context window, rather than relying on a single representation.

What the Improved AI System Can Do:

Maximize Robustness: The system overcomes format-specific biases (e.g favoring HTML or Markdown) by leveraging complementary structural cues inherent in each encoding.
Enhance Accuracy: It achieves higher consistency, particularly for complex tables, as different representations allow the model to verify information across multiple structural frameworks (e.g, JSON's clear data hierarchy vs. HTML's explicit markup).