OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources".

Tom: The OARelatedWork dataset introduces a large-scale resource for generating entire related work sections from open access sources,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So let's talk about who did this, Martin Docekal, Martin Fajcik, and Pavel Smrz from Brno University of Technology. They are the authors of OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources.

Jane: It’s interesting because their whole goal is to move beyond just generating parts of literature reviews based on abstracts and toward generating entire related work sections using all the available content.

Lu: They are specifically aiming to shift the field away from just summarizing parts of related work sections from abstracts only, and instead focusing on creating whole sections from everything they have.

Meng: But they also set up this dataset so that every cited paper is available in full text for controlled evaluation, which is crucial for testing long-form generation properly.

Lalam: It means the context these models get to work with is much richer than what we usually see, because they aren't just looking at snippets.

The paper's summary: Tom: Now let’s look at what they actually did in the paper. They created this OARelatedWork collection which contains ninety-four thousand four hundred fifty papers and five million eight hundred twenty-four thousand six hundred eighty-nine unique cited papers from multiple domains.

Jane: The core of the task is generating a whole related work section, denoted as Y equals (y1, y2, y3... yL), where L is the length of that section.

Lu: They give the system three main inputs: first, a combination X which includes abstracts and full texts for both the target paper and all its cited papers.

Meng: Second input is the target length L we want to generate, and third is information about whether we should follow a structured format with subsections or a plain unstructured form.

Lalam: They also tried different versions of this task during generation, specifically testing using abstracts instead of full texts to see how performance changes under those different conditions.

Tom: And they are evaluating this by using statement-level judges which is a way to check factual accuracy on a very fine scale rather than just looking at whether the whole section looks right.

The paper's improvements: Jane: One of the big things they point out is that they evaluate diverse baselines and conduct a human evaluation, which revealed a tension between abstractive generation and factual faithfulness in this specific context.

Lu: They demonstrated that advanced models can actually surpass human writers when it comes to strict evidence-grounded factuality when the evaluation setup is correct.

Meng: But they also found something interesting: under statement-level judging, a model like GPT4o-mini’s evidence-grounded True rate drops from ninety two point nine percent when using just abstracts down to eighty three point eight percent when using full texts.

Lalam: That drop is significant because it shows that providing the full text context really changes how accurate the generation becomes in these complex tasks. They also showed that their fine-grained statement-level faithfulness evaluation showed a kappa of zero point six one six nine, which is higher than the zero point four three zero one they saw for coarser document-level faithfulness.

Tom: So they are establishing this robust framework with structured data and fine-grained factual annotations to evaluate this kind of long-form generation using OARelatedWork, the dataset for related work sections with full texts from open access sources.

Conclusion: Jane: So wrapping up on OARelatedWork, the main idea is that advanced LLMs can actually beat human writers on strict, evidence-grounded factuality when they have the necessary context.

Lu: They pointed out a tension between abstractive generation and factual faithfulness, which shows that models can get pretty good at sticking to the source material if you set up the evaluation correctly.

Meng: Even with all those full texts, under statement-level judging, a model like GPT4o-mini’s evidence-grounded True rate drops from ninety two point nine percent when using just abstracts down to eighty three point eight percent when using full texts.

Lalam: Plus, their fine-grained statement-level faithfulness evaluation showed a kappa of zero point six one six nine, which is higher than the zero point four three zero one they saw for coarser document-level faithfulness.

Tom: So they’re establishing this robust framework with structured data and fine-grained factual annotations to evaluate this kind of long-form generation, which is what we're talking about with OARelatedWork, the dataset for related work sections with full texts from open access sources.

Jane: It really shows that providing the right context and evaluation tools allows these models to perform better in generating complex literature reviews than we thought possible before.

Lu: This paper sets a good foundation for future research into how large models handle massive, structured contexts like those found in academic related work sections.

Meng: We’ll need to keep watching how these statement-level evaluations evolve because that’s where the real testing for factual accuracy in these long outputs is happening right now.

Lalam: It moves the field toward a more realistic paradigm of generating full sections by using comprehensive full-text contexts, which is what makes this dataset so significant.

Tom: It’s interesting how this kind of structured data collection lets us see exactly where the current limits are for synthesizing massive literature reviews.

Jane: And it really helps us understand that context matters a lot when we’re talking about deep synthesis versus just summarization.