The GELATO Dataset for Legislative NER

summary

Video file (mp4)

The gist

The gist The GELATO dataset introduces a two-level named entity recognition ontology for U.S.

In short

The GELATO dataset introduces a two-level named entity recognition ontology for U.S. congressional legislation from the 118th Congress, consisting of six top-level classes and thirty subclasses. Researchers fine-tuned transformer models like RoBERTa for first-level prediction, followed by using LLMs to complete the second level. The results show RoBERTa's superior performance over BERT for this task.

Key concepts

GELATO Ontology
This is a specialized, two-level tagging system created specifically for U.S. legislative texts. It has six main top-level categories (like Person or Act) and thirty specific subcategories tailored to how entities appear in congressional bills, making it precise for legislative research.
Two-Level NER
This is a two-step process used to label text entities. First, models predict the broad category (Level 1), and then a separate method, like an LLM prompt, refines that prediction into a more specific category (Level 2). This helps capture complex relationships within legal documents.
Transformer Models
These are advanced deep learning architectures (like BERT and RoBERTa) used to analyze text. They are fine-tuned on the GELATO data to accurately predict the first level of entity tags, demonstrating their capability in understanding legislative language.
LLMs as Second-Level Predictors
Large Language Models (like Qwen) are used after the initial model prediction to make more detailed, granular classifications. This combination tests whether general knowledge from LLMs can effectively refine the specific predictions made by smaller transformer models.

Terminology used across episodes

This episode discusses

The paper

The GELATO Dataset for Legislative NER · Read on arXiv

Matthew Flynn, Timothy Obiso, Sam Newman

Brandeis University

This paper introduces GELATO (Government, Executive, Legislative, and Treaty Ontology), a dataset of U.S. House and Senate bills from the 118th Congress annotated using a novel two-level named entity recognition ontology designed for U.S. legislative texts. We fine-tune transformer-based models (BERT, RoBERTa) of different architectures and sizes on this dataset for first-level prediction. We then use LLMs with optimized prompts to complete the second level prediction. The strong performance of RoBERTa and relatively weak performance of BERT models, as well as the application of LLMs as second-level predictors, support future research in legislative NER or downstream tasks using these model combinations as extraction tools.

DOI: 10.63317/3axxkz9oh5th

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The GELATO Dataset for Legislative NER".

Jane: Here's the gist of the paper. The gist The GELATO dataset introduces a two-level named entity recognition ontology for U.S.

Tom: congressional legislation, which is important for supporting various types of research and more advanced NLP methods and applications How it works The paper introduces GELATO, a two-level NER ontology designed for U.S.

Jane: congressional legislation This ontology consists of six top-level classes and 30 subclasses tailored to U.S.

Tom: legislative text The top level has six tags: Person, Organization, Document, Act, Abstraction, and Class The two levels of labels for each document and mention in GELATO were completed in two steps with three total annotators The first level of GELATO annotation for each document was completed by two annotators using Label Studio All three annotators participated in the adjudication process, and all tokens with differing tags were discussed by the group of annotators and resolved through group consensus To obtain the second-level GELATO labels, the three annotators reviewed the gold labels together For each gold label mention, the annotators decided which second-level label of the respective first-level label was most appropriate, and tagged it accordingly Dataset Details The dataset consists of 131 U.S.

Paper summary: Tom: So to recap this segment, we're talking about "The GELATO Dataset for Legislative NER". The core idea is that they introduced a dataset of U.S. House and Senate bills from the 118th Congress annotated with a new two-level named entity recognition ontology designed for U <ref:2603.14130#pg1,House and Senate bills from the 118th Congress annotated>.S. legislative texts >

Jane: They claim this matters because it supports various types of research and more advanced NLP methods, specifically for legislative NER or other downstream tasks where you need to extract specific entities from congressional documents >

Tom: That’s right. The thesis is that by fine-tuning models like BERT and RoBERTa on this GELATO dataset for the first-level prediction, and then using LLMs with optimized prompts to finish the second level prediction, we can get a good look at how these combinations work >

Lu: They are essentially testing if this two-stage approach is effective for legislative text entities >

Jane: It’s about seeing how strong RoBERTa is for the first step and where the LLMs step in for that more complex second level labeling >

Tom: The results show that RoBERTa has a strong F1 score of eighty-eight point seven two on the first level, while BERT models are relatively weaker > <ref:2603.14130#pg3>

Meng: That’s a specific number to look at. So, what does that eighty-eight point seven two tell us about the quality of the initial entity extraction <ref:2603.14130#pg3>?

Tom: It tells us that RoBERTa is performing very well for this task when it comes to identifying the first set of entities in legislative text >

Jane: And then they use LLMs to handle the more granular second level predictions, and while BERT performs worse at that stage, the LLM can still complete the task >

Lu: That supports future research into using these model combinations as extraction tools for legislative NER because it shows a potential path forward >

Meng: But we have to be careful about those LLMs, right? They bring their general knowledge with them, which means their performance isn't guaranteed on this specialized data >

Jane: Exactly. The paper also points out that the similarity between entities in the Person and Class classes is subtle, and future work needs to explore why models might be missing those class predictions >

Tom: So the main takeaway here is that GELATO provides a structured way to benchmark model performance for legislative NER using this specific two-stage pipeline >

Lu: And it’s a useful benchmark for investigating the performance of models at NER in the legislative domain because it sets a standard for what to expect when working with this specific type of text >

Tom: The structure of GELATO is what makes it a useful benchmark for seeing how different model architectures handle the task >

Conclusion: Jane: We’re wrapping up our discussion on "The GELATO Dataset for Legislative NER". The authors are Matthew Flynn, Timothy Obiso, and Sam Newman from Brandeis University >

Tom: So the key implication is that this paper provides a structured benchmark to investigate model performance at Named Entity Recognition specifically in the legislative domain >

Jane: In simple terms, it shows us how important it is to tailor our data and our models when we are working with specialized text like congressional bills rather than just general news articles >

Tom: It also suggests that a two-stage prediction pipeline, using a strong model for the first level and an LLM for the second level, is a viable way to approach complex extraction problems in this area >

Lu: The finding that RoBERTa did better on the first level and BERT struggled there tells us we need to pay attention to which base models we choose when building our pipelines >

Meng: And we have to consider that the limitations they mentioned, like the dataset being limited to bills one hundred through two hundred of the 118th Congress > <ref:2603.14130#pg2>

Jane: Which means for practical application, we shouldn't assume this performance holds across all legislative texts without further testing >

Tom: So in a nutshell, GELATO is a useful resource for benchmarking how different AI models handle the specific challenges of legislative text NER and it points toward more targeted research down that road >

Lu: It’s a structured approach to annotation that sets a good precedent for creating high-quality datasets in specialized domains >

Meng: And we still need to ensure that whatever we build based on this data can handle the scale and the complexity of real legislative documents without breaking at scale >

More episodes

← Home