TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting

summary

Video file (mp4)

The gist

The gist The authors introduce a new TKG evolution benchmark to address shortcuts in existing temporal knowledge graph forecasting datasets by incorporating bias-corrected datasets and novel tasks

In short

The authors introduced TKGB, a new benchmark to fix shortcuts in temporal knowledge graph forecasting datasets where models could cheat by counting co-occurrences instead of using time information. They created four bias-corrected datasets and two novel tasks—generative knowledge forecasting and obsolescence prediction—to test models on how they actually learn the evolution of knowledge over time.

Key concepts

Co-occurrence Shortcut
This is a flaw in existing benchmarks where models achieve high scores by simply counting how often entities or relations appear together, without actually using any temporal information. The authors found this happens because datasets are biased and tasks are too simple, allowing models to memorize preferred patterns instead of learning true evolution.
Knowledge Evolution Stages
The benchmark includes two new tasks designed to simulate different phases of knowledge growth. Generative forecasting tests a model's ability to create future knowledge based on history, while obsolescence prediction tests its understanding of when existing facts will become invalid due to new information.
Textual Annotations
These are extra descriptions added to the knowledge graph data that explain the meaning and context of the relationships. The authors found that using these rich semantic details significantly improved model performance on forecasting tasks, showing that understanding 'why' knowledge changes is crucial.
TKG-ICL
This is a specific LLM-based model proposed by the authors. It performs well because it integrates time-interval knowledge and uses textual annotations to better grasp the causal mechanisms behind how temporal knowledge graphs evolve, outperforming other models in this new evaluation.

Terminology used across episodes

This episode discusses

The paper

TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting · Read on arXiv

Jiasheng Zhang, Zhengpin Li, Mingzhe Wang, Jie Shao, Jiangtao Cui, Hui Li

Xidian University · Peking University · University of Electronic Science and Technology of China · XI’AN University of Posts&Telecommunications

Temporal knowledge graph (TKG) forecasting has advanced remarkably in recent years. Yet we found existing benchmarks inadvertently introduce a shortcut: For a subset of test queries, models can achieve optimal evaluation metrics without using any time information. We trace this to two pieces of evidence: inherent dataset biases and an overly simplistic task formulation that allows models to exploit biases, revealing fundamental flaws in current evaluation protocols for differentiating true forecasting ability. Moreover, existing datasets rely on fixed history that poorly reflects real-world dynamics, lack semantic context, and fail to explicitly mitigate shortcut, preventing thorough assessment of forecasting abilities. To address these, we propose TKGB, a comprehensive benchmark for evaluating the capability of TKG forecasting models. By integrating four complementary tasks with an automatic pipeline for constructing up-to-date and context-enriched TKG datasets with controllable shortcuts, TKGB reveals capability differences and failure modes hidden under conventional evaluation.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting".

Jane: The gist The authors introduce a new TKG evolution benchmark to address shortcuts in existing temporal knowledge graph forecasting datasets by incorporating bias-corrected datasets and novel tasks that simulate knowledge…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper, TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting, and basically, they're saying that existing tests have a huge flaw where models can cheat by just counting how often things happen together without actually using any time information.

Jane: That sounds like a big problem because if models can get high scores by just looking at co-occurrences, they aren't learning the actual evolution of the knowledge graph, which is what we really want to measure.

Tom: Exactly. The authors point out this shortcut is happening because of inherent biases in these existing datasets and a way the task is set up that lets those biases be exploited >

Lu: What they are observing on page two is that in current TKGs, each entity interacts with only a very limited set of entities and relations, and most of these interactions have already been seen during training > <ref:2602.08353#pg3>

Meng: So if an entity only connects to a small group of things that the model has already seen, it's easy for it to just guess the missing piece based on what it has memorized >

Jane: And they conclude that under the old way of testing, once you know the subject entity and relation, the missing object can be predicted by simply memorizing those preferred entities >

Tom: It’s a huge problem because that means high performance isn't necessarily proof of understanding how knowledge evolves; it's just pattern matching on a limited set of connections >

Jane: So what do they propose to fix this, and why is this whole TKGB benchmark important for the field right now?

Tom: They introduce TKGB, which is a comprehensive benchmark for evaluating TKG forecasting models. They create four new datasets and two novel evaluation tasks designed to simulate different stages of knowledge evolution >

Lu: Those datasets include FinWiki, ICEWSWiki, YAGO130K, and WIKI500K. A key thing they did was making sure each dataset incorporates aligned timestamp and time-interval knowledge, which reveals mutual influences within the knowledge evolution >

Meng: And to fight that co-occurrence shortcut they mentioned on page three they constructed these datasets using strategies like a popularity-based strategy during construction and a k-core-based filtering strategy > <ref:2602.08353#pg2>

Jane: That sounds like they are trying to build datasets that actually reflect how knowledge evolves over time, rather than just being static snapshots >

Tom: They also introduced two novel tasks: generative knowledge forecasting, which directly generates all knowledge involving an entity in the next timestamp instead of just completing a missing fact >

Lu: And then there's knowledge obsolescence prediction, which aims to determine whether currently valid knowledge will become invalid in the future because of other new knowledge >

Meng: So they’re not just asking "what happens next" but also "what will disappear," which tests a much deeper understanding of the evolution process >

Jane: That sounds like a much more robust way to test if a model really grasps causality in the knowledge graph, moving beyond simple prediction >

Tom: The experimental results show that co-occurrence-based scoring performs poorly on these new benchmark datasets, often ten times worse than baselines >

Lu: They found that existing models tend to fit this co-occurrence shortcut; they learn to predict by counting co-occurrences instead of capturing the underlying evolutionary mechanisms >

Meng: And they specifically showed that TKG-ICL outperforms other models on this task, suggesting the ability of large language models in understanding the knowledge evolution mechanism with sufficient semantic information >

Jane: So it seems like integrating textual annotations, which describe knowledge semantics, really helps these LLM-based models perform better on those forecasting tasks >

Tom: That’s what they found when comparing their generative and obsolescence prediction tasks; precise semantics for understanding the causality behind knowledge evolution are crucial >

Lu: They also showed that in the inductive setting, the LLM-based model TKG-ICL maintains consistent performance, which demonstrates the potential of leveraging the inherent knowledge in LLMs to build more generalizable TKG forecasting models >

Meng: From a production standpoint, if a model like TKG-ICL can maintain that consistency using these semantic inputs, it means we might be able to deploy better systems even if they don't have perfect access to all temporal data >

Jane: So the paper is really showing us how to build better evaluation protocols that actually differentiate true forecasting ability from just exploiting dataset biases >

Tom: The main implication here is that we need benchmarks that actively resist these shortcuts, and TKGB seems like a strong step in providing a more comprehensive testbed for knowledge evolution learning >

Lu: It moves the focus away from what scores are high on old datasets and toward testing models on tasks that simulate the actual challenges of TKG evolution >

Meng: For us in production, this means we need to look at how these new evaluation modes translate into real-world system resilience when dealing with dynamic knowledge graphs >

Jane: It’s interesting how they used both aligned timestamp and time-interval knowledge as context for different models, showing its effectiveness in helping understand the knowledge formation process >

Tom: So, if you're driving or cooking and you think about how your routine changes over weeks, this paper is about testing if an AI can understand that subtle change versus just memorizing the usual sequence of events >

Conclusion: Tom: So we've been digging into this paper called TKGB, which is a new benchmark for temporal knowledge graph forecasting, and what they found is that existing tests have a problem where models can cheat by just counting how often things happen together without actually using any time information.

Jane: That sounds like a real trap. So what exactly did these authors do to try and stop that?

Tom: Well, they built this whole new benchmark, TKGB. They created four different datasets and two new tasks designed to simulate different stages of how knowledge evolves over time, trying to get around that co-occurrence shortcut we talked about earlier.

Lu: The four datasets—FinWiki, ICEWSWiki, YAGO130K, and WIKI500K—are all made distinct but they all have this key feature where each one has these aligned timestamp and time-interval knowledge that shows how the knowledge actually influences itself over time.

Meng: And to fight that shortcut specifically, they didn't just use the old data; they used strategies during construction, like a popularity-based strategy and a k-core filtering strategy, to make sure those datasets weren't just memorized patterns.

Jane: So the point is that instead of just looking at snapshots of knowledge, they are testing models on tasks that simulate building and breaking knowledge over time.

Tom: Right. And the results show that co-occurrence scoring performs way worse on this new benchmark, often ten times worse than other methods we've seen.

Jane: It sounds like they’re pushing the field to use more meaningful tests, not just high scores on old metrics. So what does this mean for a model developer right now?

Tom: It means that if you want a model that actually understands how knowledge evolves—not just memorizes patterns—you need to test it on something like this TKGB benchmark.

Jane: And the authors point out that models like TKG-ICL, which uses large language models, perform much better on these evolution tasks because those semantic details from the text are crucial for understanding the causality behind how knowledge forms.

Tom: Exactly. It’s about showing that leveraging the inherent knowledge in those LLMs can actually help build more generalizable forecasting models when you give them good semantic context.

Jane: It makes me wonder how much of our current forecasting success is just based on these kinds of shortcuts we discussed, and what it would take to truly measure deep evolution understanding.

More episodes

← Home