DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion

summary

The gist

The gist: DeKoR proposes an efficient framework for GNN and LLM knowledge fusion by decomposing LLM knowledge into local and global views, distilling them into lightweight experts, and using a

In short

DeKoR proposes a method to efficiently combine Graph Neural Networks (GNNs) and Large Language Models (LLMs) by breaking down LLM knowledge into local and global views. It distills these views into lightweight experts that don't need the LLM during use. A node-adaptive router then intelligently combines predictions from these experts, improving accuracy while significantly reducing computational cost.

Key concepts

Local Knowledge View (Z): This view preserves the fine-grained details of a node's original representation from different prompt scales (ego, one-hop, two-hop). It keeps the specific, detailed information about each individual node intact, ensuring that subtle local features are not lost during the knowledge distillation process.
Global Knowledge View (Q): This view captures the overall semantic structure of the graph. It is created by generating global prototypes using K-means clustering on node representations and measuring how similar each node's representation is to these prototypes. This helps characterize the broader relationships within the entire graph.
Lightweight Experts: These are simplified models derived from the original LLM knowledge views. They are trained using specific loss functions (like InfoNCE and KL divergence) to capture either local or global semantic information efficiently. Crucially, these experts can be used for prediction without needing the large LLM model to run during deployment.
Node-Adaptive Router (phi): This component learns how much a specific node should trust each expert's prediction. It takes structural information and the predictions from both local and global experts to calculate personalized weights ($\alpha_v$). This allows the final output for any given node to be a balanced combination of its local and global knowledge.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Flow Matching — explained in 15 episodes

This episode discusses

The paper

DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion · Read on arXiv

Peiyuan Li, Yongqi Huang, Jitao Zhao, Di Jin, Dongxiao He

Tianjin University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion".

Jane: The gist: DeKoR proposes an efficient framework for GNN and LLM knowledge fusion by decomposing LLM knowledge into local and global views, distilling them into lightweight experts,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We're still talking about "DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion." The authors propose this framework to solve the problem of mismatched knowledge spaces between LLMs and GNNs by decomposing the LLM knowledge into local and global views.

Jane: So, to break it down, the paper says the local view preserves fine-grained node information, while the global view characterizes each node through its relations to global prototypes.

Lu: Exactly. They use K-means on node representations to get those prototypes for the global view Q, and then they compute similarity with those prototypes to form that global view Q, while Z keeps the original LLM representations from three different context scales: ego, one-hop, and two-hop prompts.

Meng: And then they distill this knowledge into lightweight experts using distillation objectives—specifically the symmetric InfoNCE loss Lnce for the local expert and a KL divergence loss Lglo for the global view.

Tom: That's how they make sure these decomposed forms are preserved in modules that don't need LLM inference during deployment.

Jane: And then after distillation, they move to node-adaptive routing, where a router learns how much each node should rely on each of those two experts to combine their predictions.

Lu: The final prediction is a linear combination of the local and global expert outputs, y hat v equals alpha loc v y loc v plus alpha glo v y glo v, and they have a balance loss Lbal to keep the routing weights balanced.

Meng: This whole process lets them use LLM knowledge as an offline teacher during training, which is key for making this setup practical.

Tom: The experiments show that DeKoR consistently improves performance across node and subgraph classification tasks while substantially reducing inference time and GPU memory usage compared to most GNN--LLM baselines.

Jane: Specifically, DeKoR achieves an average accuracy of eighty-five point seven eight percent across the five datasets, exceeding the strongest baseline, GLANCE, which was at eighty-four point five eight percent.

Lu: The ablation study is telling too; removing any of the key components—the router or either of those specific distillation losses—leads to a performance drop on every single test case.

Meng: That tells me the router isn't just some clever extra part; it’s crucial for how they balance using those two views effectively during routing.

Tom: So, what does this mean for us? It means we can combine the deep reasoning of language models with the structural understanding of graphs without crippling our computational budget.

Jane: It’s a practical way to get that LLM power into real-world graph applications efficiently by reorganizing how it learns first.

Conclusion: Tom: So we're wrapping up on "DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion." The title itself really captures the essence of what they did, which is breaking down knowledge and routing it intelligently.

Jane: It’s about taking a complex model, not just making it bigger, but reorganizing how it learns so we can use it where we need it most.

Lu: They are showing that you don't always have to keep the whole giant model running; sometimes you just need the right pieces of knowledge delivered efficiently.

Tom: The authors did have some caveats, though; they mentioned that their approach relies on using the LLM only as an offline teacher, which means it might not capture all the dynamic interactions that happen when you are running things in real-time.

Meng: If this works at scale, it could seriously lower the barrier for using LLM reasoning in things like network analysis or molecular modeling where computation is tight.

More episodes

← Home