ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion

The paper, published at ACM MM '26 by Jiafan Li, Mengxue Yang, Jiaqi Zhu, Liang Chang, Ying Li, and Hongan Wang, addresses the task of multimodal knowledge graph completion (MMKGC). The authors note that "Knowledge graph completion (KGC) aims to infer missing entities or relations from incomplete graph structures, and has evolved into multimodal knowledge graph completion (MMKGC), where entities are associated with multiple modalities such as text and images."

The paper identifies two key limitations of existing approaches:

1. Traditional embedding-based methods: Traditional representation learning approaches follow the embedding-based paradigm and may struggle when relation-specific evidence is limited. Such methods fail particularly on few-shot or zero-shot relation types, as illustrated by the portrayer relation example in Figure 1.

2. LLM-based reasoning methods: These typically linearize graph structures into textual prompts, which obscures structural topology and neglects vital visual information.

Meanwhile, the authors observe that vision-language models (VLMs) excel at multimodal reasoning, they cannot natively interpret structured graph topology, particularly when it comes to knowledge graphs where nodes and edges carry complex semantics. This motivates the central research question: in the VLM era, can knowledge graph completion, particularly for multimodal knowledge graphs, directly benefit from visual representations of graph structures?

The paper identifies three key challenges: (1) subgraph extraction — VLMs can only process a single image, so a compact, query-relevant subgraph must be extracted; (2) subgraph visualization — the subgraph must be rendered in a layout that enables VLMs to effectively identify triples, understand their correlations, and capture associated textual/visual semantics; and (3) prompt integration — the prompt must "jointly incorporate local structural information from the subgraph and global features from representation learning, while harmonizing the interplay between the large structural subgraph image and small embedded entity images."

The paper makes three main contributions:

• First formulation of MMKGC as visual subgraph reasoning for VLMs: "This is the first work to formulate MMKGC as a visual subgraph reasoning problem for VLMs, through effectively integrating three complementary capabilities: multimodal representation learning to capture global semantics, VLM-based reasoning to extract local relational evidence, and VLMs' internal memory to provide supplementary commonsense knowledge."

• A systematic pipeline that "extracts a query-aware subgraph based on graph structure as well as node semantics, and converts it into a visually interpretable image, facilitating VLMs to jointly reason over graph topology and multimodal entity information."

• Extensive experiments on two real-world MMKG datasets demonstrating the coordinated synergy between relational reasoning and commonsense knowledge enabled by VLMs.

An MMKG is defined as a quadruple G = (E, R, T, M) where E is entities, R is relations, T ⊆ E × R × E is the set of factual triples, and M = M e denotes multimodal attributes for each entity. The task is to predict the missing entity: tˆ = argmax F (h, r, t ′) for tail prediction, with the symmetric formulation for head prediction.

The framework consists of four stages:

1. Multimodal Entity Representation. The authors adopt IMF as the multimodal encoder, which provides effective joint representations across textual, visual, and structural modalities, achieving near-best prediction performance among embedding-based methods. After training, each entity is associated with four embeddings: a textual embedding x t, a visual embedding x v, a structural embedding x s, and a fused multimodal embedding x m. They note this component can in principle be replaced by other representation learning models.

2. Query-aware Subgraph Extraction. Since directly reasoning over the entire knowledge graph is infeasible for VLMs, the method extracts a compact subgraph containing three categories of candidate edges:

• Relation-consistent edges (C rel): edges whose relation type r i matches the query relation r, while the head entity differs from the query entity — providing additional instances of the relational pattern.

• One-hop neighbor edges (C 1-hop): edges directly connected to the query entity h — offering the most immediate structural context.

• Two-hop neighbor edges (C 2-hop): edges connected to the one-hop neighbors of h, but excluding those that still contain the query entity h — capturing indirect structural relations and potential multi-hop reasoning paths.

Each candidate edge is scored via: score(h i, r i, t i) = λ sim(r, r i) + (1 − λ) max[sim(h, h i), sim(h, t i)], where entity similarity is computed as the average cosine similarity across the four embedding views. A structural bonus is added for edges sharing an element with the query: score∗(h i, r i, t i) = score(h i, r i, t i) + γ · δ((h i, r i, t i) ∈ C 1-hop ∪ C rel).

3. Subgraph Visualization. The extracted subgraph is rendered as an image via graph rendering operation R(·) using Graphviz. Based on empirical comparison of layout algorithms (dot, circo, twopi, neato, fdp, sfdp), the authors select dot as the default: the dot layout arranges nodes in a hierarchical manner and reduces edge crossings, making the resulting graph structure more readable and interpretable.

4. VLM Reasoning. The prompt combines textual input (the incomplete query triple, a textual serialization of the subgraph's key triples, the candidate entity set computed by representation learning, and task instructions) with visual input (the rendered subgraph image I sub and entity images I e). The prediction follows: eˆ = arg max P(e i T prompt, I sub, I e e∈V sub). The candidate list serves as guidance rather than a hard constraint. The VLM backbone is Qwen3VL.

Experiments are performed on FB15K-237 and DB15K, two widely used MMKG datasets. The test sets undergo strict preprocessing... retaining only triples with a unique correct answer for each query to remove ambiguity. FB15K-237 has 14,541 nodes, 237 relation types, and 272,115/17,535/6,102 train/val/test triples; DB15K has 14,777 nodes, 279 relations, and 56,881/9,903/8,355 triples. Evaluation uses Hit@1 and Hit@3, reported separately for head and tail predictions. Hyperparameters: k max = 15 edges, k rel = 10, k nei = 10, λ = 0.5, γ = 0.1, on an NVIDIA RTX 4090.

ViSR-KGC achieves the best performance across all metrics and datasets, e.g., FB15K-237 tail Hit@1 = 0.8027 and Hit@3 = 0.8836; DB15K tail Hit@1 = 0.6715 and Hit@3 = 0.7802. The paper draws three key observations:

1. Traditional embedding-based methods struggle with complex reasoning. The gap confirms that pure representation learning, which relies mainly on graph structure and embedding spaces, struggles to capture implicit relational patterns. Notably, "ViSR-KGC leverages commonsense knowledge beyond candidate sets obtained from the representation learning. Around 9.96% and 11.24% of correct predicted entities on FB15K-237 and DB15K respectively, fall outside the candidate set. An example is given of correctly answering (?, creator, Mike Henry(voice actor)) with This Is The Cleveland Show" by analyzing related entities (Family Guy, American Dad) plus internal knowledge of television animation production. On sparse relation types (fewer than quartile frequency, 7.6%/6.4% of edges), IMF achieves Hit@1 of 77.14%/37.5% versus ViSR-KGC's 85.16%/48.44%.

2. LLM-based methods lack explicit graph structure perception. Even multimodal LLM methods process graph structure implicitly through feature propagation and remain below our ViSR-KGC. The paper's case study shows that for query (University of Pittsburgh, affiliation, ?), an LLM predicts Pennsylvania (mistaking it for geographic association), while ViSR-KGC successfully identifies the correct entity by inspecting the visualized subgraph structure — specifically revealing the path University of Pittsburgh → Pennsylvania → Pennsylvania State University → Association of American Universities, which implicitly encodes the affiliation relationship.

3. Asymmetry between head and tail prediction. All methods perform significantly better on tail entity prediction than on head entity prediction because head entities are often more specific subjects (e.g. a person or organization), and tail entities are more general objects.

The ablations validate each component:

• W/o relation-consistent edges (C rel) drops tail Hit@1 to 0.6232; w/o two-hop edges (C 2-hop) drops to 0.6598, confirming multi-type edges provide focused evidence.

• W/o subgraph entirely drops tail Hit@1 to 0.4545, showing subgraph structure is critical. Removing only subgraph text drops to 0.5075, while removing only the subgraph image drops to 0.6667 — text is more critical, but the image adds additional structural cues.

• Layout changes (twopi/sfdp) worsen accuracy, confirming rendering the subgraph with an appropriate layout is crucial.

• W/o candidate entities drops tail Hit@1 to 0.6533, showing the list guide[s] the reasoning space of VLMs and prevents hallucination.

• W/o entity images drops to 0.6566, showing entity images provide complementary entity-level visual features.

• Replacing the encoder with ConvE, ConvKB, or HGNN-IMA substantially lowers accuracy (best among them: tail Hit@1 = 0.6392), confirming the quality of multimodal representations directly impacts the subgraph construction.

• k max: Performance improves from k max=10 to 15, then Hit@3 slightly decreases at 18 because excessive subgraphs may introduce noise and increase visual complexity. Default k max=15.

• λ: Both metrics increase as λ rises from 0.3 to 0.5, then decrease beyond 0.5; relation similarity is indeed beneficial, but over-emphasizing this measure would weaken the model's ability to distinguish entities. Default λ=0.5.

The paper concludes that ViSR-KGC effectively harnesses VLMs' capabilities in multimodal understanding and associative reasoning grounded in internal memory and integrates textual, visual, and structural information, supported by a carefully chosen multimodal encoder and the layout strategy. The synergy enables the capture of three levels of semantic correlations for accurate link prediction: global and local evidence within the knowledge graph, as well as implicit commonsense knowledge residing in VLMs.

Future work directions include "more efficient subgraph extraction strategies to handle larger-scale multimodal knowledge graphs, integrating additional modalities such as audio or video to enrich multimodal semantics, and adapting the approach to other reasoning tasks on semantic graphs, as well as adapting VLMs to MMKGC through reinforcement learning with verifiable rewards derived from link prediction correctness."

Based on ViSR-KGC, I would make the following concrete improvements to AI systems:

1. Add explicit visual subgraph reasoning to VLMs. Instead of linearizing knowledge graph triples into text, render a query-relevant subgraph as an image and feed it directly to the VLM along with entity images. Use a hierarchical, low-edge-crossing layout (like Graphviz dot).

Resulting capability: The system can perceive graph topology—paths, branching, and relation context—rather than only sequential text, enabling it to correctly answer multi-hop queries such as “University of Pittsburgh → Pennsylvania → Pennsylvania State University → Association of American Universities” that text-only LLM prompting gets wrong.

2. Use query-aware subgraph extraction with heterogeneous edge types. Select three complementary edge sets: relation-consistent edges (same relation, different subjects), one-hop neighbor edges, and two-hop neighbor edges. Score each candidate edge by a weighted combination of relation similarity and entity embedding similarity across text, image, structure, and fused modalities, plus a structural bonus for edges sharing the query head.

Resulting capability: The system can reason effectively on sparse or few-shot relations, focusing on the most relevant evidence while avoiding irrelevant graph noise.

3. Combine global embedding scores, local subgraph evidence, and VLM internal commonsense in one prompt. Use a multimodal representation model (e.g., IMF) to produce global entity candidates; render the subgraph image; provide the textual triple list; include the candidate set as soft guidance rather than a hard filter.

Resulting capability: The system can make correct predictions outside the embedding-based candidate set—about 10–11% of correct answers in the paper—by leveraging VLM commonsense knowledge (e.g., inferring “This Is The Cleveland Show” as creator of a voice actor from related TV shows), while the candidate list still reduces hallucination.

4. Provide both subgraph image and subgraph text simultaneously. Do not rely on image alone or text alone. Keep the visual subgraph for structural cues and the textual serialization of triples for precise relational facts.

Resulting capability: The system remains accurate even when image resolution or graph rendering is imperfect; ablations show that losing subgraph text drops tail Hit@1 from 0.8027 to 0.5075, while losing only the image still degrades performance, confirming the need for both modalities.

5. Use layout-aware graph rendering tuned for VLM perception. Choose a layout that minimizes edge crossings and hierarchical ambiguity (dot layout in the paper), and validate layout choice empirically.

Resulting capability: The system can reliably read graph structure from rendered images; switching to twopi or sfdp layouts measurably reduces accuracy, so the improved system treats visualization as part of the reasoning model, not as a fixed preprocessing step.

6. Make the multimodal encoder replaceable and representation-quality-aware. Train an encoder that produces textual, visual, structural, and fused embeddings, then use these embeddings both for scoring and for subgraph candidate selection.

Resulting capability: The system can be upgraded as better multimodal encoders appear; higher-quality embeddings directly improve subgraph construction and downstream link prediction (replacing the encoder with weaker models drops tail Hit@1 from 0.6715 to 0.6392 or less).

7. Handle head prediction and tail prediction asymmetrically. Because head entities are often specific subjects (people/organizations) while tails are general objects, generate direction-specific prompts, candidate scoring, and subgraph emphasis for head prediction.

Resulting capability: The system can close the gap between head and tail prediction accuracy, improving overall KG completion rather than optimizing only the easier tail direction.

8. Incorporate scalable subgraph extraction and RL-based adaptation for larger graphs. Replace exhaustive candidate scoring with scalable sampling or hierarchical subgraph retrieval; optionally fine-tune the VLM with reinforcement learning using link-prediction correctness as a verifiable reward.

Resulting capability: The system can scale to larger multimodal knowledge graphs and continuously improve its reasoning from its own correct/incorrect predictions.

Overall, an AI system improved with these changes can perform multimodal knowledge graph completion by jointly reasoning over global embedding semantics, local visual graph structure, entity images, and VLM internal commonsense—yielding substantially higher Hit@1 and Hit@3 on benchmarks like FB15K-237 and DB15K, especially for sparse relations and multi-hop relational queries.

Knowledge graph completion (KGC) aims to infer missing entities or relations from incomplete graph structures, and has evolved into multimodal knowledge graph completion (MMKGC), where entities are associated with multiple modalities such as text and images. Traditional representation learning approaches follow the embedding-based paradigm and may struggle when relation-specific evidence is limited. Meanwhile, LLM-based reasoning methods typically linearize graph structures into textual prompts, which obscures structural topology and neglects vital visual information. While vision-language models (VLMs) excel at multimodal reasoning, they cannot natively interpret structured graph topology, particularly when it comes to knowledge graphs where nodes and edges carry complex semantics. To bridge this gap, we propose ViSR-KGC, a visual subgraph reasoning approach for KGC. It integrates three complementary capabilities to capture semantic correlations: identifying global topology dependencies via representation learning, analyzing local multimodal evidence using VLMs, and providing necessary commonsense knowledge inherent in pre-trained models. Based on learned multimodal embeddings, our framework first extracts a compact and query-aware subgraph from the MMKG. Then, this subgraph is transformed into a visually interpretable image using a layout strategy selected through empirical comparison. Finally, the visualized subgraph, entity images, textual descriptions, and candidate answers are combined into a unified prompt, enabling the VLM to infer the missing entity.