An Agentic Hybrid Top-Down and Bottom-Up Approach to Knowledge Graph Generation

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "An Agentic Hybrid Top-Down and Bottom-Up Approach to Knowledge Graph Generation".

Jane: The paper was written by Emma Jouffroy, Warren Jouanneau and Marc Palyart from Malt.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: Alright, we finally get to dig into this one.

Jane: And it hits close to home for anyone who's ever cleaned up a messy spreadsheet.

Tom: The crew at Malt — the European freelancer marketplace — built a knowledge graph generator.

Jane: Their raw material is brutal. Freelancers typing "gestion de projet web", "Web PM", "Project Management".

Lu: Across five languages, with typos, jargon, and fresh skills appearing every week.

Meng: The paper's core thesis is a hybrid — marry top-down structure to bottom-up discovery.

Tom: Top-down means anchoring every recognized skill to a stable Wikidata entity.

Jane: Bottom-up means letting an LLM spot long-tail skills that Wikidata never registered.

Lu: Then wiring those new skills into the graph with proper metadata.

Meng: They wrap that in an "agentic reflection" loop — the model critiques and corrects its own output.

Tom: The headline results are strong. 36,037 raw strings went into the pipeline.

Jane: 77 percent of them resolved to a knowledge graph node. The rest got flagged as noise or non-skills.

Lu: Those mapped strings collapsed into 13,298 canonical skill nodes.

Tom: That's a 52.1 percent compression rate.

Jane: So the platform went from 36,000 messy strings to about 13,000 clean concepts.

Meng: And each node carries labels in all five languages. 66,490 labels total.

Lu: That matters for matching — a search in Dutch finds the same node as a search in French.

Tom: They also benchmarked against a gold standard annotated by five domain experts.

Jane: Found coverage reached 84.9 percent — when the model attempts a match, it's usually right.

Meng: The overall wrong guess rate sits at 19.1 percent.

Lu: Sounds high, until you see the input. Real profiles are beautifully chaotic.

Lalam: Step back — the broader promise is a living graph for a fast-moving labor market.

Tom: And that matters because matching humans to work is genuinely high-stakes.

Jane: Plus you can audit every node back to its source. That's rare in this space.

Meng: I want to see how they keep the LLM from hallucinating skills out of thin air.

Tom: Then page one is our stop. It frames that exact problem.

Page 1: Jane: We've got the thesis. Page one now frames where this fits in the world.

Tom: The paper opens with the two classic ways to build these structures.

Lu: Rigid top-down methods lean on fixed ontologies like ESCO.

Meng: ESCO gives you consistency and high precision, sure.

Jane: But it struggles with emerging and hyper-local concepts.

Tom: Then there's chaotic bottom-up — purely generative clustering.

Lu: That's flexible, it can catch novel trends.

Meng: But it has no guardrails, so equivalent concepts fragment into redundant clusters.

Tom: They use a project management story to make it concrete.

Jane: "Project Management" and "Web Project Management" should sit together in a hierarchy.

Meng: A rigid ontology misses the web specialization entirely.

Lu: A wild bottom-up system might split them apart with no relationship at all.

Tom: And worse — it can hallucinate links. Their example shows "Excel" somehow attached.

Jane: Excel got linked to project management? That's the kind of noise HR systems can't tolerate.

Lu: Exactly. The hybrid they propose anchors the baseline to Wikidata entity Q179012.

Meng: So "Project Management" gets a stable, verifiable anchor.

Tom: Then agentic reflection synthesizes "Web Project Management" as a distinct child node.

Jane: With relational metadata tying it back to the parent.

Lu: And that absorbs all the shortcuts — "Web PM" — into one clean structure.

Tom: The key move is dividing labor.

Jane: Recognized concepts get grounded in the knowledge graph. Unrecognized ones get generative reflection.

Meng: That split is what makes the whole thing auditable.

Lalam: The philosophical point is you don't trust the model with everything; you trust it only at the edges.

Tom: It's a nice division of responsibility.

Jane: And it quietly addresses the hallucination worry, at least in principle.

Meng: I want to know how previous attempts failed before they landed here.

Tom: That's literally page two. Related work has the autopsy.

Page 2: Meng: So we're on to the autopsy. Page two walks through the related work.

Tom: Taxonomy creation started with expert-curated ontologies and static knowledge bases.

Jane: Expensive and slow. Then automated ML methods arrived.

Lu: Early skill extraction models, even efficient encoders, produced flat lists.

Meng: Flat lists mean semantic ambiguity. "Java" the island, "Java" the language, "Java" the coffee.

Tom: Top-down methods tried to fix that by anchoring to knowledge graphs.

Jane: They used domain seeds, tree expansion, LLM-driven ranking.

Lu: But static methods can't keep up with a moving job market.

Meng: Bottom-up clustering scales well, yet its labels are often uninterpretable.

Jane: And the clusters lack relational metadata — you get bags, not graphs.

Tom: Then LLM-based work started structuring categories through abstractive prompting.

Lu: Localized induction, specialized schemas, end-to-end generation.

Jane: End-to-end batch processing still fragments hierarchies.

Meng: Recent multi-agent frameworks added reflection and dynamic alignment.

Tom: Self-correction, prompt optimization, tree search — the reasoning toolkit got bigger.

Jane: So with all that machinery, what's still missing?

Lu: The paper names three persistent failures.

Meng: Hallucinated skills, formatting instability, and bias.

Tom: And that's where the hybrid architecture makes its case.

Jane: The LLM drives the pipeline, but its reasoning is anchored to a deterministic knowledge graph for anything recognized.

Lu: Unconstrained generation only happens for unmapped skills and their relational metadata.

Meng: So the model's imagination is fenced in.

Lalam: That's the real contribution — freedom at the frontier, discipline at the core.

Tom: I like that framing. Discipline at the core.

Jane: Now I need to see that discipline in practice. Page three shows the machinery.

Meng: Five stages, one iterative loop. Let's look.

Page 3: Tom: So the machinery. The paper runs everything as an iterative loop, not a straight line.

Jane: Five stages, and the whole thing cycles until the graph stabilizes.

Lu: They chose Wikidata as the anchor because its multilingual coverage is massive and open.

Meng: And they drive it with Gemini 1.5 Flash — a lightweight model chosen for cost and availability.

Tom: Stage one is reconciliation. That's where raw text meets Wikidata.

Jane: The engine pulls the top ten Wikidata candidates for the input string.

Lu: But a bare string like "gestion de projet web" carries almost no context.

Meng: So they enrich it with two empirical features from the freelancer's profile.

Tom: The top fifteen co-occurring peer skills.

Jane: Plus the top five professional categories from their history.

Lu: That context lets the LLM disambiguate properly.

Meng: The output is a structured JSON — chosen QIDs, confidence scores, step-by-step reasoning.

Tom: And two critical flags: is skill and is compound.

Jane: So the model has to decide whether something is even a skill at all.

Lu: And whether it's actually several skills mashed together.

Meng: The multilingual trick here is smart. Candidates are evaluated in all five languages at once.

Tom: That forces "Réseaux sociaux" and "Social Media" onto the same global QID.

Jane: No language silos. No splitting by translation.

Lu: Stage two is canonicalization — grouping validated inputs by their QID combinations.

Meng: The model generates human-readable preferred labels for all five languages.

Tom: There's a strict fallback hierarchy.

Jane: First priority goes to the most-used label on the platform.

Lu: Then the official Wikidata title.

Meng: And only if neither works does the model synthesize something new.

Tom: Every label carries a provenance tag — malt, wikidata, or generative.

Jane: That tag is how you keep the whole graph explainable.

Lalam: And that's the quiet revolution. The model proposes, but the source of truth stays visible.

Tom: Wait until you see what happens to the skills that don't fit. That's page four.

Jane: The curation stage, and the orphan queue.

Page 4: Jane: Page four takes us into the heart of the system — the curation stage.

Tom: This is where the agentic reflection actually bites.

Lu: The curation agent checks whether every raw skill is truly equivalent to its group.

Meng: If it's not equivalent, the model explains why.

Tom: Seven granular rejection criteria — ambiguous, specialization, semantic mismatch, not a skill, methodology, context, sub-task.

Jane: The interesting part is that rejections aren't just thrown away.

Lu: The model generates a suggested pref label for each rejected concept.

Meng: So a rejected specialization keeps its identity, just under a better name.

Tom: There's a safety valve too. Any node with over 50 percent rejection gets flagged for human review.

Jane: That prevents one bad cluster from cascading through the whole graph.

Lu: Stage four handles consolidation — deduplication across batches.

Meng: They use asymmetric bootstrapping to avoid comparing everything to everything.

Tom: The first batch builds the baseline graph. Every later batch compares only against that.

Jane: Lightweight heuristics flag potential merges first — shared skills or low edit distance between labels.

Lu: Then the LLM makes the final call: merge or keep separate.

Meng: And those decisions get cached, so future epochs don't re-litigate them.

Tom: Stage five is the cleverest bit. Orphan recovery.

Jane: Remember "gestion de projet web" getting rejected as a specialization?

Lu: It becomes an orphan with a synthetic identifier derived from its suggested label.

Meng: Identical concepts get identical suggested labels, so they naturally group together.

Tom: Next epoch, those orphans are re-injected and form a stable sub-branch.

Jane: Linked back to the parent Wikidata entity. The graph heals itself.

Lu: The loop repeats until the orphan queue is empty — full semantic convergence.

Meng: They call it a self-healing loop. That's not hype; it's literally the architecture.

Lalam: This is where the hybrid earns its name. Wikidata gives the skeleton, orphans build the new muscle.

Tom: And the whole thing runs without a human babysitting each merge.

Jane: The question is whether it actually works at scale.

Lu: That's exactly what page five answers. Numbers time.

Page 5: Lu: Numbers time. Page five opens with the coverage results.

Tom: 36,037 raw strings went in. 27,743 got resolved — a 77 percent global coverage rate.

Jane: The remaining 8,294 got flagged as non-skills or semantic noise.

Meng: The mapped strings grouped into 15,010 semantic groupings first.

Tom: Then streamlined down to 13,298 canonical skill nodes.

Jane: That compression rate — 52.1 percent — is the whole point. Downstream redundancy vanishes.

Lu: And the average sits at 2.08 variations per canonical node.

Meng: Then comes the cross-lingual claim, and it's striking.

Tom: 100 percent of the 13,298 nodes are fully supported across all five locales.

Jane: That's exactly 66,490 standardized labels. Perfect symmetry.

Lu: The Pareto distribution backs it up too.

Tom: The top 1,000 canonical skills cover 82.74 percent of platform usage.

Jane: The top 5,000 capture 97.25 percent.

Meng: So the long tail is basically pure specialty — and mostly noise-free after cleanup.

Tom: The paper draws a sharp line between lexical long tail and semantic long tail.

Jane: Typos and redundant variants get compressed hard.

Lu: But rare, emerging capabilities get preserved through the orphan loop.

Meng: That's why the final graph stays rich despite the aggressive filtering.

Tom: Then comes the gold standard evaluation. Five domain experts, no overlap.

Jane: Global alignment coverage is 79.7 percent. Found coverage is 84.9 percent.

Lu: Wrong guess rate lands at 19.1 percent on the full input set.

Tom: And performance varies by domain — that's the honest part.

Jane: Video games hit 91.8 percent found coverage.

Lu: Communication lags at 81 percent — softer, more subjective terminology.

Meng: The rejection table tells its own story. SCORE REJECTED leads at 31.9 percent.

Tom: NODE CURATION follows at 28.1 percent. NOT A SKILL blocks 13.7 percent.

Jane: So the pipeline is rejecting aggressively and explaining every refusal.

Lalam: Those percentages are the real audit trail. Every decision leaves a paper trail.

Tom: Exactly. And page six pushes that explainability even further.

Jane: Provenance data and the bias discussion. That's next.

Page 6: Tom: Page six opens with provenance, and the numbers are reassuring.

Jane: 80.65 percent of the knowledge graph is anchored in factual data.

Lu: Split between 67.28 percent empirical platform usage and 22.08 percent Wikidata titles.

Meng: Only 19.35 percent of labels are purely generative.

Tom: So the model's imagination is a sliver, not the foundation.

Jane: They also report outlier rates of 0.01 to 0.06 per sub-graph after human review.

Lu: That's tiny. The curation agent is doing its job.

Meng: Then the discussion turns to real-world deployment.

Tom: A streamlined taxonomy from this graph is already running in Malt's candidate matching.

Jane: The graph gives the matching engine an interpretable intermediate layer.

Lu: That's huge for auditability. You can trace why a match happened.

Meng: They're also honest about Wikidata's weaknesses.

Tom: It lags on niche HR jargon, and its generalist nature creates structural inconsistencies.

Jane: But the orphan recovery loop acts as a safety net for exactly those gaps.

Lu: Then comes the bias section, and it's thoughtful.

Meng: LLMs are English-centric. Forcing everything into English structures erases local nuance.

Tom: They want to preserve cross-lingual variation instead of over-normalizing.

Jane: And gender-marked terms — French and German occupational titles carry grammatical gender.

Lu: Their goal is mapping gendered variants to shared concepts while keeping the distinct forms.

Meng: That's a genuinely hard problem, and they're naming it early.

Tom: The future work roadmap has three tracks.

Jane: Extended evaluations against ESCO, TnT-LLM, and CLIMB.

Lu: Cost and scalability — token consumption, latency, optimization.

Meng: And failure analysis — tracking manual interventions and systematic mapping errors.

Lalam: The responsible deployment checklist. Rare to see it laid out so explicitly.

Tom: And rare to see a paper admit what's still unbenchmarked.

Jane: The consolidation phase is new and hasn't been formally evaluated yet.

Lu: That honesty makes the whole thing more credible.

Meng: Alright. I think we're ready to close the book on this one.

Conclusion: Tom: So let's wrap this up. The paper gave us a hybrid pipeline that builds skills knowledge graphs.

Jane: Five stages — reconciliation, canonicalization, curation, consolidation, iteration.

Lu: Wikidata grounds the recognized concepts. Agentic reflection catches the rest.

Meng: 36,037 messy strings became 13,298 clean, multilingual skill nodes.

Tom: 77 percent coverage, 84.9 percent found coverage, and a 52.1 percent compression rate.

Jane: Every node traceable back to a source — platform usage, Wikidata, or a labeled synthesis.

Lalam: The bigger takeaway is the architecture philosophy. Ground first, generate only at the edges.

Tom: That's what makes it both scalable and defensible.

Jane: And the self-healing orphan loop means the graph keeps up with the market.

Lu: New skills don't break the structure; they grow it.

Meng: The bias discussion shows they're thinking about who gets represented and how.

Tom: For HR platforms, this is a blueprint.

Jane: For anyone building knowledge graphs from noisy text, it's a template.

Lu: And they published all the prompts in the appendix. You can reproduce the pipeline.

Meng: With a different model, different data — the architecture stands on its own.

Tom: The honest caveat remains. The consolidation phase still needs formal benchmarking.

Jane: But as ongoing work, this is a strong foundation.

Lalam: It moves the field from static taxonomies to living graphs.

Tom: And that's a genuinely useful step forward.

Jane: Alright, we're saying goodbye to this one.

Tom: Thanks to the Malt team — Emma, Warren, and Marc — for sharing the work.

Jane: Goodbye, paper. We'll see what's next on the arXiv feed.

Meng: I'm curious what follows this. The feed never sleeps.

Tom: Then let's find out together.