page_by_page
The hosts discuss a paper from Malt on generating knowledge graphs from messy freelancer skill data. They explain the hybrid approach: anchoring recognized skills to Wikidata entities while using LLMs to discover and integrate new skills, with an agentic reflection loop for self-correction. Results show 77% resolution and 52% compression, with a focus on auditability and bias.
Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Next we'll be talking about the paper "An Agentic Hybrid Top-Down and Bottom-Up Approach to Knowledge Graph Generation".
Jane: The paper was written by Emma Jouffroy, Warren Jouanneau and Marc Palyart from Malt.
Tom: Stay tuned as we take you through the paper and discuss its implications.
Paper summary: Tom: Alright, we finally get to dig into this one.
Jane: And it hits close to home for anyone who's ever cleaned up a messy spreadsheet.
Tom: The crew at Malt — the European freelancer marketplace — built a knowledge graph generator.
Jane: Their raw material is brutal. Freelancers typing "gestion de projet web", "Web PM", "Project Management".
Lu: Across five languages, with typos, jargon, and fresh skills appearing every week.
Meng: The paper's core thesis is a hybrid — marry top-down structure to bottom-up discovery.
Tom: Top-down means anchoring every recognized skill to a stable Wikidata entity.
Jane: Bottom-up means letting an LLM spot long-tail skills that Wikidata never registered.
Lu: Then wiring those new skills into the graph with proper metadata.
Meng: They wrap that in an "agentic reflection" loop — the model critiques and corrects its own output.
Tom: The headline results are strong. 36,037 raw strings went into the pipeline.
Jane: 77 percent of them resolved to a knowledge graph node. The rest got flagged as noise or non-skills.
Lu: Those mapped strings collapsed into 13,298 canonical skill nodes.
Tom: That's a 52.1 percent compression rate.
Jane: So the platform went from 36,000 messy strings to about 13,000 clean concepts.
Meng: And each node carries labels in all five languages. 66,490 labels total.
Lu: That matters for matching — a search in Dutch finds the same node as a search in French.
Tom: They also benchmarked against a gold standard annotated by five domain experts.
Jane: Found coverage reached 84.9 percent — when the model attempts a match, it's usually right.
Meng: The overall wrong guess rate sits at 19.1 percent.
Lu: Sounds high, until you see the input. Real profiles are beautifully chaotic.
Lalam: Step back — the broader promise is a living graph for a fast-moving labor market.
Tom: And that matters because matching humans to work is genuinely high-stakes.
Jane: Plus you can audit every node back to its source. That's rare in this space.
Meng: I want to see how they keep the LLM from hallucinating skills out of thin air.
Tom: Then page one is our stop. It frames that exact problem.
Page 1: Jane: We've got the thesis. Page one now frames where this fits in the world.
Tom: The paper opens with the two classic ways to build these structures.
Lu: Rigid top-down methods lean on fixed ontologies like ESCO.
Meng: ESCO gives you consistency and high precision, sure.
Jane: But it struggles with emerging and hyper-local concepts.
Tom: Then there's chaotic bottom-up — purely generative clustering.
Lu: That's flexible, it can catch novel trends.
Meng: But it has no guardrails, so equivalent concepts fragment into redundant clusters.
Tom: They use a project management story to make it concrete.
Jane: "Project Management" and "Web Project Management" should sit together in a hierarchy.
Meng: A rigid ontology misses the web specialization entirely.
Lu: A wild bottom-up system might split them apart with no relationship at all.
Tom: And worse — it can hallucinate links. Their example shows "Excel" somehow attached.
Jane: Excel got linked to project management? That's the kind of noise HR systems can't tolerate.
Lu: Exactly. The hybrid they propose anchors the baseline to Wikidata entity Q179012.
Meng: So "Project Management" gets a stable, verifiable anchor.
Tom: Then agentic reflection synthesizes "Web Project Management" as a distinct child node.
Jane: With relational metadata tying it back to the parent.
Lu: And that absorbs all the shortcuts — "Web PM" — into one clean structure.
Tom: The key move is dividing labor.
Jane: Recognized concepts get grounded in the knowledge graph. Unrecognized ones get generative reflection.
Meng: That split is what makes the whole thing auditable.
Lalam: The philosophical point is you don't trust the model with everything; you trust it only at the edges.
Tom: It's a nice division of responsibility.
Jane: And it quietly addresses the hallucination worry, at least in principle.
Meng: I want to know how previous attempts failed before they landed here.
Tom: That's literally page two. Related work has the autopsy.
Page 2: Meng: So we're on to the autopsy. Page two walks through the related work.
Tom: Taxonomy creation started with expert-curated ontologies and static knowledge bases.
Jane: Expensive and slow. Then automated ML methods arrived.
Lu: Early skill extraction models, even efficient encoders, produced flat lists.
Meng: Flat lists mean semantic ambiguity. "Java" the island, "Java" the language, "Java" the coffee.
Tom: Top-down methods tried to fix that by anchoring to knowledge graphs.
Jane: They used domain seeds, tree expansion, LLM-driven ranking.
Lu: But static methods can't keep up with a moving job market.
Meng: Bottom-up clustering scales well, yet its labels are often uninterpretable.
Jane: And the clusters lack relational metadata — you get bags, not graphs.
Tom: Then LLM-based work started structuring categories through abstractive prompting.
Lu: Localized induction, specialized schemas, end-to-end generation.
Jane: End-to-end batch processing still fragments hierarchies.
Meng: Recent multi-agent frameworks added reflection and dynamic alignment.
Tom: Self-correction, prompt optimization, tree search — the reasoning toolkit got bigger.
Jane: So with all that machinery, what's still missing?
Lu: The paper names three persistent failures.
Meng: Hallucinated skills, formatting instability, and bias.
Tom: And that's where the hybrid architecture makes its case.
Jane: The LLM drives the pipeline, but its reasoning is anchored to a deterministic knowledge graph for anything recognized.
Lu: Unconstrained generation only happens for unmapped skills and their relational metadata.
Meng: So the model's imagination is fenced in.
Lalam: That's the real contribution — freedom at the frontier, discipline at the core.
Tom: I like that framing. Discipline at the core.
Jane: Now I need to see that discipline in practice. Page three shows the machinery.
Meng: Five stages, one iterative loop. Let's look.
Page 3: Tom: So the machinery. The paper runs everything as an iterative loop, not a straight line.
Jane: Five stages, and the whole thing cycles until the graph stabilizes.
Lu: They chose Wikidata as the anchor because its multilingual coverage is massive and open.
Meng: And they drive it with Gemini 1.5 Flash — a lightweight model chosen for cost and availability.
Tom: Stage one is reconciliation. That's where raw text meets Wikidata.
Jane: The engine pulls the top ten Wikidata candidates for the input string.
Lu: But a bare string like "gestion de projet web" carries almost no context.
Meng: So they enrich it with two empirical features from the freelancer's profile.
Tom: The top fifteen co-occurring peer skills.
Jane: Plus the top five professional categories from their history.
Lu: That context lets the LLM disambiguate properly.
Meng: The output is a structured JSON — chosen QIDs, confidence scores, step-by-step reasoning.
Tom: And two critical flags: is_skill and is_compound.
Jane: So the model has to decide whether something is even a skill at all.
Lu: And whether it's actually several skills mashed together.
Meng: The multilingual trick here is smart. Candidates are evaluated in all five languages at once.
Tom: That forces "Réseaux sociaux" and "Social Media" onto the same global QID.
Jane: No language silos. No splitting by translation.
Lu: Stage two is canonicalization — grouping validated inputs by their QID combinations.
Meng: The model generates human-readable preferred labels for all five languages.
Tom: There's a strict fallback hierarchy.
Jane: First priority goes to the most-used label on the platform.
Lu: Then the official Wikidata title.
Meng: And only if neither works does the model synthesize something new.
Tom: Every label carries a provenance tag — malt, wikidata, or generative.
Jane: That tag is how you keep the whole graph explainable.
Lalam: And that's the quiet revolution. The model proposes, but the source of truth stays visible.
Tom: Wait until you see what happens to the skills that don't fit. That's page four.
Jane: The curation stage, and the orphan queue.
Page 4: Jane: Page four takes us into the heart of the system — the curation stage.
Tom: This is where the agentic reflection actually bites.
Lu: The curation agent checks whether every raw skill is truly equivalent to its group.
Meng: If it's not equivalent, the model explains why.
Tom: Seven granular rejection criteria — ambiguous, specialization, semantic mismatch, not a skill, methodology, context, sub-task.
Jane: The interesting part is that rejections aren't just thrown away.
Lu: The model generates a suggested_pref_label for each rejected concept.
Meng: So a rejected specialization keeps its identity, just under a better name.
Tom: There's a safety valve too. Any node with over 50 percent rejection gets flagged for human review.
Jane: That prevents one bad cluster from cascading through the whole graph.
Lu: Stage four handles consolidation — deduplication across batches.
Meng: They use asymmetric bootstrapping to avoid comparing everything to everything.
Tom: The first batch builds the baseline graph. Every later batch compares only against that.
Jane: Lightweight heuristics flag potential merges first — shared skills or low edit distance between labels.
Lu: Then the LLM makes the final call: merge or keep separate.
Meng: And those decisions get cached, so future epochs don't re-litigate them.
Tom: Stage five is the cleverest bit. Orphan recovery.
Jane: Remember "gestion de projet web" getting rejected as a specialization?
Lu: It becomes an orphan with a synthetic identifier derived from its suggested label.
Meng: Identical concepts get identical suggested labels, so they naturally group together.
Tom: Next epoch, those orphans are re-injected and form a stable sub-branch.
Jane: Linked back to the parent Wikidata entity. The graph heals itself.
Lu: The loop repeats until the orphan queue is empty — full semantic convergence.
Meng: They call it a self-healing loop. That's not hype; it's literally the architecture.
Lalam: This is where the hybrid earns its name. Wikidata gives the skeleton, orphans build the new muscle.
Tom: And the whole thing runs without a human babysitting each merge.
Jane: The question is whether it actually works at scale.
Lu: That's exactly what page five answers. Numbers time.
Page 5: Lu: Numbers time. Page five opens with the coverage results.
Tom: 36,037 raw strings went in. 27,743 got resolved — a 77 percent global coverage rate.
Jane: The remaining 8,294 got flagged as non-skills or semantic noise.
Meng: The mapped strings grouped into 15,010 semantic groupings first.
Tom: Then streamlined down to 13,298 canonical skill nodes.
Jane: That compression rate — 52.1 percent — is the whole point. Downstream redundancy vanishes.
Lu: And the average sits at 2.08 variations per canonical node.
Meng: Then comes the cross-lingual claim, and it's striking.
Tom: 100 percent of the 13,298 nodes are fully supported across all five locales.
Jane: That's exactly 66,490 standardized labels. Perfect symmetry.
Lu: The Pareto distribution backs it up too.
Tom: The top 1,000 canonical skills cover 82.74 percent of platform usage.
Jane: The top 5,000 capture 97.25 percent.
Meng: So the long tail is basically pure specialty — and mostly noise-free after cleanup.
Tom: The paper draws a sharp line between lexical long tail and semantic long tail.
Jane: Typos and redundant variants get compressed hard.
Lu: But rare, emerging capabilities get preserved through the orphan loop.
Meng: That's why the final graph stays rich despite the aggressive filtering.
Tom: Then comes the gold standard evaluation. Five domain experts, no overlap.
Jane: Global alignment coverage is 79.7 percent. Found coverage is 84.9 percent.
Lu: Wrong guess rate lands at 19.1 percent on the full input set.
Tom: And performance varies by domain — that's the honest part.
Jane: Video games hit 91.8 percent found coverage.
Lu: Communication lags at 81 percent — softer, more subjective terminology.
Meng: The rejection table tells its own story. SCORE_REJECTED leads at 31.9 percent.
Tom: NODE_CURATION follows at 28.1 percent. NOT_A_SKILL blocks 13.7 percent.
Jane: So the pipeline is rejecting aggressively and explaining every refusal.
Lalam: Those percentages are the real audit trail. Every decision leaves a paper trail.
Tom: Exactly. And page six pushes that explainability even further.
Jane: Provenance data and the bias discussion. That's next.
Page 6: Tom: Page six opens with provenance, and the numbers are reassuring.
Jane: 80.65 percent of the knowledge graph is anchored in factual data.
Lu: Split between 67.28 percent empirical platform usage and 22.08 percent Wikidata titles.
Meng: Only 19.35 percent of labels are purely generative.
Tom: So the model's imagination is a sliver, not the foundation.
Jane: They also report outlier rates of 0.01 to 0.06 per sub-graph after human review.
Lu: That's tiny. The curation agent is doing its job.
Meng: Then the discussion turns to real-world deployment.
Tom: A streamlined taxonomy from this graph is already running in Malt's candidate matching.
Jane: The graph gives the matching engine an interpretable intermediate layer.
Lu: That's huge for auditability. You can trace why a match happened.
Meng: They're also honest about Wikidata's weaknesses.
Tom: It lags on niche HR jargon, and its generalist nature creates structural inconsistencies.
Jane: But the orphan recovery loop acts as a safety net for exactly those gaps.
Lu: Then comes the bias section, and it's thoughtful.
Meng: LLMs are English-centric. Forcing everything into English structures erases local nuance.
Tom: They want to preserve cross-lingual variation instead of over-normalizing.
Jane: And gender-marked terms — French and German occupational titles carry grammatical gender.
Lu: Their goal is mapping gendered variants to shared concepts while keeping the distinct forms.
Meng: That's a genuinely hard problem, and they're naming it early.
Tom: The future work roadmap has three tracks.
Jane: Extended evaluations against ESCO, TnT-LLM, and CLIMB.
Lu: Cost and scalability — token consumption, latency, optimization.
Meng: And failure analysis — tracking manual interventions and systematic mapping errors.
Lalam: The responsible deployment checklist. Rare to see it laid out so explicitly.
Tom: And rare to see a paper admit what's still unbenchmarked.
Jane: The consolidation phase is new and hasn't been formally evaluated yet.
Lu: That honesty makes the whole thing more credible.
Meng: Alright. I think we're ready to close the book on this one.
Conclusion: Tom: So let's wrap this up. The paper gave us a hybrid pipeline that builds skills knowledge graphs.
Jane: Five stages — reconciliation, canonicalization, curation, consolidation, iteration.
Lu: Wikidata grounds the recognized concepts. Agentic reflection catches the rest.
Meng: 36,037 messy strings became 13,298 clean, multilingual skill nodes.
Tom: 77 percent coverage, 84.9 percent found coverage, and a 52.1 percent compression rate.
Jane: Every node traceable back to a source — platform usage, Wikidata, or a labeled synthesis.
Lalam: The bigger takeaway is the architecture philosophy. Ground first, generate only at the edges.
Tom: That's what makes it both scalable and defensible.
Jane: And the self-healing orphan loop means the graph keeps up with the market.
Lu: New skills don't break the structure; they grow it.
Meng: The bias discussion shows they're thinking about who gets represented and how.
Tom: For HR platforms, this is a blueprint.
Jane: For anyone building knowledge graphs from noisy text, it's a template.
Lu: And they published all the prompts in the appendix. You can reproduce the pipeline.
Meng: With a different model, different data — the architecture stands on its own.
Tom: The honest caveat remains. The consolidation phase still needs formal benchmarking.
Jane: But as ongoing work, this is a strong foundation.
Lalam: It moves the field from static taxonomies to living graphs.
Tom: And that's a genuinely useful step forward.
Jane: Alright, we're saying goodbye to this one.
Tom: Thanks to the Malt team — Emma, Warren, and Marc — for sharing the work.
Jane: Goodbye, paper. We'll see what's next on the arXiv feed.
Meng: I'm curious what follows this. The feed never sleeps.
Tom: Then let's find out together.