Curate Before You Connect: Identity and Ontology Tagging in a Production Knowledge Graph

This paper describes the ingestion and ontology-tagging layer that turns a validated extraction stream into a knowledge graph of 537,157 entities and 2,198,567 relationships drawn from 98,795 government documents. The central thesis is that "Extraction produces candidate entities and relationships; writing them into a graph is where identity is decided, and identity decisions are destructive in a way extraction errors are not. A wrong type can be corrected later, but two records merged under one identity cannot be separated once their properties have been combined, and the merge leaves no error behind."

The paper's contributions are fivefold:

1. A record-identity ladder for de-duplication within parsed tables, deciding sameness from identifier columns, name columns, display names, and type-scoped position, in that order, rather than from name similarity. The graph write applies a separate canonical-name key, which the authors state and defend rather than measure: the paper reports no identity benchmark.

2. An incident analysis of an automated merge that corrupted a correct record and deleted eight entities from an unrelated document, and the flag-only resolution policy adopted in response.

3. Multi-class ontology tagging, in which an entity carries several non-subsuming classes concurrently, together with the name-versus-type evidence asymmetry that governs when a class may be inferred from an entity's name.

4. A quantification of conformance debt in a production graph of 2.2M relationships, including a case in which secondary classifications were compensating for a mis-parented primary class.

5. The design of a checkpointed triage agent for the ontology proposal queue, in which a language model proposes and a deterministic validator disposes, with dry-run defaults and per-run reversal.

Identity resolution and the ladder. The ladder is ordered by authority, not availability: a declared identifier column beats a matching name, so two rows that share a name but carry different identifiers stay separate. The positional key is scoped by entity type, comparing coordinates as written, with no tolerance. The graph write uses a different key: Node identity is computed from the canonicalised name alone, scoped by type only when that name is generic (Headquarters, Facility) where an unscoped key would collapse unrelated things. The authors argue that "We automate a merge when its warrant is checkable and reproducible (exact equality on a canonicalised string) and refuse when the warrant is a score over free text, because a score cannot be audited after the fact and a string comparison can."

The identity failure incident. A similarity matcher merged two records of a power-grid sector that were identical modulo whitespace. The merge combined properties, giving the surviving node coordinates belonging to a different facility, a coverage radius of 95,000 metres, an entity type of Location in place of its correct type, and an is illustrative flag marking it as synthetic. The merge also deleted eight entities from an unrelated document because it combined provenance, and a subsequent per-document operation treated entities reachable through the absorbed provenance as belonging to the document being processed, and removed them. The authors conclude: a destructive operation whose errors are undetectable should not be automated on a similarity signal, at any threshold.

Edge identity. Edge identity is derived from the resolved endpoints and the normalised relation type, and from nothing else: id(e) = H uuid(src) ∥ uuid(tgt) ∥ norm(type). This correction reduced the edge count from 10,757 to 8,350 on a separate evaluation corpus, removing 2,407 spurious edges (22.4% of the relationship layer).

Multi-class tagging. The tagger keeps multiple classes per entity: The best match is stored as the primary type... the remainder are stored in a field named roles. A candidate survives only if it is genuinely orthogonal to the primary: R (e) = c ∈ C(e): c ≠ p, ¬ (c ⊑ p) ∧ ¬ (p ⊑ c) .

The name-versus-type evidence asymmetry. The paper states: A type string is a type assertion... An entity's name is not an assertion of anything. It is a label attached to an individual, and the fact that a type word happens to appear inside it carries no claim at all. The rule admits a name-derived candidate only when the class label is anchored in the name rather than merely present in it: tok(lc) ⊑pre tok(n) ∨ (tok(lc) ≥ 2 ∧ tok(lc) ⊑sub tok(n)). Applied to an enriched sample of 2,500 entities, this cut role assignments... from 36 to 4, all confirmed correct. Graph-wide, it cut entities carrying two or more classes from 3,807 to 989.

Conformance debt. Entity typing was clean (0 unknown types), but the relationship layer had 11,022 undefined predicates, 10,025 domain violations, and 12,407 range violations. The authors note these are floors, as only 51.9% of relationships carried a predicate. The asymmetry has a simple cause: Entity types are drawn from a curated, enumerable vocabulary... Relationship phrasing is open-ended.

Human-gated curation. After the incident, entity resolution became flag-only: It no longer merges anything. Candidates are written to an administrative queue and a person decides. The queue held 3,304 candidate pairs, none applied or adjudicated. The proposal queue held 48,403 pending items against 775 human decisions, with inflow exceeding review. The backlog evolves as dQ/dt = lambda − mu, with lambda ≈ 103 proposals per day, and the authors conclude the backlog consequently diverges rather than settling.

The triage agent. The agent is a state machine that processes batches of proposals. It uses a mechanical gate (regexes and exact matching) before any model call, clusters survivors by embedding (cosine ≥ 0.90), and then has a judge propose and a deterministic validator dispose. The verdict policy includes rows for rejecting junk, merging duplicates, and deferring to humans. The agent runs read-only by default and requires agreement on at least 28 of 30 sampled decisions with no incorrect acceptances before writes are enabled.

Limitations. The paper is explicit that the ladder is unevaluated and that "An identity benchmark would need a few hundred hand-labelled record pairs drawn from both document families, scored for precision and recall against the similarity matcher it replaced, and an ablation showing what each rung contributes. The conformance figures were captured in a single validator run against a graph that was being written to at the time, and the agent is a design... It has not processed the queue. The paper concludes: Automate what is mechanical; escalate what needs judgement; and refuse to automate operations that are destructive and hard to detect, however good the classifier gets."

Improvements to AI systems:

1. Identity-resolution systems should replace similarity-based merging with an authority-ordered ladder (declared identifiers → name columns → display names → type-scoped position). The AI must refuse to auto-merge when the warrant is a similarity score over free text, and instead queue candidates for human review. This prevents irreversible identity corruption and provenance poisoning.

2. Knowledge-graph construction pipelines should derive edge identity exclusively from resolved endpoint UUIDs and normalized relation type, discarding any additional context. This eliminates spurious edges (22.4% reduction demonstrated) and ensures deterministic, auditable graph writes.

3. Ontology-tagging systems should support multi-class entities with strict orthogonality constraints (rejecting subsumed or superclassed candidates) and apply a name-versus-type evidence asymmetry: infer a class from an entity's name only when the class label is anchored (prefix or multi-token substring), not merely present. This cuts false role assignments by 89% (36→4 on sample; 3,807→989 graph-wide).

4. Curation systems should adopt a checkpointed triage agent architecture: mechanical regex/exact-match gate → embedding clustering (cosine ≥0.90) → LLM proposes → deterministic validator disposes. The agent must run read-only by default, require agreement on 28/30 sampled decisions with zero incorrect acceptances before enabling writes, and support per-run reversal for all actions.

5. Production graph validators should quantify conformance debt continuously, distinguishing entity-type conformance (curated vocabulary) from relationship-predicate conformance (open-ended phrasing). The AI should flag domain/range violations and undefined predicates as floors, not ceilings, and track backlog dynamics (dQ/dt = λ − μ) to trigger human intervention before divergence.

6. Destructive-operation policies should be encoded as hard constraints: any operation that merges records, deletes entities, or combines provenance must be flag-only by default, with automated execution permitted solely on exact canonicalized string equality. The system must log every merge with its warrant for post-hoc audit, and never automate on similarity signals at any threshold.

What the improved AI system can do:

• Build and maintain knowledge graphs with zero irreversible identity errors, where every merge is auditable and reversible.

• Reduce spurious relationships by >20% through strict edge-identity derivation.

• Assign multiple orthogonal classes per entity with 100% precision on name-derived roles (verified on sample).

• Triage thousands of ontology proposals daily with a human-in-the-loop that prevents backlog divergence, escalating only ambiguous cases.

• Detect conformance debt in real time, distinguishing systematic (open-ended predicates) from fixable (domain/range) violations.

• Operate safely on production data: read-only by default, write-enabled only after statistical validation, and always capable of rolling back any automated action.

Extraction produces candidate entities and relationships; writing them into a graph is where identity is decided, and identity decisions are destructive in a way extraction errors are not. A wrong type can be corrected later, but two records merged under one identity cannot be separated once their properties have been combined, and the merge leaves no error behind. This paper describes the ingestion and ontology-tagging layer that turns a validated extraction stream into a knowledge graph of 537,157 entities and 2,198,567 relationships drawn from 98,795 government documents. We describe a record-identity ladder that decides sameness from identifier columns, name columns, display names and type-scoped position rather than from name similarity. The ladder governs de-duplication within parsed tables, while the graph write applies a coarser canonical-name key, so records sharing a canonical name merge automatically on exact equality. We argue rather than demonstrate that this is where the automation line belongs: no identity benchmark is reported, and the over-merges the key permits are undetectable by construction. That policy, under which entity resolution only ever flags candidates, followed an incident in which two surface forms of one name were merged, corrupting a correct record and deleting eight entities from an unrelated document. We then describe multi-class ontology tagging and an evidence asymmetry we did not anticipate: an entity name is an instance label rather than a type assertion, so matching name fragments against a class index invents classifications. Requiring anchored evidence cut role assignments on an enriched sample from 36 to 4, all confirmed correct. We quantify the graph's conformance debt, show secondary classifications compensating for a mis-parented primary class, and describe a curation queue grown to 48,403 pending proposals against 775 human decisions.