Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study

arXiv:2608.03172 · cs.AI, cs.LG · Submitted 2026-08-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study".

Jane: The paper was written by Qiming Bao, Sherry J. H. Feng, Kim Chester Eugenio and Meng Fon from Custodian Labs.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title and Authors: Tom: Welcome back to the show, everyone. Today we're digging into a paper that's got a mouthful of a title — "Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study." Jane, I gotta say, when I first read that title I had to read it twice.

Jane: You and me both, Tom. But once you unpack it, it's actually about something really practical. So "PHI" stands for protected health information — think patient names, dates of birth, medical record numbers. When hospitals share data for research, they have to strip that stuff out. The old way is to just replace it with a tag like NAME, which is safe but makes the text read like a robot wrote it.

Tom: And that's where the "surrogate substitution" part comes in. Instead of writing NAME, you replace "John Smith" with "Maria Lopez" — a fake but realistic name. The paper's authors, Qiming Bao and the team at Custodian Labs, call this structure-preserving de-identification. The text stays fluent, so downstream tools that were built for real clinical notes keep working.

Lu: What's clever about this paper is the question they ask. Everyone assumes a fake name is just as detectable as a real one — that a second-pass privacy tool will still flag it. But nobody had actually tested that assumption rigorously. The authors built a protocol to check whether the surrogate is as visible to detectors as the original PHI was.

Meng: And I appreciate that they didn't just test one detector. They ran eleven different ones — rule-based systems, fine-tuned models, and big LLMs. That's the kind of breadth that makes me trust the result, because if it only held for one architecture, it could just be a quirk of that model.

Jane: Right, and the result is surprisingly reassuring. On the spans that actually got masked, recall only dropped from seventy-six point one percent to seventy-four point nine percent — that's about one point. And they used a statistical test called equivalence testing to show that difference is effectively zero within a two-point margin.

Tom: So the title is basically saying: swapping real PHI for fake PHI doesn't hide the fake from other detectors. That's a big deal for anyone who wants to share clinical data without breaking the tools that analyze it.

Lu: It also reframes the conversation. The paper separates coverage — how much PHI you find and replace — from utility — whether the replacement is still detectable. Those are two different problems, and conflating them has muddied a lot of prior work.

Meng: I'll be honest, I came in skeptical. But the redaction floor experiment won me over. When they replaced PHI with asterisks instead of surrogates, the rule-based detectors collapsed to near zero recall. That shows the surrogate isn't just decoration — it's carrying real signal.

Jane: And that's the hook for our next segment, because the methodology is where this paper really shines. Stick around.

Summary: Tom: So we've established what the paper is about — "Surrogate Substitution Preserves PHI Detectability" — but let's get into how they actually proved it. Jane, walk us through the setup.

Jane: Picture this. You have a clinical document with real PHI in it. You run it through the transform, which swaps each sensitive value for a same-type fake. Then you take both versions — original and transformed — and run the exact same eleven detectors over both. Because the only thing that changed between the two is the substitution, any difference in detection has to be caused by the substitution itself.

Lu: That paired design is the backbone. It isolates the variable. And they were careful to only score the spans that actually got masked — so a detector that misses a lot of PHI doesn't accidentally make the transform look bad. Coverage and utility are measured separately.

Meng: The numbers are impressive. Fifty-seven thousand paired spans across seven benchmarks and seven languages. That's a lot of data. And the headline result — recall moving from seventy-six point one percent to seventy-four point nine percent — they tested that with TOST, which is the two one-sided tests procedure.

Tom: For our listeners who aren't statisticians, what does TOST actually do that a regular p-value doesn't?

Jane: Great question. With fifty-seven thousand samples, any tiny difference will be "statistically significant" under a normal test — even if it's practically meaningless. TOST flips the question. Instead of asking "is there a difference?", it asks "is the difference small enough to be considered equivalent?" They set a margin of two points, and the result came back equivalent with a p-value around three times ten to the minus nine.

Lu: And they showed the trap explicitly. The same data, run through McNemar's test, comes out "significant" — which would make you think the transform hurts. But the effect size is one point. That's the large-N trap in action, and this paper is a great demonstration of why equivalence testing should be standard in NLP evaluation.

Meng: What I found really convincing was the per-benchmark breakdown. The pooled result isn't hiding a disaster in one language. Five of the seven benchmarks sit inside the two-point margin. The two that don't — MEDDOCAN in Spanish and the Dutch PII set — need a three-point margin. Those are the dense, identifier-heavy non-English sets where surrogate generation is hardest.

Tom: So the residual loss isn't diffuse — it's concentrated. Which brings us to the error analysis, and that's where the paper gets really interesting.

Jane: Exactly. They found that the small drop in detectability isn't because detectors got worse at finding PHI. It's because some surrogates are just badly formed. We'll dig into that next.

Improvements: Tom: Welcome back. We're still on "Surrogate Substitution Preserves PHI Detectability," and we just heard that the overall effect is statistically equivalent to zero. But Jane, you teased that the error analysis tells a more interesting story.

Jane: Right. So they looked at the cases where a detector found the original PHI but missed the surrogate. About three thousand three hundred span-detector pairs out of over forty thousand. And they hand-coded those failures into a typology. The biggest culprit? Malformed surrogates.

Lu: Truncation and garbling. Things like "Chicago" becoming "Illino" — just cut off mid-word. Or "Ciudad de la Habana" becoming "Cuidad de la Havana" — a misspelling. When the surrogate no longer looks like a real name or place, detectors that learned lexical patterns just don't recognize it.

Meng: That's a generator problem, not a detector problem. And they proved it with the Faker baseline. They swapped in an open-source generator that produces clean, canonical values, and suddenly recall on those same hard benchmarks went right back up. The commercial transform's residual loss tracks its own generation quality.

Tom: So the fix isn't "make better detectors" — it's "make better surrogates."

Jane: Exactly. They also found a second failure mode: loss of salience. If you replace "Cedars-Sinai" — a famous hospital — with "Vidant" — an obscure one — weaker detectors lose the prior they had from pre-training. The surrogate is well-formed, but it doesn't trigger the familiarity signal.

Lu: And the third mode is actually privacy-positive. When they replace an email like "nachorutor@..." with "nxxxxxxxxx@...", the x-masking breaks the realistic token pattern, so detectors miss it. But that's the original value being destroyed — which is what you want for privacy, even if it costs a few points of recall.

Meng: The typology quantifies it nicely. Seventy-seven percent of surrogates are well-formed. Of the twenty-three percent with defects, truncation dominates at twelve and a half percent, x-masking at nine, and salience loss is rare at one and a half. And the well-formed rate tracks the recall rate almost exactly — detectors recover the good surrogates and miss the bad ones.

Jane: That's the key insight. The residual loss is a surrogate-generation problem, not a detection problem. Which means the improvement path is clear: fix the generator, don't retrain the detectors.

Lu: And that's a much more tractable engineering problem. The paper even shows that a cleaner open-source generator closes the gap on exactly the benchmarks where the commercial one struggles.

Tom: So what does this mean for the real world? That's where I want to take us in the final segment.

Conclusion: Tom: Alright, let's wrap this up. "Surrogate Substitution Preserves PHI Detectability" — we've covered the title, the method, and the error analysis. What's the big picture, Jane?

Jane: The big picture is that structure-preserving de-identification is viable. You can replace real patient names with fake ones, keep the text fluent, and downstream detectors will still find the fakes almost as well as the originals. That's a green light for sharing clinical data more broadly — for research, for model training, for analytics — without breaking the tools that need to work on that text.

Lu: And the methodological contribution is just as important. The paired multi-detector design, the masked-span scoring, the equivalence testing — this is a template for auditing any privacy transform. Anyone building a de-identification tool can now run this protocol and get a rigorous answer about whether their surrogates preserve utility.

Meng: From an engineering standpoint, the actionable takeaway is the typology. If you're building a surrogate generator, you know exactly what to fix: avoid truncation, keep salience, and decide deliberately about x-masking. The paper gives you a checklist, not just a score.

Tom: And the redaction floor experiment really drove it home for me. When they replaced PHI with asterisks, the rule-based detectors collapsed to near zero. That's the alternative — and it's much worse. Structure preservation isn't just nice-to-have; it's what keeps the data usable at all.

Jane: There are limitations, of course. Coverage is reported but not the focus — a transform can preserve utility on what it masks while under-masking overall. And the comparison experiments only ran four detectors, not all eleven. But the core claim — that detectability is preserved within a two-point margin — is solid.

Lu: The implications for healthcare AI are significant. If de-identified clinical text can flow into training pipelines without degrading downstream detection, that unlocks more data for research while keeping privacy protections intact. It's a win for both sides.

Tom: Alright, we've said our piece on "Surrogate Substitution Preserves PHI Detectability." Great paper, great protocol, and a clear path forward for the field. Thanks to everyone who tuned in — we'll see you next time with a fresh arXiv paper to dig into.

Qiming Bao, Sherry J. H. Feng, Kim Chester Eugenio, Meng Fon

Custodian Labs

cs.AI, cs.LG

Submitted: 2026-08-13

Updated: 2026-08-14

Comments: 12 pages, 3 figures, 10 tables. Code, data, and interactive dashboard: https://custodianai.pages.dev ; repository: https://github.com/Custodian-Labs/guardian-layer-phi-benchmark

Code: https://github.com/Custodian-Labs/guardian-layer-phi-benchmark

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 69/100

The gist: The paper "Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study" by Qiming Bao, Sherry J.

Key concepts

Protected Health Information (PHI): Sensitive patient data, such as names or medical record numbers, that must be removed from original clinical documents before they are used in research or shared with AI tools.
Surrogate Substitution: A method of de-identification where replacing real sensitive data is swapped for a realistic but fake value (e.g, swapping 'John Smith' with 'Maria Lopez'), ensuring the text remains fluent and usable.
Equivalence Testing (TOST): A statistical test used to determine if two results are essentially the same, even if a small numerical difference exists. It checks if the difference is within a pre-set margin, rather than just checking for any statistical significance.

Terminology

Benchmark — 48× in this paper · explained in 2 episodes
Decoupling — 6× in this paper · explained in 2 episodes
Association for Computational Linguistics — 5× in this paper
Core — 4× in this paper · explained in 2 episodes
Transformer — 4× in this paper · explained in 2 episodes
Data Augmentation — 3× in this paper · explained in 7 episodes
Failure Modes — 3× in this paper · explained in 3 episodes
Aggregation — 2× in this paper · explained in 1 episode
Alignment — 2× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Empirical Methods in Natural Language Processing — 2× in this paper
Adam — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes
World Model — 1× in this paper · explained in 7 episodes

Summary

The paper Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study by Qiming Bao, Sherry J. H. Feng, Kim Chester Eugenio, and Meng Fon (Custodian Labs) investigates whether structure-preserving de-identification—which replaces protected health information (PHI) with realistic same-type surrogates (Anna S. becomes Maria S., not [NAME])—preserves the detectability of those surrogates by downstream PHI detectors.

The authors state: Structure-preserving de-identification instead substitutes each PHI value with a plausible, same-type surrogate (John Smith → Maria Lopez), keeping the document readable and machine-parseable. They note that The promise of structure preservation rests on an unstated assumption: the surrogate carries the same downstream signal as the value it replaced.

The paper identifies three pitfalls in prior evaluation approaches: (1) Coverage confounds utility (a de-identifier masking 60% vs. 95% of PHI cannot be compared on whole-document F1); (2) The large-N significance trap (with tens of thousands of paired spans, any non-zero difference is statistically significant under a standard test, even when operationally meaningless); and (3) Aggregate error hides mechanism (a single −1.2 points says nothing about why the loss happens).

The contributions are: "(i) scoring utility only on masked spans, decoupling coverage from utility; (ii) using equivalence testing (TOST) rather than null-hypothesis significance testing, which is uninformative at our sample size (57k paired spans); and (iii) building a surrogate-failure typology separating fixable generator defects from intrinsic detector limits."

The experimental design uses a paired multi-detector evaluation protocol across 11 detectors, 7 benchmarks, and 7 languages (1,750 documents). The detector panel includes rule/statistical—Microsoft Presidio, OBI deid roberta, a RoBERTa tagger; open LLMs (local)—Gemma 4 31B / E4B, Qwen 3.5- 4B, 9B, 35B-A3B, Llama 3.1-8B / 3.3-70B, DeepSeek V2-Lite; frontier API—OpenAI GPT-5. The benchmarks are ASQ-PHI (English clinical queries), MEDDOCAN (Spanish clinical), MultiCoNER v2 (multilingual NER), and PII-Masking-300k in English, Dutch, French, German.

The core result: "Restricting to the 57,112 masked spans and pooling all 11 detectors... recall moves 76.1% → 74.9% (−1.2 pts; 95% CI [−1.5, −1.0]). The TOST equivalence test (∆ = 2) rejects non-equivalence at p ≈ 3 × 10−9: the change is statistically bounded within ±2 points of zero. Ranking is preserved; Llama 3.3-70B is essentially unchanged (∆F1 +0.003)."

Per-benchmark equivalence testing shows four benchmarks are equivalent within ±2 points; MEDDOCAN and PII-nl require ±3 (both dense, identifier-heavy, non-English—where surrogate generation is hardest); MultiCoNER has too few masked spans (220) to test.

The redaction floor comparison (C1) replaces masked values with ***** and finds: "Rule-based Presidio and the fine-tuned OBI tagger collapse to ≈0–4% everywhere... Open LLMs behave differently—they partly recover the masked span (Qwen-9B 6–58%, Gemma-31B 13–80%)—because a ***** run is itself a legible redaction cue. The open-surrogate baseline (C2) using Faker shows every detector retains high masked-span recall on the open surrogates—Presidio 78–98%, OBI 64–100%, Qwen-9B 77–100%, Gemma-31B 83–100% (family means 84–95%)—so detectability preservation is a property of well-formed, same-type substitution, not of one vendor or one detector."

The error analysis identifies three failure modes, all generator-side: (1) Malformed/truncated surrogates (largest cause): Chicago → Illino, El Paso → El, Ciudad de la Habana → Cuidad de la Havana; (2) Loss of salience: a canonical entity replaced by an obscure one (Cedars-Sinai → Vidant); (3) x-masking of IDs/emails: nachorutor@... → nxxxxxxxxx@.... A stratified 200-span sample shows "77.0% of surrogates are well-formed, same-type values; of the 23% with a defect, truncation/garbling dominates (12.5%...), followed by x-masking (9.0%), with salience loss rare (1.5%...); type-consistency is near-total (99.5%)."

The authors conclude: "Structure-preserving substitution does not hide well-formed PHI from downstream detection. Because the effect is equivalence-bounded within ±2 points and ranking-preserving across a heterogeneous 11-detector panel, a transform of this quality can be inserted ahead of detection/analytics pipelines built for real clinical text without materially degrading downstream detection."

Coverage is reported separately: it masks 80.9% of gold PHI on ASQ-PHI and 48.3% on MEDDOCAN, and less (≈26%) on general-domain NER/PII corpora. The coverage gap is mostly a replacement-step issue—≈77% of missed clinical identifiers were flagged by the transform's own detector but not substituted.

The paper releases the evaluation subsets, scoring code, and an interactive dashboard at https://custodianai.pages.dev and code at https://github.com/Custodian-Labs/guardian-layer-phi-benchmark.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems, along with what the improved system can do:

Improvement: Replace null-hypothesis significance testing (NHST) with Two One-Sided Tests (TOST) for any paired evaluation where sample size is large (e.g., >10,000 paired spans). The module computes a 90% confidence interval for the effect size and declares equivalence only if the interval lies within a pre-specified margin (e.g., ±2 points). It also sweeps margins (1, 2, 3 points) to report robustness.

What the improved system can do:

Avoids the large-N trap where trivial differences (e.g., −1.2 points) are flagged as statistically significant.
Produces a bounded, operationally meaningful claim: The change is within ±2 points of zero with a p-value (e.g., p ≈ 3×10−9).
Automatically reports McNemar’s test alongside TOST to demonstrate the contrast, so users see both the trap and the remedy.

Improvement: A scoring function that measures downstream performance only on the spans that a de-identification transform actually masks. It uses character-level alignment to re-project gold PHI spans onto the transformed text, then computes recall, precision, and F1 under three matching modes (exact, type, overlap). It reports leakage (1 − recall) separately from coverage.

Improvement: A rule-based + LLM-assisted classifier that labels each surrogate span into one of three failure modes: (a) malformed/truncated (e.g., Chicago → Illino), (b) salience loss (e.g., Cedars-Sinai → Vidant), (c) x-masking (e.g., nachorutor@... → nxxxxxxxxx@...). It also flags type-consistency (same-type substitution) and length-preservation.

Improvement: A comparison harness that runs the same detector panel on three conditions: (1) original text, (2) transformed text (commercial or custom), (3) redacted text (all masked values replaced with *****), and (4) open-source surrogate baseline (e.g., Faker). It reports masked-span recall for each condition per detector family.

Improvement: A function that runs TOST independently on each benchmark (e.g., ASQ-PHI, MEDDOCAN, PII-en, PII-nl, PII-fr, PII-de, MultiCoNER) and reports the required margin for equivalence. It flags benchmarks that need a wider margin (e.g., ±3) and those with too few masked spans to test (e.g., MultiCoNER with 220 spans).

Improvement: A module that compares the ranking of detectors (by F1) on original vs. transformed text. It reports Spearman’s rank correlation and flags any detector whose relative performance changes by more than a threshold (e.g., >2 positions).

Improvement: A dashboard that displays, per benchmark and per entity type: (a) coverage (fraction of gold PHI masked), (b) utility (masked-span recall retention), (c) leakage, (d) failure-mode breakdown, and (e) equivalence test results. It also includes a detection vs. replacement diagnosis (e.g., 77% of missed spans were detected but not substituted).

Improvement: A packaged pipeline (scripts, configs, and data subsets) that runs the full evaluation on any structure-preserving transform. It includes:

analyze equivalence.py (TOST + McNemar)
score masked spans.py (overlap/exact/type matching)
classify failures.py (typology classifier)
compare baselines.py (redaction + Faker)
A 250-document subset per benchmark (1,750 total) and a 120-document subsample for LLM-heavy comparisons.

Improvement: A metric that reports leakage (1 − recall on masked spans) as a primary privacy indicator, not just a secondary number. It also flags cases where x-masking (e.g., nxxxxxxxxx@...) is privacy-positive but counts against recall, so users understand the trade-off.

Improvement: A validation suite that runs the same protocol across 7 languages (English, Spanish, Dutch, French, German, plus two more from MultiCoNER) and 2 formats (free-text and JSON). It checks that the transform preserves non-PHI structure (e.g., JSON keys, abbreviations, foreign syntax) byte-for-byte.

Bottom line: The improved system is a rigorous, reproducible, multi-detector, multi-language evaluation framework that replaces misleading significance tests with equivalence bounds, isolates utility from coverage, attributes residual loss to generator defects, and provides a clear go/no-go decision for deploying any structure-preserving de-identification transform.

Abstract

Structure-preserving de-identification replaces protected health information (PHI) with realistic same-type surrogates -- "Anna S." becomes "Maria S.", not [NAME] -- so that clinical text stays fluent and downstream tools keep working. But this only helps if the substitution does not itself corrupt the signal those tools rely on. We ask a narrow, testable question: on the spans a de-identifier actually masks, can downstream PHI detectors still find the surrogate? We introduce a paired, multi-detector evaluation protocol that (i) scores utility only on masked spans, decoupling coverage from utility; (ii) uses equivalence testing (TOST) rather than null-hypothesis significance testing, which is uninformative at our sample size (57k paired spans); and (iii) builds a surrogate-failure typology separating fixable generator defects from intrinsic detector limits. Across 11 detectors, 7 benchmarks, and 7 languages (1,750 documents), recall on masked spans moves from 76.1% to 74.9% -- a change our equivalence test shows is statistically equivalent to zero within a +/-2-point margin (p 3e-9), with detector ranking preserved. The residual loss does not reflect detectors getting worse at PHI: it concentrates in malformed and out-of-distribution surrogates (truncation Chicago -> Illino, salience loss Cedars-Sinai -> Vidant). A redaction floor and an open-source surrogate baseline indicate the effect is a property of well-formed substitution, not of one tool. We release the evaluation subsets, scoring code, and an interactive dashboard at https://custodianai.pages.dev so the protocol can audit any structure-preserving transform.