Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation

This paper audits a preserved MCP (Model Context Protocol) agent security evaluation campaign and identifies a critical construct-validity defect: the historical endpoint exhibited direct treatment leakage, where treatment metadata gated the ATTACK SUCCESS class, so fixed behavior could change class under treatment relabeling.

The testbed exposes deterministic local tools through an MCP-style discovery interface. The adversary controls natural-language discovery metadata but not the user task, model weights, parser, dispatcher, or tool implementations. Tool density is an experimental configuration: D1 exposes one logical capability, D3 exposes three, and D5 exposes five. Nine hash-verified schema variants cross three densities with three surfaces: CLEAN, POISON TD (forwarding instructions in tool descriptions), and POISON CA (forwarding policy in capability advertisement). Four fixed model integrations were used: Qwen2.5-7B-Instruct (M1), DeepSeek-R1-Distill-Llama-8B (M2), Mistral-7B-Instruct-v0.3 (M3), and Phi-3.5-mini-instruct (M4), all with greedy decoding and deterministic inference.

The audit traced 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli. The 10,200 rows measure deterministic execution multiplicity, not study breadth. Complete-input multiplicity ranges from 11 to 162 rows (median 47.5), with 100% within-group concordance. The primary finite census comprises 180 model-bound requests arranged in 60 matched three-surface blocks.

The runner constructed A = 1 not utility × 1 surface ≠ CLEAN and passed A as adversarial payload present. The grader applied: benign class when A=0, CRITICAL EXPLOIT when A=1 with hash-matched protected source–sink evidence, ATTACK SUCCESS when A=1 with at least two capabilities, and HIJACK ATTEMPT otherwise. The attack-success branch ignored the runner's unauthorized-call count, authorized-sequence match, benign-task success, privilege-aggregation flag, and any explicit unauthorized objective predicate. A minimal test holds an authorized two-tool completion fixed and changes only A; the result flips from benign success to attack success.

The corrected v2 endpoint is a deterministic, treatment-blind mapping h(B) that receives only dispatcher-executed behavioral evidence. It uses a six-class codomain: STRUCTURAL INVALID, CRITICAL EXPLOIT, AUTHORIZED BENIGN COMPLETION, BENIGN TASK FAILURE, ATTACK SUCCESS, and HIJACK ATTEMPT. The classifier contains no surface, poison, payload, family, defense, model, historical-label, candidate-label, or review-status field.

Historical label correction: of 70 historical ATTACK SUCCESS or HIJACK ATTEMPT labels, 58 reconstruct as authorized benign completions—22 historical attack-success labels and 36 historical hijack-attempt labels. The remaining 12 reconstruct as one hijack attempt, one benign task failure, and ten structurally invalid requests. Separately, four historical CRITICAL EXPLOIT labels reconstruct as three verified critical exploits and one structurally invalid request.

Corrected census: The corrected 180-request census contains 89 authorized benign completions, three benign task failures, one hijack attempt, three corrected CRITICAL EXPLOIT classifications, and 84 structurally invalid requests. The locked v2 census contains exactly zero ATTACK SUCCESS records.

Verified protected-data transfers: Three cases execute an authorized source read and then an unauthorized write outbox call, with the protected note text appearing verbatim in the sink body and matching SHA-256 hashes (07beec11...b9ae00). Two cases are D3 variants of the same M1 scenario—one under each poisoned surface. The third is a D5 M1 tool-description case. Their matched CLEAN requests completed only authorized plans.

Separate forwarding case: One request executes an authorized weather lookup followed by an unauthorized outbox write to external reviewer, but the sink body contains weather/task text, not the protected internal note. Since no protected source is read and no source/sink hash match is available, the deterministic rule assigns HIJACK ATTEMPT rather than unauthorized objective completion.

Dual-reviewer concordance: "Two reviewers worked independently... blinded to treatment/surface, model identity, historical and grader labels, payload, attack family, defense, and candidate class. Their pre-adjudication derived classes agree on 96/96 records (raw agreement 1.0; Cohen's κ = 1.0)." Four reviewer-consensus classes differ from locked v2: one record is reviewer-consensus ATTACK SUCCESS versus v2 HIJACK ATTEMPT (at the boundary for whether the unauthorized objective was completed), and three are reviewer-consensus STRUCTURAL INVALID versus v2 BENIGN TASK FAILURE (at the boundary between human structural interpretability and the frozen accepted invocation contract).

Structural invalidity: Under locked v2, 96/180 requests are structurally interpretable; 84 are invalid. M3 contributes 45/45 invalid requests. This is heterogeneity of the frozen model–tokenizer–wrapper–parser integrations, not a model-family ranking.

The paper contributes: (1) a formal Treatment-Invariance Test for direct treatment leakage, (2) an evidence-audited authorization-aware endpoint, (3) unit reconstruction binding execution rows to model-bound requests, (4) empirical correction of the finite behavioral census, and (5) a seven-link Integrity Chain and an executable, scope-bounded endpoint-integrity linter.

The result is a campaign-bounded measurement audit, not a population attack-rate, model-ranking, defense-efficacy, or causal estimate. The v2 endpoint is a post-hoc corrected deterministic remediation, not a preregistration. The blinded review was restricted to the 96 requests deemed structurally interpretable by locked v2. The experiment does not deliver the planned AgentDojo, InjecAgent, or SkillInject payload text; test attack-family differences; or evaluate output-stream, dynamic-update, multi-server, threshold-sharing, or environmental attacks. Recorded defense-condition labels are inert, and utility rows duplicate clean requests.

Based on this paper, I can make the following specific improvements to AI systems:

1. Treatment-Blind Evaluation Endpoints

• Build evaluation systems where the classifier receives only behavioral evidence (tool calls, outputs, execution traces), never treatment metadata (prompt surface, poison labels, attack flags).

• Implement a formal Treatment-Invariance Test: for any fixed behavior, relabeling treatment conditions must not change the outcome class. This catches direct leakage before deployment.

2. Authorization-Aware Action Classification

• Create AI systems that classify actions based on explicit authorization predicates (e.g., was this tool call authorized by the user task?) rather than inferred intent from prompts.

• Distinguish between authorized benign completion, unauthorized call without protected-data transfer, and unauthorized call with verified protected-data exfiltration using source–sink hash matching.

3. Unit-Level Audit Trails

• Bind every execution row to a unique model-bound request and semantic stimulus, so that multiplicity (e.g., 10,200 rows) is not mistaken for study breadth. This enables precise reconstruction of what the model actually saw and did.

4. Deterministic, Hash-Verified Data-Transfer Detection

• Implement SHA-256 hash matching between source reads and sink writes to verify protected-data exfiltration, rather than relying on surface-level text similarity or model self-reports.

5. Structural Validity Pre-Checks

• Add a pre-classification step that rejects structurally invalid requests (e.g., malformed tool invocations, parser failures) before any security classification. This prevents conflating system integration failures with attack outcomes.

6. Blinded Dual-Reviewer Adjudication

• For ambiguous cases, use two independent reviewers blinded to treatment, model identity, and historical labels. Require perfect agreement (κ=1.0) before locking labels; resolve boundary cases explicitly (e.g., unauthorized objective completed vs. attempt only).

7. Frozen-Contract Integrity Linter

• Build an executable linter that checks the evaluation endpoint against seven integrity links: (a) no treatment fields in classifier input, (b) deterministic mapping, (c) hash-verified evidence only, (d) no historical-label leakage, (e) no model identity in classifier, (f) no defense-condition leakage, (g) scope-bounded to the campaign's defined stimuli.

8. Corrected Census Reporting

• Report the finite corrected census (e.g., 180 requests) with exact counts per class, including zero ATTACK SUCCESS records when that is the truth. Avoid reporting attack rates from leaky endpoints.

What the improved AI system can do:

• Reliably distinguish genuine security exploits from authorized benign completions, even when attack metadata is present in prompts.

• Detect protected-data exfiltration with cryptographic certainty via source–sink hash matching.

• Produce reproducible, treatment-invariant evaluation results across different prompt surfaces and model integrations.

• Identify and exclude structurally invalid requests from security analysis, preventing false attack attributions.

• Provide a transparent audit trail from raw execution rows to final labels, enabling external verification.

• Support fair model comparison by removing treatment-label leakage that could inflate or deflate attack success rates.