Privacy in Personalized AI Is a System Property, Not Just a Model Property

Individual model- or componentlevel analyses may not capture all privacy risks arising in personalized AI systems, motivating a system-level perspective on privacy.

The gist

Privacy in personalized AI should be treated as a property of the system as a whole, with the full user–system interaction and its information flows across components and over time as the unit of analysis.

Personalization Expands the Privacy Surface

Personalized AI systems expand the privacy surface because system behavior depends on information collected, inferred, and reused about users. Unlike standalone models, these systems may accumulate information across interactions, maintain explicit or implicit user representations, retrieve information when needed, route information through external tools, and adapt its outputs accordingly. Privacy risks arise not only from what is directly observed but also from what the system accesses, infers, exposes through its behavior, and combines over time.

Privacy-Risk Channels In Personalized AI

The paper distinguishes four interconnected privacy-risk channels to characterize how privacy loss manifests:

1. Data-access leakage: This occurs when private information available to the system may be accessed or transmitted beyond what is necessary, expected, or authorized for the current task. Examples include an assistant retrieving medical information or a shopping agent transmitting a home address to an external tool.

2. Inferential leakage: A system can infer sensitive information by combining patterns, such as a recommender system inferring political interests even if the user never declared them. This becomes privacy-relevant when the system derives knowledge beyond what is necessary or reasonably expected for the intended task, particularly when that knowledge is retained, reused across contexts, or exploited against the user’s interests or expectations.

3. Behavioral leakage: This concerns what others can learn from personalized outputs and actions. Rankings, recommendations, explanations, advertisements, or other actions may all expose private characteristics of the user, such as a feed recommending content associated with a political movement.

4. Compositional leakage: Privacy loss emerges when multiple components, contexts, or interactions are considered together. An observer might correlate information across contexts—for example, combining a calendar entry and an earlier conversation to reveal that a user is undergoing medical treatment.

Requirements for System-Level Privacy Evaluation

To guide system-level privacy evaluation in future privacy audits of personalized AI, the paper proposes four requirements:

1. Evaluate interaction trajectories, not only individual responses: Evaluations should include longitudinal user histories and allow adaptive probing based on prior outputs because several individually innocuous responses may jointly reveal sensitive information that no single response exposes.

2. Audit internal information flows, not only final outputs: Audits must examine which information is retrieved, which tools or agents receive sensitive data, and what is retained afterward, focusing on whether access is necessary, authorized, and appropriate for the current task and context.

3. Do not neglect inferential and behavioral leakage: Evaluations should assess what sensitive attributes the system can infer from user behavior as well as what an external observer can learn about the user from the system’s personalized outputs or actions.

4. Assess privacy jointly with personalization utility: Evaluation must consider utility alongside privacy loss, recognizing that unrestricted access to user data may improve personalization while increasing risks across the four channels, and connecting this to data minimization: the relevant question is not only whether information is protected, but whether the system needs to access or retain it in the first place.

Discussion

The framework complements model-level analyses by describing how privacy loss manifests rather than focusing on mechanisms that trigger it. The four channels are interconnected; for instance, a private medical event can trigger data-access leakage, which leads to inferential leakage, which alters behavioral leakage, and is ultimately exposed through compositional leakage. System-level audits must be precise: the model is differentially private, the model did not memorize the user’s data, and the system protects the user’s privacy are distinct statements. The paper stresses that personalization creates interconnected risks across components and over time, which isolated component evaluations cannot capture. Formal guarantees like differential privacy remain valuable but do not characterize risks arising when sensitive information is introduced through profiles or external tools, which is the scope of system-level audits.

Conclusion

The paper argues that privacy in modern personalized AI applications cannot be reduced to a model property and proposes a cohesive basis for system-level privacy evaluation by analyzing four interconnected leakage channels and proposing four corresponding audit requirements. These contributions aim to provide a systematic approach for evaluating the complex privacy risks inherent in personalized AI.

The gist

Privacy in personalized AI should be treated as a property of the system as a whole, with the full user–system interaction and its information flows across components and over time as the unit of analysis.

How it works

Personalized AI systems expand the privacy surface because "system behavior depends on information collected, inferred, and reused about users.

Based on the research presented in Privacy in Personalized AI Is a System Property, Not Just a Model Property, here are specific improvements for building more robust and privacy-aware personalized AI systems:

The proposed system-level privacy framework suggests moving beyond isolated model audits to evaluate the entire user-system interaction. The following improvements target the four identified risk channels (data-access, inferential, behavioral, compositional leakage) through the proposed four requirements for system evaluation.

1. [Requirement: Evaluate interaction trajectories]

2. [Requirement: Audit internal information flows]

3. [Requirement: Do not neglect inferential and behavioral leakage]

4. [Requirement: Assess privacy jointly with personalization utility]

Here is what the improved AI system can do, categorized by the improvement focus:

1. [Based on Requirement 1 (Interaction Trajectories)] The system will be capable of Adaptive Privacy Probing.

2. This means when a user asks a series of seemingly innocuous questions over multiple turns (e.g., discussing travel plans, then health concerns, then political views), the system will monitor the accumulation of these interactions to detect patterns that constitute compositional leakage. If the sequence reveals sensitive information (like medical treatment) that no single response would expose, the system will pause and request explicit user consent before proceeding with any subsequent personalized action based on that inferred knowledge.

3. [Based on Requirement 2 (Internal Information Flows)] The system will implement a Contextual Data Access Gate.

4. This means the system will not simply output information; it must log and justify every piece of data retrieved from user profiles, external tools, or memory components before use. If the system retrieves a sensitive item (e.g., home address for a shopping request) but does not explicitly state in its internal flow logs that this retrieval is necessary for the current task's objective (and authorized by policy), it will halt and require explicit authorization from the user or an administrator before transmitting or utilizing that data, preventing unauthorized data-access leakage.

5. [Based on Requirement 3 (Inferential and Behavioral Leakage)] The system will incorporate a Sensitivity Guardrail for Inferences.

6. This means the system will be trained not just to provide accurate recommendations but also to be calibrated against the risk of inferring sensitive attributes (like political interests or health status). If the personalization engine attempts to derive an attribute that is deemed highly sensitive (e.g., inferred political affiliation), it must flag this inference as a potential inferential leakage and refuse to use that inferred knowledge for further personalization unless a higher level of explicit user consent is provided specifically for that type of inference.

7. [Based on Requirement 4 (Privacy-Utility Trade-off)] The system will employ Utility-Aware Data Minimization.

8. This means the system will be architected to dynamically adjust its level of data access and inferential scope based on the immediate task's utility requirement versus the associated privacy risk. For example, if a user asks for a simple restaurant recommendation (high utility, low risk), the system will only use explicitly provided preferences and avoid accessing long-term interaction history. If a more complex task requiring deeper memory retrieval is requested, it will automatically trigger a higher privacy check to ensure that the necessary information is accessed minimally and that any resulting behavioral leakage is constrained to the immediate context of that task.

In personalized AI applications, such as conversational assistants and recommender systems, users interact not with models in isolation but with broader systems that access, infer, and reuse user information across components and over time. While such use of user information is integral to personalization, it also raises important privacy questions. In this paper, we argue that individual model- or component-level analyses may not capture all privacy risks arising in such systems, motivating a system-level perspective on privacy. We distinguish and analyze four interconnected privacy-risk channels in personalized AI, and subsequently propose four requirements for system-level privacy evaluation, covering interaction trajectories, internal information flows, indirect leakage, and the privacy-utility trade-off. We argue for their systematic incorporation into privacy audits of personalized AI.