On Reliability of Membership Inference Vulnerability Evaluation

Membership inference attacks (MIAs) are popular methods for empirically assessing data leakage, but reliably estimating their true positive rate (TPR), especially at low false positive rates (FPRs), requires excessive computational resources. This work addresses these limitations by demonstrating two key weaknesses in efficient MIA evaluation pipelines and proposing post-processing methods to calibrate the false positive rates across different samples, while also identifying a finite population bias in commonly used efficient likelihood-ratio attacks.

The gist

Evaluating TPR based on MIA scores concatenated across multiple individuals is not calibrated across per-sample FPRs, making it unreliable for auditing differential privacy; furthermore, the efficient likelihood-ratio attack suffers from a finite population bias in estimating in/out-distribution variances.

Vulnerability Evaluation Challenges and Context

The primary difficulty in evaluating privacy risk stems from three sources: the statistical nature of privacy risk requiring repeated observations, the difficulty in identifying maximally vulnerable samples, and uncertainty regarding whether an applied attack is optimal. Membership inference attacks are based on a binary hypothesis test: H0 (sample is not in training data) versus H1 (sample is in training data). The success of MIA is related to vulnerability to other attacks like differential privacy, which can be interpreted as a bound on MIA success. Evaluating the true positive rate (TPR) at fixed low false positive rates (FPRs) is most sensitive to an adversary's ability to detect a small number of true members, but reliably estimating these TPR values requires a large amount of membership data.

Concatenated MIA Aggregation Weaknesses

When evaluating MIA across multiple target models and points, scores are often averaged over these evaluations. This approach has two main weaknesses:

1. Concatenating MIA scores reports an average over per-sample vulnerabilities, but a single global threshold induces heterogeneous per-sample FPRs, which miscalibrates conclusions about privacy risk.

2. The global FPR from this process is usually not what would be expected, leading to misleading evaluations of the privacy risk.

Proposed Post-Processing for Calibration

To solve the calibration problem of concatenated approaches, a post-processing method is proposed to effectively calibrate the FPR across different samples. This involves transforming the in/out distributions using an affine transform:

1. Define a mean gap, denoted as ∆x = µin,x − µout,x.

2. Consider the affine transform fx(t) = sign(∆x)(t − µout,x).

3. This yields standardized in/out distributions: Sˆout x:= fx(S out x) ∼ N (0, 1), and Sˆin x:= fx(S in x) ∼ N ∆x σ2in,xσ2out,x !.

4. By Lemma 2.1, this standardization does not change the trade-off function T(P, Q)(α).

Finite Population Bias in Efficient LiRA

The efficient likelihood-ratio attack (LiRA) implementation suffers from a finite population bias because it samples training sets without replacement from a finite superset Dfull. This leads to estimators of the in/out variances being biased towards smaller values compared to independent draws from the underlying distribution P. The standard finite population correction factor is FPC = 1 − N/N+ (Equation 28). Not accounting for this bias leads to variance miscalibration that grows with the sampling ratio N/N+.

Experimental Findings and Conclusion

Experiments using TabPFN models on Adult and Credit datasets show that naive concatenation consistently overestimates the average privacy risk. Post-processed concatenation (Concatenated TPR (PP)) provides a calibrated, conservative aggregate approximation to Average TPR/Sample, especially as the number of models (M) increases. The Average TPR/Sample computation converges toward Concatenated TPR (PP) for large M, suggesting that the post-processed concatenated statistic is a reliable finite-sample proxy for average per-sample privacy risk. Furthermore, using Student’s t-distribution for the decision threshold can yield accurate results when M ≥ 256. The work provides a relatively cheap and easy alternative for evaluating MIA vulnerability as a valid TPR at known FPR, which is not available from previous methods.

References

M. Aerni, J. Zhang, and F. Tramèr (2024). Evaluations of Machine Learning Privacy Defenses are Misleading. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 1271–1284, 2024.

N. Carlini et al. (2022). Membership Inference Attacks From First Principles. In 43rd IEEE Symposium on Security and Privacy (SP), pages 1897–1914, 2022.

J. Dong, A. Roth, and W. J. Su (2022). Gaussian differential privacy.

As a fastidious researcher, I have analyzed this paper, On Reliability of Efficient Membership Inference Vulnerability Evaluation, and identified several critical areas for improvement in AI systems, particularly concerning privacy auditing and risk assessment.

Here are the specific improvements that can be made to AI systems based on this research:

The core improvement focuses on shifting from efficient but unreliable MIA evaluation metrics to calibrated and reliable ones, while simultaneously mitigating known biases in efficient attack implementations.

1. The deployment of a new, post-processing method for aggregating Membership Inference Attack (MIA) scores.

2. The integration of finite population correction factors into the estimation of model uncertainty during MIA evaluation.

3. The adoption of per-sample vulnerability assessment over simple concatenated aggregate metrics for privacy risk quantification.

By implementing these improvements, the resulting AI systems and auditing frameworks can achieve the following specific capabilities:

1. The system will provide a reliable estimate of the true False Positive Rate (FPR) and True Positive Rate (TPR) of membership identification, even in regimes where few observations are available (low FPR). This prevents overestimation of privacy risk when using techniques like Differential Privacy or when auditing models with tight privacy budgets.

2. The system will offer a more conservative and stable measure of overall model privacy risk by replacing naive score concatenation with post-processed concatenation methods (specifically, the Concatenated TPR (PP) approach). This ensures that the aggregated risk metric is not artificially inflated by heterogeneous per-sample score variances.

3. The system will be capable of distinguishing between the vulnerability of an entire dataset and the vulnerability of specific data points. This allows security teams to identify and focus on most vulnerable samples rather than relying solely on noisy averages, leading to more targeted defense strategies (e.g., identifying specific training examples that are highly sensitive).

4. The system will incorporate a bias correction mechanism (using the Finite Population Correction factor, FPC) when estimating model uncertainty from finite subsets of data. This ensures that the calculated vulnerability score is scaled appropriately to reflect the true underlying population distribution, preventing overly optimistic privacy risk assessments that occur when training sets are sampled without replacement from a finite pool.

5. The system will offer a more robust and accurate assessment of attack efficacy by identifying and correcting for finite population bias present in efficient likelihood-ratio attacks (like LiRA). This ensures that the reported vulnerability score accurately reflects the risk associated with using these specific, computationally efficient attack algorithms on real-world datasets, rather than being skewed by the finite nature of the sampled training data.

Membership inference attacks (MIAs) are popular methods for empirically assessing the leakage of sensitive information in the training data through models or statistics learned from the data. The MI vulnerability is often evaluated through a binary classifier that tries to predict whether a particular sample was in the training data. In order to evaluate the effectiveness of MIAs multiple shadow models are trained using random partitions of a larger dataset. After training the shadow models the MI vulnerability can be evaluated for all the samples for which we obtained shadow models. In order to evaluate the MI vulnerability reliably one needs a lot of shadow models which can be computationally infeasible. Therefore instead of reporting the actual sample level vulnerabilities aggregates over multiple samples are often reported in practice. We demonstrate two key weaknesses in typical MIA evaluation pipeline. First, we show that sampling the shadow datasets from a fixed superset leads to finite sample bias inflating the vulnerability estimates. Second, we show that evaluating the true positive rate (TPR) by concatenating MIA scores across multiple individuals, commonly used in the very low false positive rate (FPR) regime, is not calibrated across the per-sample FPRs. For both weaknesses we propose fixes that in the most simple approximate form do not incur any additional computation cost. We show that with additional computation one can further improve the reliability of the vulnerability estimation.