Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning".

Elias: Here's the gist of the paper. The gist: This research presents a first theoretical and empirical analysis of a hidden privacy risk in logit-based FL methods, specifically that a semi-honest server may learn clients’ private models from logits during training.

Nadia: Quantifying the Privacy Risk The paper introduces the Adaptive Model Stealing Attack (AdaMSA) to quantify this risk by leveraging historical logits during training.

Elias: AdaMSA adaptively steals the private model by approximating its intermediate training states from prior iterations during training.

Nadia: The target logit is formulated as pˆT = ∑t=T−T0 t Z · p t, where T0 is the threshold governing the extent of past predictions to be considered, and Z = 1/∫T k=T − T0 k is a normalization factor that ensures the weights w t = 1/Z sum to 1.

Title and authors: Nadia: So we're looking at this paper called "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning." It sounds like they are tackling a specific weakness in how these methods share model outputs, which are called logits, instead of just sending model weights or gradients.

Elias: Exactly. The title tells us they have two main goals: first, to quantify this hidden privacy risk, and second, to develop a defense against it. It points out that logit-based FL methods are different because they share outputs on public data during training instead of raw data itself, and the paper argues that this difference introduces a risk.

Nadia: Right. So what's the core problem they're pointing out here? Is it just a general privacy issue, or is it tied specifically to these logit-based approaches?

Elias: It’s specific to semi-honest servers, which is an adversary who follows the protocol but tries to learn something private from the information they observe. The paper argues that this server can actually learn clients' private models just by looking at the logits being sent during training, which is a risk that has been overlooked.

Priya: From my side, I wonder what kind of information an adversary could actually learn from just the logits. Does it give them direct access to the private data, or is it something more abstract?

Nadia: It’s about approximating the private model itself through these transmitted logits. The paper introduces an attack called Adaptive Model Stealing Attack, or AdaMSA, to make this risk concrete and measurable.

Elias: AdaMSA works by using historical logits from previous training iterations to create a better target logit for the attacker's model. They formulate this target logit as pˆT equals sum t from T minus T0 t Z times p t, where T0 is a threshold governing how much history they look at.

Nadia: That sounds like they’re trying to build an attacker that can mimic the victim model by looking back through time during training. And what does this attack actually do in practice?

Elias: The goal of AdaMSA is to minimize the distance between the attacking model's output and a target logit, which they formulate as minimizing LCE between pˆT and p˜T using gradient descent to train an attacking model, denoted as ˜theta.

Priya: So, if I understand this correctly, they are saying that by looking at all those past logits—the historical ones—an attacker can build a very good approximation of the private model's state during training. That’s a lot of information to extract just from the outputs.

Nadia: Precisely. And then they quantify how bad this risk is by measuring the accuracy of that attacking model on some private test dataset, which they call R, where R is evaluated as E x∼Dk Acc f (x, ˜theta).

The paper's summary: Nadia: So we talked about what AdaMSA is. Now let’s look at the broader picture the authors are presenting in this paper. They lay out this framework to show that logit-based FL has a hidden privacy risk, and then they offer a concrete way to measure it and fix it.

Elias: The paper summarizes the situation by first setting up a theoretical analysis of the risk bound in logit-based FL, which involves looking at how empirical risks of mixed datasets relate to private datasets. They introduce Theorem three point four which bounds that difference between empirical risks by a term involving alpha and lambda, where lambda is defined as the sum of the empirical risks on both public and private datasets for a specific hypothesis h*.

Nadia: That seems like they’re trying to establish some mathematical boundaries on how much privacy you can expect when mixing public and private data in this context. So, what does that bound actually mean for anyone using these systems?

Elias: It means that the gap between the performance on a mixed dataset and the performance on the truly private dataset is controlled by a term (one minus alpha) times something involving D H, which is related to the difference in data distributions between public and private datasets <ref:2606.08252#pg1>. And this bound goes to zero when alpha approaches one.

Priya: That's interesting because it suggests that if you use more of the private data in your mix, the performance gap closes down mathematically. But what about the practical implication for a user who is just trying to keep their model safe?

Nadia: Well, practically speaking, this theoretical analysis gives them a way to understand how mixing data affects privacy guarantees under Rényi Differential Privacy. It shows that if you use an ensemble result computed with perturbed outputs, it stays (alpha, one/ki)-Rényi differentially private for any alpha greater than the maximum i alpha i <ref:2606.08252#pg1>.

Elias: And they link this to the sensitivity Delta squared of a function f. The composition property of Rényi Differential Privacy algorithms ensures that if you use perturbed outputs, the ensemble result remains (alpha, one/ki)-Rényi differentially private if gamma is greater than or equal to two times the natural log of one point two five divided by delta times Delta squared epsilon squared <ref:2606.08252#pg1>.

Priya: So they’re giving us a concrete condition on that perturbation scale gamma that needs to be met to maintain a certain level of privacy, and it depends on the sensitivity of the function f. That’s something you have to calculate beforehand.

Nadia: It sounds like they are moving from just identifying a risk, to providing the math needed to control it through data mixing and perturbation techniques. This sets up their next step, which is building a defense strategy.

The paper's improvements: Elias: Now they propose the defense mechanism called Federated Logit Perturbation, or FedLP. This strategy involves perturbing the transmitted logits of local models to defend against that model stealing risk we discussed earlier. They aim to prevent an adversary from replicating the functionality of the private model while keeping training performance high.

Nadia: The defense is framed as a constrained optimization problem where they want to maximize maxp′ G(thetaT−one p′) − G(thetaT−one p)a subject to the constraint that the distance between original and perturbed logits is bounded by some non-negative constant gamma <ref:2606.08252#pg1>.

Elias: The heuristic solver they suggest is quite simple: they just perturb the logit by setting p′ equals p plus Z times e j, where e j is a one-hot vector with a one in the j-th dimension of p and zeros everywhere else <ref:2606.08252#pg1>.

Nadia: That sounds like a very specific, localized change to the logit—basically picking one dimension and adding a small value based on some normalization factor Z. Is that how simple it is for an attacker to bypass this?

Elias: The paper shows that this perturbation based defense strategy preserves (epsilon, delta)-DP if and only if gamma is greater than or equal to two times the natural log of one point two five divided by delta times Delta squared epsilon squared <ref:2606.08252#pg1>.

Priya: So the defense depends on a specific mathematical relationship between the noise level you introduce, which is controlled by gamma, and how sensitive the function f is. If you choose a small enough gamma that satisfies that inequality, you get your privacy guarantee.

Nadia: It sounds like they are providing a way to tune this trade-off between utility and privacy directly by setting that specific threshold for gamma based on the desired epsilon and delta values. That’s useful because it moves beyond just applying generic noise.

Elias: And they mention that the composition property of Rényi Differential Privacy algorithms means you can use these perturbed outputs in an ensemble calculation, and the resulting result stays (alpha, one/ki)-Rényi differentially private for any alpha greater than the maximum i alpha i <ref:2606.08252#pg1>.

Priya: So what we have here is a mechanism where we take a set of perturbed logits and combine them, and as long as the perturbation scale gamma is set correctly according to those parameters, the final result maintains that privacy level. That’s a useful way to think about maintaining privacy across multiple clients.

Conclusion: Nadia: So we've gone through AdaMSA and FedLP. To wrap up this paper "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning," the main point is that logit-based FL methods have a hidden privacy risk where a semi-honest server can learn private models from logits.

Elias: And they gave us tools to measure this risk with AdaMSA and then a concrete defense, FedLP, which uses logit perturbation to keep the utility high while bounding the perturbation scale by gamma.

Nadia: They showed that you can tune that gamma based on your desired privacy parameters epsilon and delta, providing a clear way to balance utility and privacy in these systems. This is important because it addresses a vulnerability specific to models that share logits during training.

Priya: I just want to say that the paper shows how important it is to be precise about what the data actually represents before trusting the claimed results, especially when dealing with these privacy guarantees under Rényi Differential Privacy, where you have to be careful with those parameters.

Elias: It’s a lot of work establishing these theoretical bounds, but they provide a framework for anyone working in this area who is looking to understand how to actually control the distance between private and public data using mixed datasets.

Nadia: This work on "Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning" gives us a solid starting point for understanding these new privacy vulnerabilities that are specific to logit-based FL.