Clone-Robust Weights in Metric Spaces: Handling Redundancy Bias for Benchmark Aggregation

arXiv:2502.03576 · cs.LG, cs.GT · Submitted 2026-02-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Clone-Robust Weights in Metric Spaces: Handling Redundancy Bias in Benchmark Aggregation".

Jane: The paper was written by Damien Berriaud and Roger Wattenhofer from ETH Zurich.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the show, everyone. Today we're diving into a paper with a title that's a mouthful: "Clone-Robust Weights in Metric Spaces: Handling Redundancy Bias in Benchmark Aggregation." Jane, what do you make of that title?

Jane: Tom, I love it. It's dense, but it's actually describing a really relatable problem. We've all seen a benchmark, like a test for AI models, that has fifty tasks, but thirty of them are basically the same thing with different names. That's the redundancy bias.

Tom: Exactly. And the authors, Damien Berriaud and Roger Wattenhofer from ETH Zurich, they're saying, look, if you just average all those tasks equally, you're letting the duplicates dominate the score. A model that's great at one specific thing could win just because that thing got cloned many times.

Jane: Right. And the "metric space" part is their clever solution. They're saying, don't just look at the tasks as a list. Put them on a map where the distance between them represents how similar they are. Then, you can see the clusters of clones.

Tom: So instead of a simple average, you're weighing each task based on its neighborhood on this map. If you're in a crowded area, you share your weight with your neighbors. If you're all alone, you get to keep more of it for yourself.

Jane: That's the core idea. They're calling it "clone-proof." The goal is to make the overall benchmark score resistant to someone just padding it with a bunch of near-identical tasks to game the results.

Tom: And that's a huge deal, right? Because these benchmarks are how we judge progress in AI. If they're easy to game, the scores don't really mean anything.

Jane: It's about making the evaluation more honest. And that's what we're going to dig into for the rest of the show. We'll get into the math, the practical side, and what this could mean for the future of how we build and use these tests.

Tom: Stick around, because this is a paper that could really change how we measure intelligence.

Summary: Tom: So, Jane, we've got the gist of the problem. Now, what's the actual solution these authors cooked up? What's the paper's main summary?

Jane: They formalize this whole idea with a set of rules, or axioms, that any good weighting function should follow. It's not just one magic formula; it's a list of properties that make the solution fair and robust.

Tom: And one of the big ones is that if you have two tasks that are almost identical, they should get almost the same weight. That's their "Uniform Clone Fairness" axiom. It sounds obvious, but it's surprisingly hard to achieve.

Jane: Right. And they also want the weighting to be continuous. That means if you tweak a task just a little bit, the weight shouldn't jump around wildly. It should change smoothly.

Tom: So no cliff edges. You can't have a task that's super important one day and then worthless the next just because a similar task was added.

Jane: Exactly. And to actually build a function that does all this, they use this really neat "local voting" idea. Imagine every point in the space around a task gets a vote, and it votes for all the tasks it can see within a certain radius.

Tom: So a point in the middle of a cluster is voting for all the clones, splitting its vote. But a point in a sparse area is only voting for the one task it's near.

Jane: You got it. Then you just tally up all those votes. The result is a weight for each task that naturally shares importance among similar ones. They prove that this specific construction satisfies all their axioms.

Tom: It's a really elegant way to think about it. You're not just looking at the tasks themselves, but at the space around them. It's like the space itself is telling you how important each task is.

Jane: And that's the big contribution. They've moved from a vague idea of "we should probably weight tasks differently" to a concrete, mathematically grounded framework for doing it.

Tom: I'm already thinking about the practical side. How do you actually compute this? That's what we need to tackle next.

Improvements: Tom: So the theory is solid, but we're on a radio show about practical things. Jane, what's the catch? How hard is it to actually calculate these weights?

Jane: That's the million-dollar question, Tom. The paper is very honest about this. The exact calculation is a nightmare. It involves figuring out the volume of the union of all these overlapping balls in the metric space.

Tom: Right, and with a hundred tasks, that's a hundred balls overlapping in a high-dimensional space. That's not something you can just do on a napkin.

Jane: Not exactly. In fact, they suspect it might be computationally impossible to do exactly, a problem that's likely #P-hard. So they don't even try. Instead, they propose a Monte Carlo method.

Tom: Monte Carlo, that's the random sampling approach, right? Just throw a bunch of darts and see where they land.

Jane: Precisely. Instead of calculating the exact volume, you randomly sample points inside the balls. For each sample, you see which tasks it's close to, and you give that sample's vote to all of them. After enough samples, the average vote share for each task converges to the true weight.

Tom: And they even give you the math on how many samples you need to be confident in the answer. They have a whole theorem about it.

Jane: They do. They show that to get within a certain error, you need a number of samples that scales with the square of the number of tasks. It's not trivial, but it's very doable.

Tom: So it's not a free lunch, but it's a lunch you can actually cook. And the improvement here is huge. They've taken a theoretically beautiful but practically impossible idea and made it usable.

Jane: Exactly. They've bridged the gap between the math and the real world. They even have a second algorithm that's more efficient if you only need the weight for one specific task.

Tom: That's great. So we have a framework and a way to compute it. But I'm still wondering, does this actually matter for the big AI models we see today?

First Page: Tom: Let's go back to the very first page of the paper, Jane. They open with this whole Morpheus bit about red and blue pills, and then they throw in indigo and navy and bordeaux. What's that all about?

Jane: I think it's a perfect metaphor, Tom. Morpheus offers Neo a choice between two distinct paths. But then the authors imagine a version where he also offers a bunch of shades of blue. It's a manipulation. It's trying to make the "blue" choice seem more important by sheer numbers.

Tom: So the colors are the tasks, and the shades of blue are the clones. And the question is, how do you make a fair choice when the deck is stacked?

Jane: Right. And they immediately connect this to a real-world example: the GLUE benchmark. That's a famous test for natural language understanding. It has tasks like CoLA and SST-two which are both about judging if a sentence is grammatically correct or positive or negative. They're very similar.

Tom: So in the GLUE benchmark, those two similar tasks are both counted fully, which gives that particular skill more weight than, say, a task about answering questions.

Jane: Exactly. And the authors argue that this is a bias baked into the benchmark. Their framework is a principled way to fix that. It's not just about picking arbitrary weights; it's about deriving them from the geometry of the task space.

Tom: And that's the real kicker. They're not just saying "here's a better weight." They're saying "here's a set of rules that any fair weight must follow, and here's a way to build one."

Jane: It's a philosophical shift. It forces you to think about what you actually mean by a "fair" benchmark. Is it fair to count every task equally, even if they're redundant? Or is it fair to give more weight to unique skills?

Tom: I think most people would say the latter. And this paper gives you the tools to actually do that.

Jane: And that's a powerful idea. It changes the conversation from "which tasks should we pick?" to "how should we weigh the tasks we have?"

Conclusion: Tom: We've covered a lot of ground on "Clone-Robust Weights in Metric Spaces." Let's wrap it up. Jane, what's the one thing you want our listeners to remember?

Jane: I think it's that the way we average scores in benchmarks is a choice, and it's often a bad one. This paper gives us a better choice. It provides a mathematical framework to ensure that a benchmark's score isn't skewed by a bunch of similar tasks.

Tom: And it's not just for AI benchmarks. The authors mention it could apply to voting advice apps, or even to figuring out how to weigh different data points in a machine learning model. It's a general tool for handling redundancy.

Jane: Right. The core problem is universal. How do you aggregate information when some of it is just a copy of other information? This paper gives a principled answer.

Tom: It's a smart, rigorous piece of work from the folks at ETH Zurich. They took a practical problem, built a solid theory around it, and even showed how to compute it in practice. That's a complete package.

Jane: Absolutely. And it's the kind of paper that could have a real impact on how we evaluate progress in AI. If we can make our benchmarks more honest, we can trust the results more.

Tom: Well said. That's all the time we have for this one. Thanks for joining us, and we'll see you on the next episode.

Jane: See you soon, everyone.

Damien Berriaud, Roger Wattenhofer

ETH Zurich

cs.LG, cs.GT

Submitted: 2026-02-16

Comments: Accepted at AAMAS'26

DOI: 10.65109/MJWJ6521

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 69/100

The gist: The paper addresses the problem of determining the relative importance of elements in a finite set within a metric space, where the distribution of elements is arbitrary and possibly adversarial.

Key concepts

Redundancy Bias
This is when a benchmark contains many tasks that are essentially the same, just with different names. If standard averaging is used, these duplicated tasks can disproportionately inflate a model's score, making it an unfair measure of true ability.
Metric Space
The authors map tasks onto a space where the distance between two points represents how similar those two tasks are. This allows them to identify clusters of 'clones' or highly related tasks, forming the basis for their weighting solution.

Terminology

Summary

The paper addresses the problem of determining the relative importance of elements in a finite set within a metric space, where the distribution of elements is arbitrary and possibly adversarial. The primary motivating application is multi-task benchmark aggregation, where a benchmark consists of tasks that map models to scores, and an aggregation rule combines task-wise scores into a single score. The key issue is that tasks may exhibit significant similarity (e.g., CoLA and SST-2 in GLUE), and the benchmark's outcome ought to remain unaffected by the inclusion of numerous highly similar tasks, as this could unfairly favor models that perform well on the original task over those excelling in other areas.

The authors propose a principled method for determining tasks' weights: the designer (i) settles on a relevant measure of similarity between tasks and embeds them in a metric space (E, d), and (ii) calculates the tasks' weights using a weighting function with desirable axiomatic properties.

A weighting function of a metric space (E, d) is defined as a function f that maps finite sets of E to probability distributions over their elements:

f: S ∈ P(E) ↦→ pS ∈ Δ(S),

where P(E) denotes the set containing all finite subsets of E (outside the empty set), and Δ(S) denotes the simplex over the elements of S.

The authors propose five axioms that weighting functions should satisfy:

Axiom 1 (Positivity): Every element of a finite set is represented with positive probability, i.e., for all finite subset S ∈ P(E) and element x in S, we have f(S)(x) > 0.

Axiom 2 (Symmetry): Elements of a set that are symmetric with respect to the metric are equally represented, i.e., for all finite subset S ∈ P(E) and self-isometry sigmaS: S ↦→ S, it holds for all x ∈ S that f(S)(x) = f(S)(sigmaS(x)).

Axiom 3 (Uniform Clone Fairness): Weighting is fair among approximate clones, i.e., for all epsilon > 0, there exists delta > 0 such that, for all finite subset S ∈ P(E) and x, y in S satisfying d(x, y) ≤ delta, it holds that f(S)(x) − f(S)(y) ≤ epsilon.

Axiom 4 (Uniform Individual Continuity): Weighting is element-wise continuous, i.e., for all epsilon > 0 and k ∈ N, there exists delta > 0 such that, for all finite subsets X, Y ∈ P(E) of cardinality X = Y = k such that dΠ(X, Y) ≤ delta, we have maxx∈X f(X)(x) − f(Y)(pi−1(x)) ≤ epsilon, where pi ∈ Π(Y, X).

Axiom 6 (Uniform alpha-Locality under Addition of Clones): The addition of a clone only changes the weights of points in the alpha-neighborhood of the clone, i.e., for all epsilon > 0, there exists delta > 0 such that, for each finite subset S ∈ P(E) and elements x ∈ S and x′ ∈ E S satisfying d(x, x′) ≤ delta, we have for all z ∈ S such that d(x, z) ≥ alpha that f(S)(z) − f(S ∪ x′)(z) ≤ epsilon.

The authors denote by Ralpha(E, d) the set of weighting functions on (E, d) satisfying Axioms 1, 2, 3, 4 and 6 with parameter alpha > 0.

The paper demonstrates that Axiom 5 (Class Continuity), which would require weights to be class-wise continuous when adding clones, is incompatible with Axiom 3. The authors construct an explicit example showing that a weighting function satisfying both Axioms 2 and 5 breaks Axiom 3.

The authors construct weighting functions based on a local voting scheme. For a fixed radius r > 0 and finite subset S ⊆ Rn, each element of the union of balls Br(S) is considered as a voter that approves only of the candidates in S close to him, spreading his voting power equally among them. The grade that each voter z in Br(S) attributes to a candidate x in S is:

gr,S,x(z) = 1Br(x)(z) / Σy∈S 1Br(y)(z)

The weighting function is then defined as:

gr(S): x ∈ S ↦→ ∫Br(S) [gr,S,x(z) / mu(Br(S))] dmu(z)

Theorem 1: For r > 0, the weighting function gr is well-defined and belongs in R2r(Rn, d2).

The proof verifies each axiom:

  • Axiom 1: Each weight is at least 1/S2

  • Axiom 2: Uses the fact that self-isometries on finite subsets of Euclidean spaces can be uplifted to full isometries (Lemma 10), and the Lebesgue measure is invariant under translations, rotations, and reflections

  • Axioms 3, 4, 6: Proved using geometric measure theory, particularly the (n−1)-dimensional Minkowski content, showing that the relevant differences are bounded by terms proportional to delta

Theorem 2: Let nu be a probability density function over [0, alpha]. Then the weighting function fnu: S ∈ P(Rn) ↦→ ∫0ɑ nu(r)gr(S) dr belongs in R2ɑ(Rn, d2).

The paper introduces Axiom 7 (Exact Computability) requiring that weighting be efficiently computable, but notes that the proposed weighting functions are unlikely to meet this criterion since computing gr(S)(x) would a priori involve averaging over as many as O(2S) disjoint cells.

Algorithm 1 provides a naive Monte Carlo estimation of gr(S). The algorithm samples points uniformly from each ball, computes the local depth (number of balls containing the sample), and averages the reciprocals.

Theorem 3: Algorithm 1 yields a consistent and asymptotically unbiased estimate of gr(S). Setting the number of samples to satisfy:

k ≥ (S2 − 1)2/(2epsilon2S2) · ln(2S/delta)

guarantees with probability at least 1 − delta that ĝr(S)(x) − gr(S)(x) ≤ epsilon.

Algorithm 2 uses the ApproxUnion algorithm from [7] as a subroutine to directly approximate the volume of the union of balls, with median amplification for confidence.

Algorithm 3 estimates fnu(S) by sampling radii from nu and averaging the corresponding estimates.

Theorem 4: For any target accuracy epsilon > 0 and confidence level delta ∈ (0, 1), setting the outer and inner sample sizes in Algorithm 3 such that:

M ≥ (8/epsilon2) ln(4S/delta), k ≥ (2(S2 − 1)2/(epsilon2S2)) ln(4SM/delta)

ensures with probability at least 1 − delta that f̂nu(S)(x) − fnu(S)(x) ≤ epsilon for every x ∈ S.

Algorithm 4 is introduced as a sample-reuse strategy across radii, potentially reducing the total expected runtime to O(kS(M + Sn2 log M)) for large S.

The framework extends to pseudo-metric spaces where two different elements may be perfect clones (distance zero). The axioms directly extend, and the representation functions fnu in Theorem 2 remain valid when the space induced by the vanishing of the pseudo-metric is Rn.

For general metric spaces, the authors note that using a Radon measure mu, one could define the weighting functions gr in full generality and show that Axioms 1, 3, 4, and 6 hold. The challenge is satisfying Axiom 2, which relies on two properties of Euclidean spaces: the uplifting of self-isometries to the entire space, and the invariance of the Lebesgue measure under translations, rotations, and reflections. For general metric spaces, one would need uniformly distributed measures (giving the same weight to all balls of the same radius), which are very rigid objects uniquely defined up to a multiplicative constant in most metric spaces (Lemma 1 from [13]).

The authors propose Axiom 8 (Topological Invariance) as a potential solution: weighting should only depend on the distance matrix associated with each finite set, not on the topological properties of the space.

The paper compares its approach with the Voronoi weighing function proposed in [44]:

V(S): x ∈ S ↦→ ∫E [1x∈piS(z) / mu(E)] dmu(z)

The authors show that the Voronoi weighting function:

  • Verifies Axiom 1 but elements may receive arbitrarily small weight (contrasting with the 1/S2 lower bound for gr)

  • Does not satisfy Axiom 2 as stated (only for full isometries on the entire space)

  • Violates Axiom 3 (discontinuous in each perfect clone)

  • Fails Axiom 4 (continuity is not uniform)

  • May satisfy Axiom 6, though the extent remains unclear

The paper discusses applications beyond benchmark aggregation:

  1. Machine learning: Tackling class imbalance in multi-label classification by giving weights to individual sample contributions to the loss; distribution-agnostic importance sampling

  2. Distributed systems: Mitigating Sybil attacks by offering tools to regulate the influence of Sybils once detected

The paper also connects to related work in domain adaptation, samples reweighting, metric learning, and hierarchical clustering.

Improvements for AI systems

Based on the paper, here are specific improvements I can implement in AI systems, along with what the improved system can do.

Improvement: Replace the standard arithmetic mean (or arbitrary weighted mean) in multi-task benchmarks (e.g., GLUE, SuperGLUE, WILDS) with the clone-robust weighting function f nu(S) = integral 0 alpha nu(r) g r(S) dr, using a Euclidean embedding of tasks (e.g., via Task2Vec or Wasserstein task embeddings).

What the improved system can do:

  • Automatically down-weight tasks that are near-duplicates (e.g., CoLA and SST-2 in GLUE), preventing benchmark creators from inflating scores by adding many similar tasks.

  • Ensure that adding a new task that is 95% similar to an existing one does not disproportionately shift the leaderboard, while still benefiting from genuinely new tasks.

  • Provide a principled, automatic weighting that requires no manual tuning, unlike current practice where weights are chosen arbitrarily by the benchmark designer.

  • Guarantee that every task retains positive weight (Axiom 1), so no task is ever completely ignored.

  • Ensure that small perturbations in task definitions (e.g., adding a few test examples) do not cause large ranking changes (Axiom 4).

Abstract

We are given a set of elements in a metric space. The distribution of the elements is arbitrary, possibly adversarial. Can we weigh the elements in a way that is resistant to such (adversarial) manipulations? This problem arises in various contexts. For instance, the elements could represent data points, requiring robust domain adaptation. Alternatively, they might represent tasks to be aggregated into a benchmark; or questions about personal political opinions in voting advice applications. This article introduces a theoretical framework for dealing with such problems. We propose clone-proof weighting functions as a solution concept. These functions distribute importance across elements of a set such that similar objects (``clones'') share (some of) their weights, thus avoiding a potential bias introduced by their multiplicity. Our framework extends the maximum uncertainty principle to accommodate general metric spaces and includes a set of axioms -- symmetry, continuity, and clone-proofness -- that guide the construction of weighting functions. Finally, we address the existence of weighting functions satisfying our axioms in the significant case of Euclidean spaces and propose a general method for their construction.

Sources

Related papers