Half-Truths Break Similarity-Based Retrieval

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Half-Truths Break Similarity-Based Retrieval".

Jane: Here's the gist of the paper. The gist The CS-CLIP method introduces unit-level supervision during fine-tuning to address half-truth vulnerabilities in CLIP-style models, achieving 69.3% Half-Truth Accuracy and improving compositional understanding by 5.7 percentage points across established benchmarks.

Tom: How it works The paper identifies a "half-truth vulnerability" where appending a plausible but incorrect detail to a correct description can increase the similarity score in CLIP-style dual encoders, which is formalized as the half-truth vulnerability <ref:2602.23906#pg2>On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation <ref:2602.23906#pg2>This vulnerability is traced to weak supervision on caption parts because contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded <ref:2602.23906#pg2> The proposed solution, CS-CLIP (ComponentSupervised CLIP), decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference <ref:2602.23906#pg2>For each caption T, the pipeline extracts entity units E(T) and relation units R(T), and generates matched foils U˜ for each unit via minimal edits <ref:2602.23906#pg2>The training objective combines a global sentence-level loss (Lglobal) with a unit-level loss (Lunit), where Lunit trains the image embedding to score the correct unit Ui,k higher than its matched foil U˜i,k and in-batch units from other images <ref:2602.23906#pg2>The final objective is defined as LCS = Lglobal + λu Lunit, which keeps standard CLIP inference unchanged at test time but makes the learned similarity more sensitive to caption structure by training the model to distinguish correct caption units from their matched foils <ref:2602.23906#pg2> Diagnostic and Results The half-truth diagnostic tests whether adding one incorrect detail increases similarity, defined as AccHT = Pr(s(I, A) > s(I, A−)) <ref:2602.23906#pg2>Figure 3 reports AccHT when the added detail is based on either an entity unit or a relation unit <ref:2602.23906#pg2>For standard CLIP, performance is close to chance for entity additions (52.9%), and drops below chance for relation additions (32.9%), meaning incorrect relation details are frequently preferred over the anchor <ref:2602.23906#pg2>CS-CLIP achieves 69.3% Half-Truth Accuracy on COCO, compared to 40.6% for CLIP and 56.5% for NegCLIP <ref:2602.23906#pg2>Relation additions are harder to penalize than entity additions across models, with CS-CLIP reaching 65.5% AccHT on relation additions with a positive similarity gap (∆ = +0.014) <ref:2602.23906#pg2> Compositional Understanding and Benchmarks CS-CLIP improves on 16 established compositional benchmarks by 5.7 percentage points, achieving the best average Image-to-Text accuracy (57.8%) and Group Accuracy among evaluated models <ref:2602.23906#pg2>Figure 6 shows CS-CLIP achieves consistent positive gains across most benchmarks, rather than improvements driven by a small subset <ref:2602.23906#pg2>The capability breakdown reveals systematic weaknesses across all models, with CS-CLIP achieving the best Role Sensitivity (40.1%), substantially outperforming CLIP (34.6%) and matching or exceeding all other COCO-trained methods <ref:2602.23906#pg2> Downstream Performance CS-CLIP achieves the best average T2I Recall@1 (71.7 vs NegCLIP 65.9) on COCO and Flickr8k, indicating that compositional improvements directly benefit retrieval tasks where fine-grained matching matters <ref:2602.23906#pg2>In zero-shot classification, CS-CLIP achieves 59.9% Acc@1 and 84.6% Acc@5 on ImageNet variants and standard recognition datasets, which is comparable to other COCO-fine-tuned baselines <ref:2602.23906#pg2>The paper concludes that unit-level supervision substantially reduces half-truth vulnerability, achieving the highest overall AccHT (69.3%) and a consistently positive similarity gap <ref:2602.23906#pg2> Ablations Table 4 shows that adding matched unit foils (Variant 4: units + foils without global negatives) produces the largest Half-Truth improvement (All 68.3; Rel 63.7), directly targeting the rejection of single incorrect additions <ref:2602.23906#pg2>The fraction of relation-type units (p) has a clear trade-off: using only entity units (p=0.0) achieves the highest entity accuracy (78.7) but poor relation accuracy (55.2), while higher p improves relation understanding at the cost of entity performance <ref:2602.23906#pg2>The authors use LR= 5×10−6 as it balances Half-Truth gains (69.3) with competitive downstream performance (59.9 zero-shot, 71.7 retrieval) <ref:2602.23906#pg2>The batch size BS= 128 is used as it achieves the best retrieval performance while maintaining competitive Half-Truth and zero-shot accuracy <ref:2602.23906#pg2>The results demonstrate that unit-level supervision addresses compositional understanding broadly rather than overfitting to the half-truth setup <ref:2602.23906#pg2> CS-CLIP achieves 69.3% Half-Truth Accuracy (vs.

Jane: CLIP 40.6%), the best compositional I2T accuracy (57.8%), and the highest Group Accuracy among evaluated methods <ref:2602.

Title and authors: Jane: We've talked about the core idea behind this paper now, which is about tackling those similarity score tricks where adding plausible but wrong details can mislead models ><ref:2602.23906#pg2>

Tom: And the authors of "Half-Truths Break Similarity-Based Retrieval" are Bora Kargi, Arnas Uselis, Seong Joon Oh ><ref:2602.23906#pg1>

Lu: They identified that contrastive training aligns images with full captions, which gives strong supervision at the sentence level but leaves weak supervision on the individual units that make up that caption ><ref:2602.23906#pg2>

Meng: So they focused their work on how to fix this weak link by adding targeted supervision directly onto those components >#page four <ref:2602.23906#pg3>

Tom: They propose CS-CLIP, which takes those full captions and decomposes them into entity and relation units, constructs a minimally edited foil for each unit, and then fine-tunes the model to score the correct unit higher than its foil ><ref:2602.23906#pg3>

Jane: So it’s like they are essentially giving the model direct instruction on how to value these specific components during training while keeping the standard dual-encoder architecture intact ><ref:2602.23906#pg2>

Lu: The goal is to make similarity scores more sensitive to this underlying structure rather than just relying on coarse overlap of words ><ref:2602.23906#pg3>

Meng: That means instead of just matching the whole thing, we want it to verify how the entities are actually composed before it gives a high score >#page four <ref:2602.23906#pg3>

Tom: The training objective involves a global sentence-level loss plus this unit-level loss where that second part forces the image embedding to score the actual correct unit higher than its foil and in-batch units from other images ><ref:2602.23906#pg2>

Jane: So it’s like they are giving the model targeted practice on how to correctly identify and value these specific parts of a caption ><ref:2602.23906#pg3>

Lu: That way, we get more compositional pressure on individual caption parts while maintaining the standard dual-encoder architecture which is what most people use ><ref:2602.23906#pg3>

Meng: It's interesting how they do that without completely breaking the existing pipeline, since we usually want to deploy things without major architecture overhauls >#page four <ref:2602.23906#pg3>

Tom: The paper goes on to detail exactly how they build those foils and what the loss functions look like for both entity and relation units ><ref:2602.23906#pg3>

Jane: It’s a lot of math, but the concept is very clear: we are making the similarity score depend on whether the correct unit is present ><ref:2602.23906#pg2>

Lu: The whole point is that we want to move beyond just testing if models can swap or reorder information, and test if they actually penalize incorrect additions ><ref:2602.23906#pg2>

Meng: That shift in what we test—from reordering to adding things—is what makes this research distinct from prior work on compositional understanding >#page four <ref:2602.23906#pg3>

Tom: So this paper really introduces a new way of supervising the model to handle these structural weaknesses in multimodal models ><ref:2602.23906#pg1>

Jane: It’s about making the similarity score sensitive to the structure of the caption instead of just looking at the whole thing as one blob ><ref:2602.23906#pg2>

The paper's summary: Tom: So, we've covered how CS-CLIP works conceptually, and now let’s look at what they actually found in terms of results for "Half-Truths Break Similarity-Based Retrieval" ><ref:2602.23906#pg1>

Jane: They show some pretty solid numbers. The main thing is that this method raises the half-truth accuracy on COCO from forty point six percent for standard CLIP up to sixty-nine point three percent with CS-CLIP ><ref:2602.23906#pg1>

Lu: That’s a substantial jump, especially since they also found that relation additions are harder to penalize than entity additions across different models ><ref:2602.23906#pg2>

Meng: From a deployment standpoint, the authors are using a learning rate of five times ten to the negative six, which they say balances those half-truth gains with competitive downstream performance on things like zero-shot classification >#page four <ref:2602.23906#pg3>

Tom: They also found that adding matched unit foils without global negatives produced the largest Half-Truth improvement, specifically getting sixty-eight point three percent for all units and sixty-three point seven for relations >#page four <ref:2602.23906#pg3>

Jane: So to wrap up, the paper on "Half-Truths Break Similarity-Based Retrieval" shows that this approach works broadly across many compositional benchmarks, improving average accuracy by five point seven percent somewhere on the list ><ref:2602.23906#pg1>

Lu: They also found that CS-CLIP achieves the best Role Sensitivity at forty point one percent compared to CLIP's thirty-four point six percent ><ref:2602.23906#pg1>

Meng: For us, I want to know how much overhead this unit decomposition adds to inference latency when we scale it up in production >#page four <ref:2602.23906#pg3>

Tom: The authors used a learning rate of five times ten to the negative six as a way to balance those half-truth gains with competitive zero-shot performance on ImageNet >#page four <ref:2602.23906#pg3>

Jane: And it seems like this approach is working broadly across many compositional benchmarks, meaning it’s not just an improvement on a tiny subset of data ><ref:2602.23906#pg1>

Lu: The result is that when you look at "Half-Truths Break Similarity-Based Retrieval," the model becomes much more sensitive to individual details in text descriptions ><ref:2602.23906#pg2>

Meng: So we have to keep in mind that they also found that relation additions are harder to penalize than entity additions across different models ><ref:2602.23906#pg2>

Tom: It’s definitely a solid piece of research, showing how targeted training on caption units can fix structural weaknesses in multimodal models ><ref:2602.23906#pg1>

The paper's improvements: Jane: So we've looked at "Half-Truths Break Similarity-Based Retrieval," and the main thing they’re showing is how unit-level supervision fixes those similarity score tricks that plague CLIP models ><ref:2602.23906#pg1>

Tom: Right, so it’s about this "half-truth vulnerability" where adding a plausible but wrong detail makes the model think a description is more accurate than the original one ><ref:2602.23906#pg2>

Lu: They break down captions into entity and relation units and train the model to score those correct units higher than their foils, while keeping the standard dual-encoder inference running normally ><ref:2602.23906#pg3>

Meng: It’s interesting how they do that without completely breaking the existing pipeline, since we usually want to deploy things without major architecture overhauls >#page four <ref:2602.23906#pg3>

Lalam: I agree, it’s about making the similarity score sensitive to the structure of the caption instead of just looking at the whole sentence as one blob ><ref:2602.23906#pg2>

Tom: And on COCO, they hit a half-truth accuracy of sixty-nine point three percent compared to forty point six for standard CLIP ><ref:2602.23906#pg1>

Jane: That’s a big jump, and it means when we use these models for retrieval tasks, the matching is actually more reliable because it’s paying attention to the right parts of the description ><ref:2602.23906#pg2>

Lu: They also found that relation additions are harder to penalize than entity additions across different models, reaching sixty-five point five percent accuracy on those ><ref:2602.23906#pg2>

Meng: For us, I want to know how much overhead this unit decomposition adds to inference latency when we scale it up in production >page four <ref:2602.23906#pg3>

Conclusion: Tom: So we've been looking at "Half-Truths Break Similarity-Based Retrieval," and the main thing is they’re showing how unit-level supervision fixes those similarity score tricks that plague CLIP models ><ref:2602.23906#pg1>

Jane: Right, so it’s about this "half-truth vulnerability" where adding a plausible but wrong detail makes the model think a description is more accurate than the original one ><ref:2602.23906#pg2>

Lu: They break down captions into entity and relation units and train the model to score those correct units higher than their foils, while keeping the standard dual-encoder inference running normally ><ref:2602.23906#pg3>

Meng: It’s interesting how they do that without completely breaking the existing pipeline, since we usually want to deploy things without major architecture overhauls >#page four <ref:2602.23906#pg3>

Lalam: I agree, it’s about making the similarity score sensitive to the structure of the caption instead of just looking at the whole sentence as one blob ><ref:2602.23906#pg2>

Tom: And on COCO, they hit a half-truth accuracy of sixty-nine point three percent compared to forty point six for standard CLIP ><ref:2602.23906#pg1>

Jane: That’s a big jump, and it means when we use these models for retrieval tasks, the matching is actually more reliable because it’s paying attention to the right parts of the description ><ref:2602.23906#pg2>

Lu: They also found that relation additions are harder to penalize than entity additions across different models, reaching sixty-five point five percent accuracy on those ><ref:2602.23906#pg2>

Meng: For us in production, I want to know how much overhead this unit decomposition adds to inference latency when we scale it up in production >#page four <ref:2602.23906#pg3>

Tom: The authors used a learning rate of five times ten to the negative six, which they say balances those half-truth gains with competitive zero-shot performance on ImageNet >#page four <ref:2602.23906#pg3>

Jane: And it seems like this approach is working broadly across many compositional benchmarks, improving average accuracy by five point seven percent somewhere on the list ><ref:2602.23906#pg1>

Lalam: The result is that when you look at "Half-Truths Break Similarity-Based Retrieval," the model becomes much more sensitive to individual details in text descriptions ><ref:2602.23906#pg2>

Tom: It’s definitely a solid piece of research, showing how targeted training on caption units can fix structural weaknesses in multimodal models ><ref:2602.23906#pg1>

Jane: It means that for future retrieval systems, we might start thinking about how to break those captions down into components instead of just looking at the whole thing >#page four

Lu: Exactly. If you want better compositional understanding, you need to train the embedding space to respect those underlying units ><ref:2602.23906#pg3>

Meng: I'm still focused on whether we can keep this unit-level loss manageable when we introduce more complex reasoning tasks later on >#page four