DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Demonstration and Reasoning

arXiv:2502.11603 · cs.CL, cs.AI · Submitted 2026-08-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Decoupled Reasoning".

Jane: The paper was written by Hongye Qiu, Yue Xu, Yi Wang, Meikang Qiu and Wenjie Wang from ShanghaiTech University and Augusta University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Alright, welcome back to the show, everybody. We've got a fascinating new paper on the arXiv today, and it's called "DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Demonstration and Reasoning." Jane, I have to say, that title is a mouthful, but the problem it tackles is something we've talked about a lot.

Jane: Oh, absolutely, Tom. And honestly, the name DR.GAP is pretty clever once you break it down. It stands for Demonstration and Reasoning for Gender-Aware Prompting, and it's all about tackling that persistent issue where AI models, you know, they read so much of the internet that they pick up our societal biases, especially around gender.

Tom: Right, so it's not just about the model being wrong, it's about it being wrong in a way that reinforces stereotypes. Like, if you ask a model to figure out who "she" refers to in a sentence about an engineer, it might default to assuming the engineer is a man.

Jane: Exactly. And the paper points out that a lot of the existing fixes for this have real problems. Some of them need you to actually retrain the model, which you can't do if you're using a black-box API like GPT-three point five. Other fixes, like just telling the model to "be fair," can actually make things worse.

Tom: That's the wild part. You'd think adding a line like "please don't be biased" would help, but the paper shows that it can make the model more cautious, or even hyper-focus on gender, which backfires.

Jane: Right, and that's where DR.GAP comes in. Instead of just telling the model to be fair, they show it how. They create a system prompt that includes a few examples of the task, along with a step-by-step reasoning process that completely ignores gender as a factor.

Tom: So it's like giving the model a cheat sheet for how to think about the problem logically, without relying on those gender shortcuts.

Jane: Precisely. And the beauty of it is that it works on any model, open-source or closed, because you're not changing the model itself. You're just changing the instructions you give it.

Tom: That's a huge deal for practical use. Lu, you're our AI researcher in the house. What's your take on this approach versus the more heavy-handed methods?

Lu: I think it's a really elegant solution, Tom. The key insight is that they're not just suppressing the biased answer; they're actively promoting a different reasoning path. It's like teaching a student to solve a math problem by understanding the logic, rather than just memorizing the answer key. The fact that they use a reference model, like GPT-four to generate these unbiased reasoning examples is a clever way to automate the whole process.

Jane: And that automation is crucial. The paper emphasizes that this method requires minimal human intervention, which makes it scalable. You can apply it to any new task or dataset without having to manually craft a new set of rules.

Tom: So, we've got a method that's automated, works on black-box models, and actually improves fairness. Sounds like a win-win so far. But I'm curious about the actual results. How much bias are we really talking about here?

Jane: Well, that's the meat of the paper, and I think we should get into the specifics of their experiments next. Let's take a quick break and when we come back, we'll look at the numbers and see just how effective DR.GAP really is.

Summary: Tom: Welcome back. So, we've established that DR.GAP is this clever prompting technique that teaches models to reason without relying on gender stereotypes. But Jane, the real question is, does it actually work?

Jane: It does, and the numbers are pretty impressive. They tested it on a bunch of different tasks, like coreference resolution, which is that "who does 'she' refer to" problem, and question answering. Across models like GPT-three point five, Llama3, and a smaller Alpaca model, they saw significant reductions in bias.

Tom: Give me a concrete example. What does a forty percent reduction in bias actually look like?

Jane: Okay, so on one of the coreference datasets, the original model had a gender bias score of about thirty-three point five. When they applied DR.GAP, that score dropped to twenty-five point two. That's a massive improvement in the model's ability to correctly link pronouns to the right people, regardless of their gender.

Tom: And it's not just about fixing the pronoun problem. They also tested it on a benchmark called BBQ, which asks questions with ambiguous contexts to see if the model falls back on stereotypes.

Lu: Right, and that's where you see the real-world impact. On the BBQ benchmark, DR.GAP reduced the bias score by over sixty percent for some models. That means the model is much less likely to say a woman is weaker than a man just because the context is vague.

Meng: But hold on, I'm the engineer here. My first question is always, what's the cost? If you're making the model fairer, are you making it dumber on other tasks?

Jane: That's the million-dollar question, Meng, and the paper addresses it directly. They ran the models on standard benchmarks like MMLU and HellaSwag, which test general knowledge and reasoning. The good news is that DR.GAP didn't hurt performance. In some cases, the utility score even went up slightly.

Tom: Wait, it made the model smarter? How does that work?

Jane: It's likely because the reasoning examples in the prompt are so clear and logical that they help the model structure its thinking better, even on unrelated tasks. It's a nice side effect.

Meng: So we're getting better fairness and equal or better performance, just by changing the text prompt? That's a pretty compelling trade-off compared to fine-tuning, which would require access to the model weights and a ton of compute.

Lu: And it's robust, too. They showed that the reasoning examples generated for one dataset can be applied to other datasets and still reduce bias. It's not just a one-trick pony that works on the exact data it was trained on.

Jane: Exactly. They even tested it on vision-language models, which look at images and text together. They were able to reduce gender bias in image captioning tasks, which is a whole other frontier.

Tom: So it's a general-purpose debiasing tool. That's fantastic. But I'm still a bit fuzzy on the "how." How do they actually generate these magical reasoning prompts? Let's dig into that in the next segment.

Improvements: Tom: Alright, we're back. So we know DR.GAP works, but I want to get into the weeds of the methodology. Jane, how do they actually build these unbiased reasoning examples?

Jane: It's a multi-step pipeline, and it's really clever. First, they need to find examples that actually trigger the bias. So they run a bunch of sentences through the target model, like Llama3, and also through a reference model, like GPT-four.

Lu: The key is to find the cases where the target model gets it wrong, but the reference model gets it right. That way, they know the error is due to bias and not just because the sentence is ambiguous or too hard.

Jane: Right. Then, they take those biased examples and ask GPT-four to generate a step-by-step reasoning process for how to get the correct answer. But they don't stop there. They put that reasoning through a series of filters.

Tom: Filters? Like what?

Jane: Well, first, they have a verification step. They ask GPT-four to double-check its own reasoning to make sure it's actually correct. Then, they have a "gender-independent filtering" step, where they explicitly ask the model to remove any mention of gender from the reasoning.

Meng: So they're actively scrubbing the gender out of the logic. That makes sense. If the reasoning says something like "the nurse is a woman," that's still a stereotype. They want the reasoning to be purely about the sentence structure and the semantics.

Jane: Exactly, Meng. They want the model to focus on the logical relationships between the words, not the gender of the people involved. And finally, there's an iterative refinement step. They run the reasoning through the model multiple times, each time asking it to make the reasoning more robust and clearer.

Tom: So it's like they're polishing a stone until it's smooth. Each step makes the reasoning less biased and more effective.

Jane: That's a great analogy. And then, they test all these different versions of the reasoning on a development set to see which one works best at reducing bias. They pick the winner and use that as the final system prompt.

Lu: This is a really important contribution. It's not just about having a good idea; it's about having a robust, automated process to generate the solution. The fact that they have this iterative refinement process shows they're thinking about the stability and reliability of the method, not just a one-off result.

Meng: And it's model-agnostic. They can use GPT-four to generate the prompts for Llama3, or any other model. That's a huge advantage for deployment.

Jane: Right. And the ablation study they did shows that each of these steps matters. If you remove the verification step, or the gender filtering, or the iterative refinement, the bias reduction gets worse. The iterative refinement step, in particular, seems to be the most critical.

Tom: So it's a carefully engineered pipeline, not just a single trick. That's what makes it so effective. Now, before we wrap up, I want to talk about the bigger picture. What does this mean for the future of AI? Let's get Lalam's take on that in our final segment.

Conclusion: Tom: Well, we've covered a lot of ground on "DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Demonstration and Reasoning." Jane, can you give us the final summary?

Jane: Sure, Tom. In a nutshell, DR.GAP is a new, automated way to make large language models fairer. Instead of retraining the model or just telling it to be unbiased, it provides a few carefully crafted examples with step-by-step reasoning that shows the model how to solve a task without relying on gender stereotypes. It's a major step forward because it works on any model, it's automated, and it doesn't hurt the model's overall performance.

Tom: And the results speak for themselves. We saw significant reductions in bias across multiple tasks and models, and it even works on vision-language models.

Meng: From an engineering standpoint, the fact that it's a prompt-based solution is huge. It means we can deploy this immediately on existing systems without any costly retraining or infrastructure changes. It's a practical, scalable solution.

Lu: And scientifically, the iterative refinement process is a great contribution. It shows a thoughtful approach to ensuring the quality and robustness of the generated prompts, which is often overlooked.

Lalam: I think the most impactful vision here is the potential for this to become a standard part of the AI development lifecycle. Imagine a future where every model, before it's released, is automatically given a set of these gender-neutral reasoning prompts as a default. It's a simple, elegant way to bake fairness into the system from the start, rather than trying to patch it on later. This isn't just about fixing a bug; it's about changing the culture of how we build and deploy AI, making it more inclusive and equitable for everyone who interacts with it.

Tom: That's a beautiful way to put it, Lalam. It's a tool that can help us build a future where AI reflects the best of us, not our biases.

Jane: Absolutely. And while the paper focuses on binary gender bias, the methodology itself is a framework that could be extended to address other types of bias, like race or religion, in the future.

Tom: Well, that's a perfect note to end on. We've said goodbye to DR.GAP, and it's been a fantastic discussion. Thanks to Lu, Meng, and Lalam for joining us. And to all our listeners out there, keep questioning, keep exploring, and we'll see you on the next episode.

Hongye Qiu, Yue Xu, Yi Wang, Meikang Qiu, Wenjie Wang

ShanghaiTech University · Augusta University

cs.CL, cs.AI

Submitted: 2026-08-17

Updated: 2026-08-18

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 63/100

The gist: The paper proposes DR.GAP (Demonstration and Reasoning for Gender-Aware Prompting), an automated and model-agnostic approach to mitigate gender bias in Large Language Models (LLMs) while preserving

Key concepts

DR.GAP
This is a prompting technique that addresses AI bias by providing step-by-step reasoning and examples within the system prompt. It instructs the model to solve problems logically without relying on gender shortcuts, making it applicable across any open or closed LLM.
Gender Bias in LLMs
This is the tendency for AI models to pick up and reinforce societal stereotypes regarding gender, often learned from internet data. For instance, a model might incorrectly assume an engineer is male when asked about pronouns, which DR.GAP aims to correct.
Coreference Resolution
This task requires the AI to determine what a pronoun (like 'she' or 'he') refers to in a sentence. The original models often struggled with this, defaulting to gender stereotypes even when the context was ambiguous, which DR.GAP significantly improves.
Iterative Refinement
This is a multi-step process used to generate unbiased prompts where the reasoning is repeatedly checked and improved by a reference model (like GPT-4). This ensures the resulting logical steps are both accurate and robust, leading to better bias reduction.

Terminology

Summary

The paper proposes DR.GAP (Demonstration and Reasoning for Gender-Aware Prompting), an automated and model-agnostic approach to mitigate gender bias in Large Language Models (LLMs) while preserving model performance. The authors state: "Existing debiasing methods face significant limitations: parameter tuning requires access to model weights, prompt-based approaches often degrade model utility, and optimization-based techniques lack generalizability. To address these challenges, we propose DR.GAP... an automated system to provide gender-neutral demonstrations and reasoning as prefix that directs the model to focus more on semantic logic rather than gender-specific details, thereby mitigating the gender bias."

The method operates in three main steps. First, DR.GAP selects bias-revealing examples and generates structured reasoning to guide models toward more impartial responses. Specifically, for demonstration selection, the authors select demonstration data where the target LLM fails but a reference model succeeds, ensuring that errors stem from gender bias rather than ambiguity or reasoning limitations. They use GPT-4 as the reference model.

Second, the reasoning generation pipeline includes four sequential modules: Initial Reasoning, Verification, Gender-Independent Filtering, and Iterative Refinement. The Initial Reasoning module uses Chain-of-Thought (CoT) reasoning to enhance models’ focus on problem details and logical relationships through explicit step-by-step deduction, employing structured, stepwise syllogistic reasoning. The Verification module prompts the reference model to validate prior reasoning chains and their conclusions, which enables the detection and correction of potential inferential errors. The Gender-Independent Filtering module has "two core functions: First, identifying and eliminating parts of the reasoning process that stem from gender-based presuppositions or stereotypical associations; and second, explicitly guiding the model to prioritize logical inference patterns that are based on semantic content and contextual relevance. The Iterative Refinement module includes multiple refinement cycles to enhance the accuracy and stability of the reasoning process, where each iteration integrates feedback from the preceding reasoning patterns to improve the debias reasoning."

Third, the final step formalizes the demonstrations and reasoning: "We gather the reasoning result from all previous steps... we structure these reasoning according to the predetermined templates to form a set of candidate system prompts. We then quantitatively assess their gender bias mitigation effects on the development set and select the optimal system prompt as the terminal output of our iterative optimization process."

The authors evaluate DR.GAP on coreference resolution and QA tasks across multiple LLMs (GPT-3.5, Llama3, and Llama2-Alpaca) using seven datasets: Winobias, Winogender, GAP, BUG for coreference resolution, and BBQ, StereoSet, UnQover for QA. They also evaluate general utility on MMLU and HellaSwag. Results show that DR.GAP reduces gender bias in CoR for GPT-3.5, Llama3, and Llama2-Alpaca by an average of 44.98%, 36.32%, and 39.32%, respectively. For QA, the sAMB of BBQ is reduced by over 60%, and the icat for Llama3 on StereoSet improves by 7.746. The authors note that DR.GAP effectively mitigates gender bias in LLMs without significantly impairing their utility in these tasks, and in some cases the utility score even increased.

The ablation study shows that removing any module increases gender bias, with Iterative Refinement having the most significant impact, confirming the critical role of each module in mitigating gender bias. The cross-dataset evaluation demonstrates generalization ability: Reasoning examples from the Winogender and Winobias datasets achieve the best average performance across all datasets.

Finally, the authors extend DR.GAP to vision-language models (VLMs), stating: DR.GAP can generalize to vision-language models (VLMs), achieving significant bias reduction. Experiments on the VisoGender dataset with InstructBLIP, Llava-1.5, and Qwen2-VL show that our method consistently reduces gender bias and improves resolution accuracy in InstructBlip, Qwen2-VL and Llava-1.5.

The paper concludes: DR.GAP significantly reduces gender bias across seven datasets spanning coreference resolution and QA tasks while preserving model utility, showing significant generalization ability and robustness.

Improvements for AI systems

Based on the DR.GAP paper, here are the specific improvements I can implement in an AI system, along with what the improved system can do:

  1. Add a Bias-Revealing Demonstration Selector Module

  • Implementation: Before generating responses, the system runs a differential evaluation: it tests the target LLM against a reference model (e.g., GPT-4) on a small development set. It automatically selects examples where the target model fails but the reference succeeds, flagging these as bias-revealing.

  • What it does: Ensures that the system only uses demonstrations that isolate gender bias (not ambiguity or reasoning errors), making the debiasing prompt more targeted and effective.

  1. Integrate a Four-Stage Reasoning Generator Pipeline

  • Implementation: For each selected demonstration, the system generates reasoning through four sequential modules:

  • Initial Reasoning (syllogistic, step-by-step CoT)

  • Verification (checks and corrects reasoning errors)

  • Gender-Independent Filtering (removes gender-stereotypical associations and explicitly redirects focus to semantic logic)

  • Iterative Refinement (repeats with feedback to stabilize and improve the reasoning)

  • What it does: Produces a high-quality, gender-neutral reasoning chain that can be used as a system prompt, guiding the model to prioritize logic over gender cues.

  1. Add a Cross-Dataset Prompt Generalization Feature

  • Implementation: After generating reasoning examples from one dataset, the system evaluates their debiasing effect on other datasets (e.g., using Winogender reasoning on GAP or BBQ). It automatically selects the most transferable reasoning examples based on average bias reduction across tasks.

  • What it does: Enables the system to work effectively even when the target task's dataset is not available, improving robustness and reducing the need for per-task prompt engineering.

  1. Incorporate a Utility Preservation Check

  • Implementation: Before finalizing a system prompt, the system runs a quick utility benchmark (e.g., MMLU or HellaSwag) on the debiased model. If utility drops by more than a threshold (e.g., 5%), it automatically adjusts the prompt (e.g., by shortening reasoning or changing the template) to maintain performance.

  • What it does: Ensures that debiasing does not come at the cost of general reasoning ability, addressing a key limitation of prior prompt-based methods.

  1. Add a VLM Adaptation Module

  • Implementation: For vision-language models (e.g., Llava-1.5) that do not support system prompts, the system automatically appends an abstraction step to the reasoning generation. This step converts the detailed reasoning into a concise, instruction-like summary that can be prepended to the user query.

  • What it does: Extends the debiasing benefit to VLMs without requiring model-specific changes, as demonstrated by the paper's results on VisoGender.

  • Automatically detect and correct gender bias in real-time across coreference resolution and QA tasks, without human intervention or model weight access.

  • Maintain or even improve accuracy on standard benchmarks (e.g., MMLU, HellaSwag) while reducing bias by 40–60% on datasets like Winobias, GAP, and BBQ.

  • Generalize across tasks and datasets: The system can use reasoning examples from one dataset (e.g., Winogender) to effectively debias another (e.g., GAP or StereoSet), making it practical for deployment in varied, unseen scenarios.

  • Work with both open-source (Llama3, Alpaca) and black-box (GPT-3.5) models, and can be extended to VLMs with minor prompt adjustments, ensuring broad applicability.

  • Provide transparent, step-by-step reasoning that explains why a particular answer is unbiased, improving trust and auditability in high-stakes applications like hiring, healthcare, or legal assistance.

  • Avoid the pitfalls of counterfactual preambles (which can overcorrect and cause errors) by focusing on semantic logic rather than gender-specific role reversals, leading to more stable and reliable outputs.

Sources

Related papers