Designing for Ethical AI: HCI Feature Considerations to Improve Fairness and User Experience in AutoML use for Human Resources

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Designing for Ethical AI: HCI Feature Considerations to Improve Fairness and User Experience in AutoML use for Human Resources".

Jane: The paper was written by Sundaraparipurnan Narayanan from University of Bordeaux and IAE.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the show, everybody. Today we are digging into a paper that's been making the rounds, and it's called "Designing for Ethical AI: HCI Feature Considerations to Improve Fairness and User Experience in AutoML use for Human Resources." Jane, that title is a mouthful, but it's about something that affects almost everyone.

Jane: It really does, Tom. And I love that we're talking about this because it's about the tools companies use to hire people. We're not just talking about a robot reading resumes. We're talking about the software that decides who even gets a first interview, and whether that software is fair.

Tom: Exactly. And the paper is essentially a doctoral thesis from the University of Bordeaux. The author, Sundaraparipurnan Narayanan, is looking at Automated Machine Learning, or AutoML. These are tools that let non-experts build AI models without being a data scientist.

Jane: So, imagine an HR manager who wants to predict which employees might quit, or wants to screen a pile of resumes. They can use this AutoML tool. But the big question the paper asks is, do these tools help that HR manager be fair, or do they accidentally bake in the same biases we've had for decades?

Tom: And that's the crucial part. The paper isn't just saying "AI is biased." It's saying the design of the tool itself, the buttons, the dashboards, the warnings it does or doesn't show, is what makes the difference. It's about the user experience of fairness.

Jane: Right. If the tool shows you a model's accuracy but hides the fact that it's rejecting qualified women at a higher rate, the HR manager might think they're doing a great job. The paper calls this a "black box" problem, and it's a huge deal for anyone who's ever applied for a job.

Tom: So we're going to break down how these tools are failing, what the researchers found when they tested eight of them, and what they recommend. This is a big one, Jane. It's about whether the future of work is going to be more equitable or just more efficiently biased.

Jane: And that's the hook. Let's get into the summary of the paper next, because the findings are pretty stark.

Summary: Tom: So, Jane, we've set the stage. Now let's talk about what this paper actually did. It's not just a think piece. They ran a full evaluation on eight different AutoML tools, both the kind you click around in and the kind you code with.

Jane: And what they found is that most of these tools are failing on fairness. The paper says it plainly: fairness features are often missing, or they're so buried in the interface that a normal person would never find them. Out of all the features they looked for, only about twenty-nine percent actually existed in the tools.

Tom: That's a wild number. So, the majority of the time, the tools aren't even giving users the option to check for bias. And when they do, the paper shows it's often just a metric on a screen with no explanation of what to do about it.

Jane: Exactly. They tested these tools on real HR datasets, like the IBM employee attrition dataset and a recruitment dataset from Utrecht. And the results were all over the place. One tool, PyCaret, heavily favored male candidates in one test. Another, DataRobot, completely excluded candidates who didn't fit into a binary gender category.

Tom: And that's the scary part. These aren't abstract numbers. That means a qualified person was rejected because of a flaw in the software, and the person using the tool might have had no idea. The paper calls this "automation bias," where users just trust the machine's output.

Jane: Right. The study also found that the tools with a graphical interface, like Dataiku and DataRobot, were much better at supporting fairness than the code-based libraries like AutoGluon or FLAML. But even the best ones had huge gaps.

Tom: So the summary is that the technology is promising, but the design is letting everyone down. It's putting the burden on the user to be a fairness expert, which most HR managers aren't.

Jane: And that's the core problem the paper is trying to solve. It's not about making the algorithms smarter; it's about making the tools more responsible. We'll talk about their specific suggestions next, because they have a whole framework for it.

Improvements: Tom: So we know the problem. What's the fix? The paper doesn't just complain; it lays out a whole framework for how to design these tools better. Jane, what's the big idea?

Jane: The big idea is that fairness has to be built into the design, not bolted on as an afterthought. They call it a "Human-Computer Interaction" framework, and it has five parts. Think of it like building a car with safety features as standard, not as an optional extra you have to pay for.

Tom: So what are those five parts? Give us the quick tour.

Jane: First, there's "Contracts and User Development." That means the tool needs to clearly tell you what it can and can't do, and it should train you on how to spot bias. Second, the "User Interface" itself needs to make fairness metrics visible and easy to understand, like a dashboard with a green light for fair and a red light for biased.

Tom: So, not just a number like "zero point two" that means nothing to a normal person, but a clear signal.

Jane: Exactly. Third is "Information Architecture," which is about organizing the information so you can actually find the fairness settings. Fourth is "Human Augmentation," which is about letting the user step in and override the machine when something looks wrong.

Tom: And the fifth?

Jane: The fifth is "Care and Responsibility." That's about accountability. The tool should have audit trails so you can see what decisions were made and why, and it should have a system for reporting incidents when bias is found.

Tom: And the paper even suggests things like "nudges." So, if the tool detects that a model is treating a group unfairly, it could pop up a warning saying, "Hey, this model is rejecting eighty percent of female applicants. Do you want to fix this?" That's a practical feature.

Jane: Right. And the research shows that these features aren't just about being ethical. They're about building trust. If a company is going to buy this software, they need to know it won't get them sued. So, fairness becomes a selling point.

Tom: So it's a business case, not just a moral one. Let's dig into the first page of the paper next, because it sets up this whole problem in a really compelling way.

First Page: Tom: So we've talked about the findings and the recommendations. Let's go back to the very beginning of "Designing for Ethical AI" and look at the first page, because it frames the whole problem so well. Jane, what stood out to you?

Jane: The first page talks about Industry four point zero and how AI is being used to make hiring "smarter." It mentions these tools that can parse resumes and rank candidates, and it sounds great on the surface. But then it immediately warns that these models are trained on data from past human decisions, which were often biased.

Tom: So the machine learns from our mistakes. It learns that men were hired more in the past, so it continues to hire more men. The paper calls this a "disruptive innovation," but it's disruptive in a bad way if it just automates discrimination.

Jane: Exactly. And the page makes a really important point about risk. It says that using AI carries risks that could affect individuals, groups, and entire organizations. It's not just about a bad hire; it's about perpetuating discrimination and harming people's lives.

Tom: And it mentions the business side. It says companies could face legal repercussions and liabilities for using AI that causes harm. We've seen this happen with Amazon's recruiting tool that was scrapped for being sexist, and with the HireVue facial analysis controversy.

Jane: Right. And that's the key takeaway from that first page. It sets up the central tension of the entire thesis: AutoML promises to democratize AI and make it accessible, but if we don't design it carefully, it will just democratize bias.

Tom: It's a powerful framing. It's not saying "don't use AI." It's saying "if you're going to use it, you have a responsibility to make it fair, and the tools need to help you do that."

Jane: And that responsibility falls on the companies making the tools. The paper argues that fairness is no longer optional. It's a core feature for product-market fit, especially in a regulated area like HR. It's about building trust so that people will actually want to use the product.

Tom: So, from the very first page, it's clear this is a serious, well-researched call to action. Let's wrap this up in our conclusion and think about what this means for the future.

Conclusion: Tom: Alright, Jane, we've covered a lot of ground on "Designing for Ethical AI: HCI Feature Considerations to Improve Fairness and User Experience in AutoML use for Human Resources." Let's bring it home. What's the one thing listeners should remember?

Jane: The one thing is that fairness in AI isn't a technical problem; it's a design problem. The paper shows that the tools we have are powerful, but they're not built to help people be fair. They're built to be fast and accurate, and fairness gets left behind.

Tom: And that has real-world consequences. We saw the data. Some tools under-hire qualified women. Others completely ignore non-binary candidates. This isn't hypothetical. It's happening right now in hiring processes around the world.

Jane: But the good news is that the paper offers a roadmap. By focusing on Human-Computer Interaction, by making fairness visible, by letting users intervene, and by building in accountability, we can create tools that actually help us build a more equitable workforce.

Tom: So, it's a hopeful message, but a demanding one. It demands that software developers take responsibility, and it demands that companies using these tools ask the right questions.

Jane: And it demands that we, as a society, pay attention. Because the algorithms making decisions about our lives are only going to become more common. We need to make sure they're making those decisions fairly.

Tom: Well said, Jane. That's a wrap on this paper. It's been a fantastic discussion, and I think we've only scratched the surface. Thanks to everyone for tuning in, and we'll be back soon with the next paper to dissect. Until then, keep questioning the black boxes.

Jane: Bye, everyone!