BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows".

Jane: The combination of these excerpts allows for a comprehensive, high-fidelity understanding of this work's scope, methodology, contributions, and critical findings.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at a paper called BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows. Jane, what's the main takeaway from the title itself?

Jane: It sounds pretty specific, doesn't it? It’s not just about making an AI write a report; it’s about testing if an AI can actually handle the whole process of doing actual investment banking work.

Tom: Exactly. The authors, Handshake AI and their team, are tackling the problem that existing benchmarks just don't cover the real professional stuff. They say they collaborated with five hundred two investment bankers from leading firms to build this thing <ref:2604.11304#pg1,collaborated with 502 investment bankers from leading firms>.

Lu: So it’s built on real-world work environments rather than just textbook problems. That’s a big difference for testing agent capability, right?

Meng: I wonder how much of that real-world context is actually baked into the environment, or if the agents are just given a bunch of documents to look at.

Lalam: It seems they really focused on making sure the AI has access to actual tools and data rooms, not just free internet search.

The paper's summary: Jane: Okay, so what is BankerToolBench actually doing in detail? They are setting up a reinforcement learning environment where an agent gets a prompt from a banker, uses tools like market data APIs and SEC EDGAR APIs to navigate data rooms.

Tom: And the goal is for that agent to generate deliverables—things like Excel spreadsheets, PowerPoint decks, and PDF reports—all in one go. The whole point is testing if an AI can handle tasks that could take a human up to twenty-one hours.

Lu: They structured the tasks with a specific grading rubric from senior bankers, which means the evaluation isn't just about whether the answer is technically correct, but how well it follows professional standards.

Meng: That sounds intense. So it’s not just checking if the numbers add up; they’re checking if the entire package—the spreadsheet and the slide deck—makes sense together.

Lalam: The paper notes that approximately thirty-six percent of these tasks require producing a complete package, which is a significant part of their focus.

The paper's improvements: Tom: Now, the authors point out that just having the benchmark isn't enough; they are proposing specific improvements to make it better for testing AI agents. They are focusing on things like making sure the AI can handle complex financial modeling tasks, like convertible structures or IPO pricing.

Jane: And they highlight that a major part of their contribution is how they designed their verifier, which is an agent called Gandalf four. This verifier acts as an agent-as-a-judge model to grade the deliverables against over one hundred detailed criteria.

Lu: The key improvement here is moving beyond simple text evaluation to checking the actual output files, like that Excel workbook, and using this verifier to ensure accuracy across all those components.

Meng: From a deployment standpoint, that sounds like a lot of work for the verification system itself. So they’re trying to build something robust enough that it can actually grade at scale without breaking down on complex financial formulas.

Lalam: They emphasize that the resulting Excel workbook needs to be accurate and auditable, and even minor analytical or presentation errors are unacceptable, which really raises the bar for what an AI has to achieve.

Conclusion: Jane: So, to wrap up this discussion on BankerToolBench: it’s a rigorous test of end-to-end workflow execution in investment banking using real industry data and human rubrics. The main implication is that we need benchmarks that test the entire chain from request to final deliverable.

Tom: Right. And the results are pretty sobering for frontier models like GPT-five point four; they only manage to pass about sixteen percent of these complex tasks autonomously, which suggests they aren't ready for real banking work yet.

Lu: The paper finds that the failure isn't usually one small thing; it’s systemic breakdowns in cross-artifact consistency and unreliable handling of assumptions, which shows models struggle with maintaining coherence across many steps.

Meng: For us shipping things to production, that means we can’t just look at raw benchmark scores; we have to see if the model can actually be trusted with the whole workflow because isolated parts are not enough.

Lalam: The paper notes specific failure modes, like code and formula generation being a big weak spot, which tells us exactly where the AI is failing in these financial tasks.

Jane: It’s clear that the critical metric isn't whether an AI can do one small part of a task well, but whether it can be trusted with the whole workflow when you give it a real-world request like this.

Tom: Exactly. So we see this as a massive step toward understanding what’s actually required for AI agents to handle high-value, labor-intensive professions like finance.

Lu: It’s a good foundation for future work because it shows the direction we need to go to build systems that integrate capability into stable execution.

Meng: We'll keep watching these results and seeing if we can get those models to pass that reliability threshold when they are actually deployed on real traffic.

Lalam: So, BankerToolBench is a really high-fidelity attempt at evaluating AI agents in an end-to-end fashion for investment banking workflows.