Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design".

Jane: The paper was written by Vincent D. Zaballa and Elliot E. Hui from University of California, Irvine.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone. Today we're digging into a paper with a real mouthful of a title: "Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design." Jane, I'm going to need you to break that down for me before my brain melts.

Jane: Happy to, Tom. So there are two big ideas in that title. First, simulation-based inference — that's when you have a computer model of some real-world process, like how a disease spreads or how cells signal each other, and you want to figure out the hidden parameters just by running simulations. The second is Bayesian optimal experimental design — that's figuring out which experiment to actually run to learn the most from your limited resources.

Tom: Right, so one is about learning from data you already have, and the other is about choosing what data to collect. And this paper says those two things are actually the same problem?

Jane: Exactly. They show that the math you use to pick a good experiment is the same math you use to train your inference model. It's like realizing you've been carrying two separate toolboxes when one set of tools does both jobs.

Tom: And that matters because simulators are expensive. Every time you run one, it might take seconds or minutes. So if you can make each run count for both training and design, that's huge.

Jane: That's the core insight. The authors are from UC Irvine, and they've built a method called SBI-BOED that does both simultaneously. They're not just theorizing either — they test it on real biological models, like the BMP signaling pathway that's important in development and disease.

Tom: The BMP model sounds like something we should come back to. But first, tell me why this connection between inference and design is so surprising.

Jane: Well, traditionally these fields developed separately. Inference folks cared about getting accurate posteriors from fixed data. Design folks cared about maximizing information gain. This paper shows that if you use the right objective — a mutual information bound — you're actually training your likelihood model and optimizing your experiment at the same time.

Tom: So it's not just a clever trick, it's a fundamental link between two fields that thought they were doing different things.

Jane: Exactly. And that's what makes this paper exciting. It's not just a new algorithm, it's a new way of thinking about the problem.

Tom: I'm sold already. Let's get into the actual method and how they made this work in practice.

Summary: Tom: So we've established the big idea — connecting inference and design through mutual information. But how did they actually pull this off? What's the concrete method?

Jane: They use something called InfoNCE, which is a way to estimate mutual information using contrastive samples. You draw a bunch of parameter values, simulate data for one of them, and then ask your model to pick out which parameter generated that data. The better it does, the more information you're capturing.

Tom: And the trick here is that they use a normalizing flow as their likelihood model. That's a type of neural network that gives you a proper probability distribution. So instead of just a classifier that says "this looks right," you get an actual likelihood function you can use for downstream inference.

Jane: Right. And here's where it gets interesting. They add a regularization term — they call it lambda — that controls the tradeoff between maximizing information gain and fitting the likelihood accurately. It's like a dial between "explore new experiments" and "learn the model well."

Tom: And they found that dial matters a lot, right? In their Two Moons benchmark, they showed that cranking up the regularization improves calibration but lowers the information gain estimate.

Jane: Exactly. It's a genuine tradeoff. But the key result is that their method, SBI-BOED, works even when the simulator is a closed box — you can't differentiate through it. That's huge because most real scientific simulators are like that. You can run them, but you can't get gradients from them.

Tom: That's the part that got me excited. Previous methods like iDAD required differentiable simulators. This one doesn't. So it opens up a whole class of problems that were previously out of reach.

Jane: And they show it works on two real models. The SIR epidemiology model — susceptible, infected, recovered — and the BMP signaling pathway. For SIR, they got better posterior accuracy than iDAD while using about fifty times fewer simulator calls.

Tom: Fifty times fewer. That's not incremental, that's transformative. For anyone running expensive simulations, that's the difference between a project being feasible or not.

Jane: And on BMP, which is a closed-box simulator, they beat the strong baseline on calibration and predictive accuracy even though the baseline had a higher estimated information gain.

Tom: Wait, that's counterintuitive. How can you have lower information gain but better predictions?

Jane: That's one of the most interesting findings. The information gain metric can be misleading. A design that scores high on paper might not give you a posterior that's actually well-calibrated. So they're arguing we need to look beyond just the EIG number.

Tom: That's a provocative claim. It challenges how the whole field evaluates experimental design methods.

Jane: It does. And it's backed by real experiments. That's what makes this paper worth taking seriously.

Improvements: Tom: We're back with the paper "Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design." Jane, you mentioned they found EIG can be misleading. What improvements do they actually propose to fix these issues?

Jane: Well, they tackle several practical problems. First, there's the issue of sparse rewards — when you're optimizing designs, sometimes the gradient signal is just flat, so your optimizer gets stuck. They solve that by optimizing a distribution over designs instead of a single design. It's like exploring a whole neighborhood of experiments rather than just one point.

Tom: So instead of asking "what's the best design," you ask "what's the best region of designs to sample from." That gives you more stability.

Jane: Exactly. They use a truncated normal distribution that starts wide and narrows over time. That way, even if the reward landscape is bumpy, the distribution has enough support to find good regions.

Tom: And they also use checkpoints, right? Like in deep learning where you save the best model during training.

Jane: Yes. They checkpoint the design that achieved the highest EIG during training, so even if the optimizer wanders into a bad region at the end, they keep the best design they found. It's a simple trick but it prevents catastrophic forgetting of good designs.

Tom: And then there's the active learning component. They don't just optimize designs — they also decide which simulator calls to make when designs are fixed.

Jane: That's the EPIG part — expected predictive information gain. They use it to prioritize which parameter values to simulate next. Instead of drawing parameters uniformly from the prior, they pick ones where the model is most uncertain. They approximate that uncertainty using MC-dropout, which is a cheap way to get epistemic uncertainty from a neural network.

Tom: So they're being smart about both the design and the training data. Every simulator call is chosen to be maximally informative.

Jane: Right. And the results show this active learning approach improves calibration faster than random sampling under the same simulation budget. It's a complete package — design optimization, likelihood training, and simulation allocation all driven by the same information-theoretic principle.

Tom: That's elegant. One principle driving everything. But I want to push on something — they also mention sequential rounds of inference. How does that fit in?

Jane: They show that you can refine your likelihood between design rounds using the observed data, like traditional SBI methods do. This improves calibration over multiple rounds. It's a modular approach — you can add refinement on top of the design optimization.

Tom: So it's not just a single-shot method, it's a framework that can be extended and improved.

Jane: Exactly. And that flexibility is what makes it practical for real scientific workflows where you might have multiple rounds of experiments.

Conclusion: Tom: Alright, we're wrapping up our discussion of "Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design." Jane, give us the final takeaway.

Jane: The big picture is that this paper unifies two fields that were working in parallel. By showing that mutual information maximization is equivalent to likelihood learning, they've given us a single objective that handles both inference and experimental design. And they've made it work for closed-box simulators, which is where most real science happens.

Tom: And the practical impact is real. Fifty times fewer simulator calls on SIR, better calibration on BMP, and a method that doesn't require differentiability. That's going to change how people run experiments in biology, epidemiology, any field with expensive simulations.

Jane: But I think the most important contribution is the cautionary note — that EIG alone isn't enough. They showed that higher information gain doesn't always mean better posteriors. That's a wake-up call for the whole BOED community.

Tom: It's a humbling reminder that our metrics can lie to us. But it's also exciting because it opens up new research directions — how do we design experiments that are both informative and produce calibrated models?

Jane: And the framework they've built is flexible enough to incorporate new generative models, like diffusion models, as they become more practical. This isn't the end of the story, it's the beginning.

Tom: Well said. Thanks to everyone who joined us today — Lu, Meng, Lalam, and all our listeners. We'll be back with another paper soon. Until then, keep questioning your metrics and stay curious.

Jane: See you next time, everyone.