One Is Not Enough: How People Use Multiple AI Models in Everyday Life

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "One Is Not Enough".

Jane: The gist: People increasingly use multiple Multimodal Large Language Models (MLLMs) concurrently, selecting each based on its perceived strengths.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "One Is Not Enough: How People Use Multiple AI Models in Everyday Life". It looks at how people actually use several Multimodal Large Language Models concurrently, choosing each one based on what they think each one is best at.

Jane: That seems like a really practical problem because when you're juggling different AIs, there are these coordination challenges. Things like figuring out how to adjust your prompts for different interfaces and trying to keep trust consistent when the models behave differently depending on what you're asking them to do.

Lu: Prior research has really focused on single-agent interactions, so this paper is looking at the orchestration of multiple AI systems, which hasn't been explored much before. They did a diary study with ten participants and some interviews to see how people organize their work across these competing systems.

Meng: I'm curious about what they found about how users structure these hierarchies. Are we talking about a fixed setup or something more flexible?

Tom: It turns out users build primary and secondary hierarchies among the models that they switch between depending on the situation. The paper shows this structure actually shifts based on whether you're using the AI for personal stuff versus professional work.

Jane: That shift is interesting because it suggests that what’s most important to a user changes depending on their context, like personal time versus work time.

Lu: They found three recurring hierarchy patterns across the participants in this study: a cross-context primary structure, a work-focused structure, and a split-primary structure.

Meng: That sounds complex to manage. Can you tell me more about what those different structures actually look like in practice?

Tom: For the cross-context primary structure, four participants centered their workflows around one favorite model they used for both personal and professional life. They saw this favorite model as a general-purpose assistant and kept their interactions pretty consolidated.

Jane: So it sounds like some people have one go-to AI that handles everything from casual questions to work tasks because they just feel familiar with it.

Lu: Then there was the work-focused structure, where three participants kept one primary model dedicated specifically to professional tasks, treating it like a productivity tool. They deliberately didn't feel a strong need to use that primary model in personal contexts.

Meng: That makes sense for someone who really needs reliability for their job; they wouldn't want their work AI getting distracted by personal errands. But what about the split-primary structure?

Tom: Three participants maintained distinct primary models for personal and professional contexts. They usually used a lighter or faster option for low-stakes personal tasks, while reserving a stronger model for work tasks that require higher reliability.

Paper summary: Jane: It’s like having two different specialists: one quick and casual, and one robust and serious. So what else was influencing these choices, beyond just the task type?

Lu: The preference wasn't always purely about perceived performance. Sometimes it came down to things like subscription fees or usage costs for the different models. For instance, one participant offloaded trivial daily queries to a fast mode version of a model just to keep operational costs low.

Meng: That makes sense from an engineering standpoint; if you're running things at scale, cost is always a factor in deployment decisions. Are there any initial factors that set these preferences?

Tom: Early experiences with a model often create what they call first-impression lock-in, which continues to shape their later choices. One participant established Claude as their primary writing agent based on a very strong initial impression they had of it.

Jane: So the very first time you try an AI, that initial feeling can stick around and guide your use for a long time, even if you switch later.

Lu: They also mentioned expert consensus and social signals, where participants would pay attention to which models were being perceived as improving or leading in community discussions. One participant used expert consensus as a strategic filter when deciding when to re-evaluate their tool choices.

Meng: So it’s not just about the raw numbers; the community chatter matters for their strategy too. But what about how they actually work together across these systems? How do they handle switching between them?

Tom: They developed coordination strategies to switch intentionally by breaking a task into different stages and assigning each stage to the model they think is strongest for that specific part of the job. For example, ideation with one model, drafting with another, and then polishing or refining with a third.

Jane: That’s smart; you don't try to use one AI for everything from start to finish; you let each tool do the thing it does best in that sequence.

Lu: They also managed effort by task difficulty. Some participants switched models based on how hard or small the immediate task was, trying to maximize efficiency by using a different model for a very simple thing versus a complex one.

Meng: That sounds like dynamic routing based on the workload, which is what we try to build into production systems, but I wonder if that’s actually scalable when you have ten different models competing.

Tom: Then there was cross-checking for credibility when information needed to be reliable. When outputs were critical, they would switch across platforms to verify the results and reduce the risk of hallucinations. One participant used a three-stage sequence with ChatGPT, Gemini, and then back to ChatGPT to find evidence before assessing validity.

Jane: It shows that users aren't just blindly trusting one model; they build in a verification step when accuracy matters most.

Paper summary: Lu: They also noted that sometimes they chose to iterate with a single model instead of migrating because the model already knew their context, making switching too costly, or because they found the problem was actually in their own prompt specification rather than the model's capability.

Meng: So it sounds like when things aren't working perfectly, users sometimes focus on fixing their instructions instead of immediately jumping to a new tool. That’s a practical way to handle it at scale.

Tom: And some people simply decided not to migrate across platforms because reestablishing the background and personal context elsewhere felt too much effort compared to any potential performance gain they might see. One participant avoided switching away from Gemini for personal hobbies because retraining a new model on their specific persona would take excessive effort.

Jane: It seems like maintaining a relationship with an AI, even if it’s not the absolute best for every single moment, can be valuable if the context is already established. So we're seeing more about these complex systems, but what does this mean for the actual design of these tools?

Lu: The findings point toward two specific opportunities for tool design. First, they suggest a workflow-aware interface that could allow users to transfer a structured task representation between models without losing continuity.

Tom: They proposed a portable task card that captures the current task state—including goals, active constraints, and prior model outputs—which the user can then present to a new model to resume work without having to re-ground everything.

Jane: That’s interesting; it means the tool doesn't have to force you into one system; it needs to help you manage the handoff between systems.

Lu: The second idea is model-level memory partitioning within a single interface that could segregate interaction histories by domain or task type. This would help users protect distinct model contexts, addressing mismatches between what the user thinks the system remembers and how the system actually stores its history.

Meng: I like the memory partitioning idea because if you can separate contexts, it might help manage what breaks at scale by isolating failures to a specific domain or task type.

Tom: So, in short, users are treating AI models as an evolving ecosystem where they construct their own priorities and negotiate those boundaries between them. The paper "One Is Not Enough: How People Use Multiple AI Models in Everyday Life" shows us that this coordination is driven by context and task demand rather than just picking the single best model for everything.

Jane: It really highlights that the challenge isn't just building a better single model; it’s about designing systems that help people effectively manage these diverse tools when they are all working at once.

Conclusion: Tom: So we're wrapping up on "One Is Not Enough: How People Use Multiple AI Models in Everyday Life." It really boils down to how people aren't just picking one perfect AI, but actually building a whole system of them.

Jane: Exactly. We talked about those hierarchies they build—the primary and secondary models, shifting depending on whether you’re doing personal stuff or work stuff. It shows that the use case dictates the tool choice more than just raw performance metrics.

Lu: What I find really interesting is how they map out these switching patterns based on task aggregation. When a user has a big project, they don't switch randomly; they have these deliberate strategies for assigning different stages of work to different models.

Meng: From my side, what I want to know is how hard it is for an actual system to handle this kind of coordination at scale. If every user is constantly switching roles across ten models, the overhead just explodes.

Tom: That’s a fair point, Meng. But the paper suggests that users are actually iterating with a single model sometimes instead of jumping between platforms because they see their own prompts as the main failure point if the output isn't right.

Jane: So it's not always about finding a better tool; sometimes it's about fixing how you talk to the tool you already have. That’s a big mental shift for how we think about AI interaction.

Lu: It also touches on trust calibration, which is new territory for me in this context. They look at how users cross-check information by switching models just to make sure the output is actually reliable before they commit to it.

Meng: That verification step adds latency and complexity to the workflow, which is something I have to worry about when deploying these things. But if that verification reduces errors enough for a critical task, then maybe it’s worth the extra computation.

Tom: The authors suggest future tool design needs to help users keep their context alive even when they switch platforms. They are talking about portable task cards that carry the entire state of a conversation so you don't have to re-explain everything.

Jane: That means we stop thinking of AI tools as isolated silos and start thinking of them as parts of a larger, shared workspace where the work travels with it.

Lu: It’s a shift from treating AI like individual software packages to treating it like an evolving ecosystem where priorities are constantly negotiated by the user.

Meng: If that's what they're saying, then for us in production, we have to design interfaces that respect those distinct contexts so that users don't lose their established relationships with the models.