The day in conversation for 2026-10-19

The discussion: Tom: That TACO finding is massive. Reducing optimizer state by 174 times over AdamW8bit while keeping accuracy stable on OPT-13B? That’s not just a tweak; that fundamentally changes the economics of training frontier models. Jane: I agree, it’s huge for hardware accessibility, but I worry we’re focusing too much on the optimizer and not enough on the actual reasoning improvements. Look at how Cross-Behavior Policy Optimization is tackling offline versus online data sources; that feels like a more direct path to better generalization than squeezing out every last byte of memory. Lu: But the memory constraint *is* the bottleneck for generalization, Jane. If we can’t fit these larger models effectively, we can’t train them on diverse datasets required for that policy optimization anyway. The TACO work is just enabling the training phase; it doesn't solve the inherent complexity of learning from heterogeneous sources. Meng: I see your point, Lu, but the summary keeps circling back to resource efficiency as the overarching theme. If we can efficiently fit a 30-32B parameter model onto one 80GB card—which is what TACO proved—that efficiency gain immediately opens up space for more ambitious architectures or longer training runs. It’s enabling bigger things, not just making small optimizations faster. Lalam: I think the real takeaway here, though, is the open question at the end: how do we systematically attribute performance gains to higher-order information? That theoretical problem overshadows the practical wins of fitting a model onto hardware. If we don't know *why* a specific learning structure performs better than another across modalities, all this memory optimization is just optimizing a black box that might still be fundamentally flawed. Tom: Lalam makes that point about the attribution being key. I think we’re treating the symptom—the lack of resources—with TACO, but we haven't fully addressed the underlying structure problem yet. We need to understand *what* higher-order information is actually influencing those cross-domain transfers before we can optimize around it effectively. Jane: Exactly