Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Compile the Table".

Jane: The gist The proposed QCOC method exploits exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today called "Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning". What is it actually trying to do?

Jane: Basically, the whole idea is that current ways of compressing those in-context examples have a problem. Fixed subsets sacrifice accuracy, and query specific retrieval limits how much you can reuse the cache across different questions.

Lu: This paper proposes QCOC, which shifts the focus from just compressing the examples themselves to compressing what happens when you query those examples. It claims that by using exchangeability and reusing those in-context examples repeatedly, you can compile their full KV cache once into a compact memory shared for later queries.

Meng: So instead of storing all the raw training examples, we're turning them into something smaller—joint-KV prototypes—that we can reuse. But they aren't just throwing away data; they are trying to keep the original example count and how many times each group appears in the clusters.

Lalam: And then they use an anchored closed-form solution to fit the values of these prototypes against attention query vectors generated by those examples, which is how they get their compressed value vectors.

Tom: That sounds pretty complex, so what's the real payoff for us? Is this just a theoretical exercise, or can we actually use this for faster inference?

Jane: The main claim is that QCOC drives the speedup through prototype compression while value fitting helps keep the accuracy close to what you get from using all the context at once. They show that on sixty-four held-out OpenML-CC18 datasets, it achieves high accuracy compared to other methods <ref:2610.11784#pg1,on 64 held-out OpenML-CC18 datasets>.

Meng: I like hearing about performance on those specific datasets, but what's the real constraint here? How much does this compilation cost for us when we start running things?

Lu: They did do analyze the component analysis and found that you need to use both the cluster-count weighting and preserving the original count N together. They said correction alone leaves attention computation inconsistent, and their effects aren't just adding up nicely.

Tom: Okay, so there are distinct roles for those parts—one corrects the way attention is weighted by the query transformation, while value fitting calibrates that resulting output. What does that mean in practice?

Jane: It means the system is doing two things at once: it’s adjusting how much importance each example gets based on your current query, and then it’s fine-tuning what those examples actually predict so the final answer is right.

Meng: From an engineering standpoint, that process of compiling the full cache sounds heavy. What are the specific costs they're talking about for this compilation step?

Paper summary: Lu: They estimated the QCOC compilation cost to be O(N squared d plus INM d plus NM squared plus M3) per head. That's a lot of operations depending on how many examples N you have and the dimension d.

Tom: So they've given us a concrete cost, which is helpful. But what’s the big picture for scaling this up? Can this really handle those long tables we see in practice?

Jane: On long tables, they showed that QCOC remains "close to Full in aggregate while reducing online serving cost across repeated queries." They specifically noted on CC18-Long that QCOC's cache is "five point three eight percentage points (ninety-five percent CI

+four point zero zero, +six point seven eight: ) and three point seven five percentage points (

+two point three one, +five point one six: )" better than the best retrieval baselines at M=thirty-two queries per head on that dataset >

Meng: That kind of improvement over retrieval baselines is significant when you’re talking about repeated queries, which is exactly what we need for efficient serving. But they also flagged a limitation, right? What's the catch?

Lu: Yes, they identified some failure cases in their analysis. They noted that a single shared memory can miss query-local structure, as seen in the Covertype result. So it’s not always perfect if you need hyper-local detail for every single query.

Tom: So we have this method that gets us close to full context accuracy while cutting online costs, but it has these specific structural limitations depending on the query pattern. Where does this leave us in terms of what we can actually use this for right now?

Jane: The paper is really suggesting that reusing a bounded cache provides an advantage over dynamic retrieval when you can represent the full-cache behavior globally. They are also pointing toward future work focusing on reducing compilation cost and selecting among multiple reusable memories.

Meng: So, it’s a step towards a more efficient system where we pre-compile based on expected usage patterns, rather than constantly searching through the whole set every time. It’s about making the initial heavy lift pay off over many subsequent uses.

Lu: And I think what they really highlight is that value fitting reliably improves full-cache fidelity, even if whether that changes the predicted class still depends on each dataset’s decision margins. That's a useful diagnostic tool for us.

Tom: It sounds like this paper gives us a way to treat compression as compiling the attention operator induced by a fixed set of examples, rather than just picking a smaller subset of rows. That's what I wanted to get across about "Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning".

Jane: And it really comes down to this: if you reuse that compiled memory enough times, the gains in speedup start to outweigh the initial compilation time, and you get performance very close to what a full context would give you.

Conclusion: Tom: So we've been talking about how this paper, "Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning," tries to squeeze massive context down into something small and fast.

Jane: It’s all about taking those big in-context examples and turning them into a shared memory structure that you can reuse many times for different queries without recomputing everything.

Lu: The authors, they are using exchangeability—meaning the examples behave similarly—to build these joint KV prototypes, which is pretty clever because it’s not just throwing away data randomly.

Meng: And they're fitting the values of those prototypes with some kind of regularized least-squares solve based on attention query vectors produced by the examples. That’s where the real magic happens in terms of accuracy versus size.

Lalam: From my side, I see this as moving from just storing text to storing an operator—a compressed version of the entire attention process defined by those examples. It lets us use a fixed set of knowledge more efficiently across repeated tasks.

Tom: So the main thing is that they’re treating compression like compiling the attention operator itself, rather than just picking a smaller subset of rows from the table.

Jane: Exactly, and they show that when you reuse this compiled memory enough times, you get predictive performance that’s very close to what you'd get from using the entire context all at once.

Lu: They did push back on one thing though, they showed that if you only use a single shared memory structure, it can miss some important local structure specific to a particular query.

Meng: That points toward how we might need multiple reusable memories instead of just one big bucket for this kind of compression to work perfectly.

Lalam: And the paper suggests that reducing the cost of that compilation step and figuring out which memory structures are best to reuse is the next natural direction for this research.