From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning".

Jane: The paper was written by Manish Kumar, Anton Frederik Thielmann, Christoph Weisser and Benjamin Säfken from BASF and Clausthal University of Technology and Amazon Music and Bielefeld School of Business, Hochschule Bielefeld.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Alright, welcome back to the show, everyone! Today we're digging into a paper with a title that's a mouthful: "From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning." Jane, when you first saw that title, what jumped out at you?

Jane: Tom, it was the word "knots" that got me. In math, a knot isn't about tying rope. It's a point where a curve changes its shape, like where a piecewise function bends. And the paper is asking whether we should just pick those bend points randomly, or let the model learn where they should be.

Lu: Exactly, Jane. And that's a bigger deal than it sounds. Tabular data—think spreadsheets, customer records, sensor readings—is everywhere in industry. But deep learning models often struggle with it because they treat each number as a single scalar. This paper says, what if we expand each number into a whole vector of spline values first?

Meng: So instead of feeding the model the raw number "forty-two" you're feeding it a richer representation that encodes where "forty-two" falls relative to those knots. That's the encoding part. But my first question as an engineer is always, does this actually help, or is it just adding complexity?

Tom: That's exactly what the authors set out to test. They ran over five thousand training runs across twenty-five datasets, three different backbones, and a whole zoo of encoding methods. They weren't just theorizing; they built a massive benchmark to see what actually works.

Jane: And the title hints at the journey. "From Uniform to Learned Knots" means they started with the simplest approach—evenly spaced knots—and then asked if letting the model move those knots during training, through backpropagation, would give better results. That's the "learned" part.

Lu: It's a natural progression. In classical statistics, free-knot splines were always a pain because optimizing knot locations is a nasty nonconvex problem. But with modern deep learning and differentiable parameterizations, you can make it work. The authors used a softmax-cumsum trick to keep knots ordered while still allowing gradients to flow.

Meng: I like that they kept the downstream models fixed—MLP, ResNet, FT-Transformer—so the only variable was the encoding. That's clean experimental design. It means any performance difference you see is genuinely from the preprocessing, not from some architectural tweak.

Tom: And the results? They're not a simple "one method wins everything." That's what makes this paper interesting. The best encoding depends on whether you're doing regression or classification, which backbone you're using, and even how many basis functions you allow. It's a nuanced picture.

Jane: Which is a great hook for what's coming next. We need to dig into what they actually found, because the summary is where the surprises start.

Summary: Jane: So we've set the stage with the title. Now let's talk about what this paper actually found. Tom, the headline result really surprised me.

Tom: Mine too, Jane. For classification, the humble Piecewise Linear Encoding—PLE—was the most robust choice across the board. It topped the critical difference diagrams at every output size we looked at. The spline methods were competitive, but they didn't consistently beat this simpler baseline.

Jane: But for regression, it was a completely different story. There was no single winner. The best method depended heavily on the output size—how many basis functions you allowed per feature. At small sizes, B-spline variants like BS-LGBM and BS-Q led the pack. At larger sizes, I-splines and the learnable-knot variants started to shine.

Lu: That's a fascinating asymmetry. It suggests that classification tasks, with their threshold-like decision boundaries, are naturally suited to PLE's piecewise-linear structure. Regression targets, on the other hand, are often smoother, and splines—especially integrated splines—can capture that smoothness more efficiently.

Meng: And the backbone mattered too, right? I remember the FT-Transformer was a different beast entirely. For that model, standard scaling was often just as good as any fancy encoding, and sometimes better. That's a practical insight: if you're using a powerful transformer, you might not need to bother with spline preprocessing at all.

Tom: Exactly, Meng. The gains from expressive encodings were much clearer for MLP and ResNet. For FT-Transformer, the benefits were smaller and less consistent. The paper even showed that increasing the output size could hurt FT-Transformer performance in regression, which is a cautionary tale.

Jane: And they didn't just look at performance. They did an efficiency case study on the SGEMM dataset. Learnable knots add parameters, but the real cost was in training time. B-spline learnable knots were relatively cheap, but M-spline and I-spline versions got dramatically slower as the output size grew.

Lu: That makes sense from the math. B-splines have local support—each basis function only depends on a few nearby knots. But I-splines are integrals, so they have cumulative dependence. A small change in one knot can affect many basis functions downstream, which makes the backward pass much more expensive.

Meng: So the practical takeaway for me is: if you want learnable knots, stick with B-splines unless you have a really good reason not to. The performance gains from I-splines might not justify the computational overhead.

Tom: And that's the kind of practical guidance that makes this paper valuable. It's not just "splines are great." It's a map of when and where each encoding strategy pays off. Which brings us to the improvements they suggest—what does this mean for the field going forward?

Improvements: Tom: We've covered the results, but what really excites me is where this paper points the field. Jane, what do you think the biggest improvement is?

Jane: For me, it's the learnable-knot parameterization itself. The authors showed you can optimize knot locations end-to-end in a stable way, using that softmax-cumsum trick to keep them ordered. That's a technical contribution that opens the door for much more adaptive preprocessing.

Lu: And it's a significant step beyond the fixed-knot approaches that dominated before. In classical spline regression, free-knot methods were notoriously fragile. The authors here introduced a collision-avoidance regularization term that penalizes knots getting too close together, which keeps the basis well-conditioned. That's a clever solution to a long-standing problem.

Meng: But let's be practical. The paper also showed that learnable knots can substantially increase training cost, especially for M-splines and I-splines. So the improvement isn't just about making it work—it's about making it work efficiently. The B-spline variant is the sweet spot.

Tom: Right, and the paper also suggests a bigger picture improvement: moving away from a one-size-fits-all approach to numerical encoding. They found that the best method depends on the task, the backbone, and the output size. So the future might be about adaptive selection—choosing the right encoding for each feature, or even learning that choice.

Jane: That's a great point. The authors explicitly mention feature-specific encoding as a future direction. Instead of applying the same spline family and output size to every numerical feature, you could let the model decide which features need more resolution and which ones are fine with a simple scaling.

Lu: And there's room for other basis families too. They mention thin-plate splines and radial basis functions as unexplored territory. Those have different properties—more global support, for example—which might be better suited for certain types of tabular data.

Meng: I'd also love to see this integrated with automated machine learning pipelines. Imagine an AutoML system that not only searches over architectures but also over numerical encodings, using the results from this paper to narrow the search space intelligently.

Tom: That's a compelling vision. The paper gives us a solid foundation—a benchmark, a methodology, and a set of findings—that future work can build on. It's not the end of the story; it's more like the first detailed map of a territory we're just starting to explore.

Jane: And that's exactly what good research should do. It answers some questions but opens up many more. The improvements here are both technical and conceptual, and they're going to shape how people think about tabular deep learning for years to come.

Conclusion: Jane: Well, we've had a great discussion about "From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning." Tom, how would you wrap this up for our listeners?

Tom: I'd say the big message is that numerical encoding is not a trivial preprocessing detail. It's a modeling choice that can have a real impact on performance. The paper shows that spline-based encodings, especially with learnable knots, are a powerful tool—but they're not magic. You have to match the encoding to the task and the backbone.

Jane: And the practical guidance is clear. For classification, PLE is your safest bet. For regression, you have more options, and you should experiment with output size and knot placement. And if you're using a strong transformer like FT-Transformer, don't assume you need fancy encodings at all.

Lu: The methodological contribution is also important. The differentiable knot parameterization with the spacing penalty is a robust way to do free-knot spline optimization in deep learning. That's going to be useful beyond just tabular data—it could apply to any model that uses spline-based components.

Meng: And from an engineering standpoint, the efficiency analysis is invaluable. Knowing that B-spline learnable knots are relatively cheap while I-splines can be expensive helps us make informed trade-offs between accuracy and compute.

Tom: So we're saying goodbye to this paper with a clear picture: it's a comprehensive, well-executed study that gives us both new tools and new understanding. It's the kind of work that will be cited for years as the reference point for numerical encoding in tabular deep learning.

Jane: Absolutely. And it leaves us excited for what's next. Feature-specific encodings, new basis families, integration with AutoML—there's a lot of fertile ground here. We'll be watching for the follow-ups.

Tom: Thanks for joining us, everyone. We'll see you next time with another paper from the arXiv.

Jane: Take care, and keep learning!