ARC-SORT: Adaptive, Robust, Compensated Multi-Object Tracking with Interacting Multiple Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ARC-SORT: Adaptive, Robust, Compensated Multi-Object Tracking with Interacting Multiple Models".

Jane: The gist The tracker measures its own uncertainty,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper now, ARC-SORT: Adaptive Robust Compensated SORT with Interacting Multiple Models by Girish Kumar Adari Athlitix Inc., and it’s a tracking-by-detection tracker that trains nothing and runs in real time >

Jane: Right, so the title itself suggests it’s not just another standard tracking method, but one that adapts based on its own uncertainty, which sounds pretty smart >

Lu: It does seem to focus on how the tracker measures its own uncertainty and uses that measurement to decide what actions to take online >

Meng: What I want to know is how much of this adaptation we're actually paying for in terms of computational load when we deploy this thing at scale >

Lalam: The core idea is that the tracker couples itself with an Interacting Multiple Model filter, which gives every track a predictive covariance, and they turn that into a per-track search region depending on the reliability of that motion prediction >

Tom: Exactly. They call this Uncertainty-adaptive Expansion IoU, or EIoU, where the less reliable the motion prediction is, the bigger the matching region gets for a detection to match >

Jane: It’s like if you’re tracking someone making a sharp turn in traffic and your prediction is shaky, you give them more space to be seen before you veto them >

Lu: It also handles identity commitment differently by having a mechanism where when two tracks cross, the assignment cost reports whether the decision was close >

Jane: That means if they are nearly identical matches, ARC-SORT marks that pair as entangled and keeps it under review, applying at most one corrective swap per episode if later evidence shows it needs to change >

Meng: So instead of making a potentially wrong commitment right away, it waits for more data before correcting the assignment >

Lalam: And for when tracks disappear for a long time, past a multi-second out-ofview gap, the paper says that motion extrapolation doesn't carry information anymore >

Title and authors: Tom: That’s interesting because usually you just use a motion gate to veto candidates that land too far from where the tracklet is predicted to be next >

Jane: But ARC-SORT drops that motion gate entirely for those long gaps and links tracklets by appearance alone instead, which they call the offline reidentification stage >

Lu: The paper notes that this removal of the motion veto for those long gaps, specifically past a multi-second out-ofview gap, is worth plus four point nine HOTA on SportsMOT validation and transfers to DanceTrack under a different embedding model > <ref:2610.22306#pg1>

Tom: That’s a big gain because it directly tackles the common failure mode of losing tracks when there are long camera exits in sports footage >

Jane: Now, I see they also talk about how using the same covariance for a hard Mahalanobis gate hurts association at every threshold they tested, and richer covariance summaries don't add anything measurable >

Meng: That suggests that just having a fixed set of parameters isn't enough; you need that continuous coupling where the uncertainty feeds back into the search region size >

Lalam: And they found empirically that the scalar relative positional uncertainty is an empirically sufficient statistic for association tolerance, meaning it’s just that one number from the covariance summary >

Tom: So, they are saying you should spend your computational effort on this continuous feedback loop rather than trying to fine-tune a fixed gate threshold >

Jane: It sounds like they are essentially saying the tracker should be more conservative when it’s unsure about its prediction, expanding its search area proportionally to that doubt >

Lu: And for deployment, they mention that using selective mask propagation only where an assignment uncertainty signal fires dispatches the video segmenter only where it needs to be, at a fraction of the cost >

Meng: That makes sense from a production standpoint; you don't want to run expensive vision models on every single frame if you aren't confident about a match anyway >

Title and authors: Lalam: So, to wrap up this section, ARC-SORT is training-free and adapts online by having the IMM covariance set a continuous per-track association tolerance, while the assignment’s permutation margin defers near ambiguous commitments until evidence settles them >

Tom: And where motion information has run out for long gaps, an appearance-only merge replaces the motion-gated one to link tracklets >

Jane: It sounds like they are making decisions based on what the system actually measures in real time rather than relying on pre-set rules that might not fit every situation >

Lu: This approach contrasts with learning motion networks or deep Kalman replacements, which require training per domain, whereas ARC-SORT adapts online through the filter without any training at all >

Meng: But what about the limitations? The paper says deferred commitment is only pairwise and single-swap, and that this mechanism’s gain tracks the embedding stack by fifty-three to one hundred twenty percent >

Lalam: And they also noted that the long-horizon stage is offline, meaning it’s not happening live during tracking >

Tom: Plus, they mention that a tracker evaluated on a leaderboard is evaluated jointly with its detector, and their comparison anchors itself in a detectordeconfounded setting >

Jane: So the implication for us listening right now is that we need to be careful about what we expect from these real-time systems when things get really messy or when the input quality varies a lot >

Lu: The paper shows that this continuous coupling, where uncertainty drives action, is what matters on identical inputs because using the same covariance as a hard Mahalanobis gate hurts association at every threshold they swept >

Meng: That means if you’re building something similar, you can’t just pick one set of parameters and hope for the best; you have to build that adaptive mechanism in >

Lalam: This paper, ARC-SORT: Adaptive Robust Compensated SORT with Interacting Multiple Models, shows how to design a tracker where the tracker measures its own uncertainty and that measurement decides how it acts >

The paper's summary: Tom: So, to recap, ARC-SORT is this tracking system that doesn't need training at all because it uses its own uncertainty to decide what to do next >

Jane: Right, so instead of using a fixed rule for matching or tracking, the entire system couples itself continuously with an Interacting Multiple Model filter >

Lu: The core idea they’re pushing is that this IMM filter gives every single track a predictive covariance, and that uncertainty measurement directly controls how wide the search region is during matching >

Meng: So when you’re tracking something, if your prediction model starts getting shaky—say, during a fast turn—the system automatically gives that track more space to match new detections >

Tom: Exactly. They call this Uncertainty-adaptive Expansion IoU, or EIoU, where the search region expands based on the predictive standard deviation supplied by that IMM filter >

Jane: It’s like if you’re tracking a car on a bumpy road and your prediction gets noisy, you don't just keep a tight box; you dilate that box to be more forgiving when matching things >

Lu: And they also have this thing called Deferred Identity Commitment, which deals with when two tracks cross paths and it’s not immediately clear if they’re the same thing >

Tom: They flag those ambiguous pairs as entangled, so they don't make a final decision right away; they hold them for a moment to collect more evidence >

Jane: It means if the evidence later shows one track was actually the correct match, it can perform at most one corrective swap per episode to fix that mistake >

Meng: That sounds like something you’d want in production because it handles those close calls where things get momentarily confusing without crashing the whole system >

Lu: And for when tracks disappear for a long time, past a multi-second gap, they ditch the motion gate entirely and link them by appearance alone >

Tom: That’s the long horizon corollary; once you lose track of where it *should* be based on movement, you just rely on what it looks like when it reappears >

Jane: So if you’re watching a video of a player walking off-screen for several seconds, ARC-SORT won't try to guess the motion across that gap and veto any potential matches >

Lu: It just drops the motion gate and lets the appearance data link those tracklets together later when they come back in view >

Meng: That offline linking by appearance is pretty smart because it saves you from having to run a complex, high-cost motion prediction model over every single frame for that long period >

Tom: So these mechanisms—the continuous uncertainty feedback, the deferred commitment, and the appearance-only merge—they are what they claim gives them a big boost in accuracy >

Jane: They say this method actually leads to gains of plus one point zero HOTA on both test servers compared to systems that don't have these adaptive features >

Lu: The authors found that the scalar relative positional uncertainty is enough for association tolerance, which means you don't need a complicated covariance analysis; just that one number matters >

Tom: That’s a clean win because it simplifies the implementation; you focus on using the covariance as tolerance rather than trying to build a perfect hard gate >

Jane: But they also have some caveats, like deferred commitment is only pairwise and involves at most one swap per episode, and the long horizon stage is offline, which means it doesn't happen live during tracking >

Meng: Those are the practical limits; you can’t expect this to be a perfect online system if you need instantaneous corrections or if you have very long gaps where you need real-time merging >

Lu: And they also noted that on some datasets, like DanceTrack, their performance is well below what specialists have achieved because the detector setup matters a lot there >

Tom: So it’s not a universal fix for every single tracking problem; it depends on the specific data and how you set up your initial detection pipeline >

Jane: Still, this paper shows that when you design your system around measuring its own uncertainty and letting that measurement govern the action, you get a more robust tracker >

Lu: This continuous coupling where uncertainty drives action is what they’re focusing on because it works even on identical inputs, which is something fixed rules can't handle well >

The paper's improvements: Tom: So we're looking at how they suggest improving ARC-SORT, and it all boils down to making those decision points even smarter >

Jane: Right, so they’re not just settling for one way to handle uncertainty; they’re layering these adaptive mechanisms on top of the core tracker >

Lu: One major suggestion is about that long horizon stage; instead of just linking by appearance, they propose using a linear interpolation of gaps less than ten frames to keep the motion information alive longer >

Tom: That means for those medium-sized gaps, you don't have to completely switch to purely visual matching; you can still use some motion extrapolation when the gap is small enough >

Jane: It’s about bridging that gap between purely visual and purely motion-based tracking without having a hard cutoff point in the middle of the tracking >

Meng: From a deployment standpoint, that makes sense because it reduces the risk of losing tracks during those intermediate moments where motion is still somewhat predictable >

Lu: Another improvement they push is making that deferred commitment mechanism more flexible; instead of just one swap, they suggest allowing a few corrective swaps if later evidence strongly points to a different identity >

Tom: So it’s about giving the system more "wiggle room" to fix its initial guess if the data later proves it wrong, which sounds like it handles messy real-world data better >

Jane: It means in situations where ambiguity is high, ARC-SORT can afford a little more correction than just a single swap allows >

Lu: And they also touch on how the uncertainty expansion factor itself could be made more dynamic based on the type of motion detected, not just a fixed function of the standard deviation >

Tom: That would mean if you see very erratic movement, the system automatically knows to expand its search region much wider than if it sees smooth movement >

Jane: It’s about making the adaptation itself more responsive to *what* kind of uncertainty you are dealing with in a specific scenario >

Meng: If we could tune that factor based on motion characteristics, it would help us manage the computational cost because we wouldn't be over-expanding when things are predictable >

Lu: The overall implication is they want to move away from fixed expansion and toward a system where the adaptation rules change based on the immediate context of what’s happening in the scene >

Tom: It sounds like they’re aiming for a tracker that feels less like it's following a rigid formula and more like it's actually thinking about its own confidence level >

Jane: This whole approach is really about moving toward models that are inherently more cautious when they aren't sure, which is something we need in complex tracking scenarios >

Lu: They’re emphasizing that the gain comes from this continuous adjustment, not some one-off tweak to the initial parameters >

Conclusion: Tom: So to wrap up, ARC-SORT is this tracking system that doesn't need training because its own uncertainty dictates every move it makes >

Jane: Right, so this paper shows how you can couple an Interacting Multiple Model filter directly into the decision-making process for a tracker >

Lu: It’s basically a continuous feedback loop where the IMM filter provides predictive covariance, and that measurement sets the association tolerance for matching regions online >

Meng: I still have to ask about deployment because if it needs that continuous covariance read per track, how much CPU overhead are we talking about when we’re running this on a fleet of edge devices >

Lalam: Based on the literature, I'm pretty sure that one covariance read is the cost, and since it's constant time relative to the number of detections, it should scale fine >

Tom: Yeah, but remember that there are limitations; they noted that deferred commitment is only for a single swap and the long horizon stage is offline >

Jane: So while this method gets a big boost on benchmarks like SportsMOT, we have to be careful about real-world deployment constraints regarding those specific limits >

Lu: The authors also pointed out that on certain datasets, like DanceTrack, the performance still lags behind what trained specialists can achieve because the base detector matters too >

Meng: That makes sense; it’s a tracking method, but it’s only as good as the detection feeding it data; we have to make sure our detector is solid before we trust any tracking result >

Tom: So, this paper, ARC-SORT: Adaptive Robust Compensated Multi-Object Tracking with Interacting Multiple Models, shows how uncertainty can actually drive the system's behavior >

Jane: It’s a great example of using internal measurements to make external decisions in a way that adapts to the input data it’s seeing >

Lu: The continuous coupling between motion prediction and search region size is what I find most compelling because it addresses how systems handle different levels of motion confidence dynamically >

Meng: I guess for us, the implication is that we need to design our AI pipelines not just for accuracy on a test set but for robustness when the input quality varies wildly in production >

Tom: Exactly. We need systems that can self-regulate their behavior based on real-time feedback instead of relying on static thresholds >

Jane: That's the big picture here, moving toward trackers that are more aware of their own limitations than just following a set of hard rules >