Run SAM only when your tracker is uncertain – #1 on SportsMOT

This started when I saw a Roboflow tutorial tracking basketball players by giving every player's box to SAM2 and propagating masks through the whole clip (https://www.youtube.com/watch?v=yGQb9KkvQ1Q). Identity preservation worked way better than any dedicated tracker I'd used before but it only ran at 1–2 fps, where both speed and memory was scaling with player count.

Most frames don't need SAM at all though. A normal tracking-by-detection tracker matches boxes frame to frame by solving an assignment problem on a cost matrix, and for most of a game that matching is completely trivial. It breaks down when players get close and two assignments look almost equally good, which you can see in the cost matrix because best and second-best assignment are equal or close to equal. I define that as the assignment margin, and this margin acts as a signal/cue for using SAM.

So the system runs a lightweight tracker (Deep-EIoU, but could use any tracker with a cost matrix e.g SORT or ByteTrack) on every frame and only brings in SAM 3 when a margin collapses. SAM gets seeded a few frames earlier, on a recent frame where the player was still clearly separated, propagates the mask through the crossing, and once things calm down we check which box the mask ended up in. If it settled into a different track than it was seeded on, the tracker swapped identities somewhere in the pile and we rename them.

There's no training anywhere, and both the tracker and SAM are used as black boxes. When I swapped SAM 2 for SAM 3 the whole system improved without touching anything else. It's currently #1 on SportsMOT with 87.2 HOTA.

Code: https://github.com/holma91/selective-mask-propagation, paper: https://arxiv.org/abs/2606.13033, leaderboard: https://www.codabench.org/competitions/13077/#/results-tab. Happy to answer questions!