OC-SORT Tutorial: Critical Insights You Can’t Miss!

In the fast-evolving world of multi-object tracking (MOT), OC-SORT has become a new topic among ML Engineers and researchers.

Its promise is bold: deliver robust, real-time tracking in the face of occlusion and non-linear object motion, two of the most persistent and disruptive challenges in the field.

The method claims to bridge the gap between the Kalman filter-based approaches and the unpredictable realities of real-world scenes, all while maintaining simplicity and speed.

But does OC-SORT truly deliver on its claims, or is it another incremental tweak dressed up as a breakthrough?

This blog will critically examine OC-SORT’s design, its real strengths and the scenarios where it may still struggle.

We’ll look beyond its claims, question its assumptions and see whether this observation-centric approach is truly the robust, plug-and-play tracker the community has been waiting for or if it’s just another step in a long, unfinished journey toward reliable multi-object tracking.

What is Object Tracking?

Object tracking is a fundamental computer vision task that involves automatically identifying and following objects as they move through video sequences over time. 

Unlike object detection, which only identifies objects in individual frames, tracking maintains continuity by assigning unique identities to objects and predicting their locations across consecutive frames.

The process typically begins with object detection to identify targets in the initial frame, followed by motion prediction and data association to maintain consistent tracking throughout the video sequence.

Common algorithms like SORT, DeepSORT, and ByteTrack follow this paradigm, using techniques such as Kalman filters for motion prediction and the Hungarian algorithm for data association between detections and existing tracks

What is OC-SORT and what is its novel approach?

Observation‑Centric SORT (OC‑SORT) is an improved version of the original SORT tracker, which is a popular method for following multiple objects in a video.

Traditional SORT uses a mathematical tool called a Kalman filter (an algorithm that predicts an object’s next position based on its past motion) and assumes that objects move in straight lines at a constant speed.

OC‑SORT changes this by putting observations (the actual detected positions of objects in each frame) at the center of its design instead of relying mostly on straight‑line predictions.

The two main new ideas in OC‑SORT are:

• Observation‑Centric Re‑Update (ORU): When an object disappears behind something else (called an occlusion) and then reappears, OC‑SORT creates a “virtual” path connecting where it was last seen to where it shows up again.
  
  It then “replays” this path to adjust the tracker’s internal state as if the object had been observed the whole time. This back‑filling step helps prevent the tracker’s predictions from drifting too far off course during the occlusion.
• Observation‑Centric Momentum (OCM): Instead of trusting the Kalman filter’s noisy estimate of the object’s direction and speed, OC‑SORT looks at the actual movement between recent observations to figure out which way and how fast an object is really going.
  
  This makes the tracker much better at handling sudden or curvy movements (what we call non‑linear motion).

By leaning on real detection results (“observations”) rather than purely on predicted motion, OC‑SORT stays more in tune with reality, leading to fewer mistakes when objects hide behind others or move unpredictably.

Problems with Previous Tracking Algorithms

Before OC‑SORT, most real‑time trackers like SORT had three main weaknesses:

1. Straight‑Line (Linear) Motion Assumption
   These trackers assume each object moves in a straight line at a constant speed. Over short times this is fine, but if an object turns a corner or swerves, the tracker’s prediction soon drifts away from the true path.
2. Error Build‑Up During Occlusion
   When an object is hidden (for example, walking behind a pillar), the tracker can’t “see” it, so it keeps guessing where it might be.
   
   Without fresh observations to correct its course, small errors add up over several frames, sometimes placing the object far from where it actually reappears.
3. Reliance on Estimated States (Estimation‑Centric)
   Traditional methods lean heavily on the Kalman filter’s internal state (its own guess of position and velocity) instead of the detector’s actual outputs.
   
   Because the filter’s predictions can be noisy, this over‑reliance can cause the tracker to lock onto the wrong spots, especially in crowded scenes where many objects look similar.

How OC-SORT Handles These Problems

OC‑SORT fixes these issues by making actual detection results the “star of the show”:

• Preventing Error Build‑Up
  With the ORU step, whenever an object reappears after an occlusion, OC‑SORT stitches together a fake path between its last known spot and its new detection.
  
  Then it uses that fake path to re‑adjust (or “re‑update”) the Kalman filter’s state for all the missing frames. This back‑correction stops small mistakes from snowballing into big ones.
• Coping with Curvy, Non‑Linear Motion
  The OCM scheme measures how an object has actually moved between recent frames, rather than what the filter predicted, and uses that real movement (“momentum”) when deciding which detection belongs to which track.
  
  As a result, if someone suddenly turns or zigzags, the tracker still follows smoothly.
• Putting Observations First
  By focusing on the detector’s outputs at each frame, OC‑SORT stays grounded in reality.
  
  It only uses the Kalman filter’s predictions as a temporary guide and quickly corrects itself whenever fresh observations arrive. This keeps identity switches (mixing up one object for another) to a minimum, even in busy scenes.

Tracking using OC-SORT

• Tracking people in crowd

• Tracking planes

• Tracking Athlete

• Tracking horses

Here, OC-SORT failed in tracking due to the failure of the Efficient Kalman Filter.

• Tracking car

Here, OC-SORT failed in tracking due to the failure of the Efficient Kalman Filter.

Implementing OC-SORT

You can implement ocsort using the BoxMot library with just a few lines of code.

!pip install boxmot ultralytics

Step 1: Basic OC-SORT Tracking

# Basic OCSORT tracking with YOLO detection
!boxmot track --source your_video.mp4 --tracking-method ocsort --yolo-model yolo12x.pt

Step 2: OC-SORT with ReID Model

# OC-SORT with lightweight ReID model

!boxmot track --source your_video.mp4 --tracking-method ocsort --yolo-model yolo12x.pt --reid-model osnet_x0_25_market1501.pt


# OC-SORT with heavyweight ReID model for better appearance features

!boxmot track --source your_video.mp4 --tracking-method ocsort --yolo-model yolo12x.pt --reid-model clip_market1501.pt

Limitations of OC-SORT

While OC‑SORT improves on many fronts, it still has a few important drawbacks to keep in mind:

• Depends Heavily on the Detector’s Quality
  OC‑SORT leans on the object detector’s output (“observations”) for most of its decisions.
  
  If the detector misses an object or gives a poor bounding box, the tracker has little extra information to fall back on. In other words, “garbage in, garbage out”: bad detections generally lead to bad tracking.
• Simple Virtual Trajectory Assumption
  The Observation‑Centric Re‑Update (ORU) creates a straight‑line path between when an object disappears and when it reappears.
  
  If an object actually took a very twisty or erratic route behind an occluder, that straight-line guess may not match reality, leading to some correction errors.
• No Appearance-Based Checks
  OC‑SORT deliberately avoids using visual appearance (like color histograms or deep neural features) to tell objects apart.
  
  While this keeps it extremely fast, it can struggle when two very similar-looking objects move together, there’s no “visual fingerprint” to help keep their identities straight.
• Challenges in Extremely Crowded Scenes
  In very dense scenes (think hundreds of people crossing a busy street), many objects move in similar ways.
  
  Even with observation‑centric momentum, OC‑SORT can end up swapping IDs when two objects cross closely or occlude each other for many frames.

Conclusion

OC‑SORT makes tracking multiple objects more reliable by using real detections rather than just predicted motion.

It fills in gaps when an object is hidden by drawing a “virtual” path between its last known and re‑seen positions, and it matches objects based on their actual movement instead of noisy estimates.

Because it keeps the simple Kalman‑filter core from SORT, OC‑SORT still runs at hundreds of frames per second on a regular CPU, so it works well for real‑time tasks like surveillance, self‑driving cars, or sports analysis.

However, OC‑SORT has a few limits. It depends on a good object detector, if the detector messes up, the tracker can’t correct itself.

Its virtual path assumes straight‑line motion, which may be wrong if an object moves erratically.

It also ignores appearance features (like color or texture), so similar‑looking objects can get swapped in crowded scenes. Finally, it needs careful tuning of its settings for different situations.

Despite these caveats, OC‑SORT strikes a useful balance of speed, simplicity, and better handling of occlusions and curves.

FAQs

References

OC-SORT Notebook