Vehicle Tracking Across Frame Sequences (Multi‑Class, with Occlusion, Invisibility, Truncation)

Project type: Multi‑object tracking (MOT)
Input: Sequential frames (video or image sequence) from a moving camera (drone)
Output: Tracked trajectories with per‑frame bounding boxes and attributes

Project Overview

Nazru’s AI platform was deployed to track vehicles across sequential frames, assigning a unique track ID to each vehicle and maintaining it over time. The project involved classifying different vehicle types (car, truck, bus, motorcycle, bicycle, etc.) and annotating per‑frame attributes including occluded, invisible, truncated, and others. The output enables temporal reasoning for traffic flow analysis, autonomous driving, surveillance, and fleet management.

The Challenge

Tracking vehicles in video sequences presents several difficulties:

• Appearance changes due to lighting, shadows, weather.

• Occlusions – vehicles partially hidden behind other objects (trees, poles, other vehicles, buildings).

• Invisibility – a vehicle may disappear completely (behind a wall, out of frame) and reappear later; the tracker must maintain or correctly re‑initialise its ID.

• Truncation – vehicles cut off at the edge of the image.

• Different vehicle types require correct classification even when only part of the vehicle is visible.

• Temporal consistency – the same vehicle must keep the same track ID across all frames where it appears.

The client required a high‑accuracy tracking solution that outputs, for each vehicle in each frame:

• A bounding box (2D, axis‑aligned or oriented).

• A unique track ID (consistent across frames).

• Vehicle class (e.g., car, truck, bus, motorcycle, bicycle, van, etc.).

• Per‑frame attributes: occluded (True/False or percentage), invisible (True/False when the vehicle is not visible but still tracked via motion or memory), truncated (True/False), and optionally confidence and uncertain.

The Nazru Solution

Nazru’s platform combined a state‑of‑the‑art multi‑object tracking (MOT) pipeline with a custom attribute annotation layer. The system:

1. Detects vehicles in each frame using a high‑accuracy detector (e.g., YOLOv8, Faster R‑CNN) that outputs bounding boxes and class labels for vehicle types.

2. Associates detections across frames using a combination of:

   • IoU (Intersection over Union) tracking for slow‑moving, densely packed scenes.

   • Re‑identification (ReID) features for vehicles that reappear after occlusion or invisibility.

   • Kalman filtering for motion prediction.

3. Assigns a unique track ID to each vehicle for the entire sequence (or until it leaves the frame permanently). When a vehicle reappears after being invisible or fully occluded, the ReID module attempts to match it to a previous track ID; if no match, a new ID is created.

4. Attributes per frame are automatically predicted or manually annotated:

   • occluded (0–100% or boolean): estimated percentage of the vehicle hidden by other objects.

   • invisible (boolean): True when the vehicle is not visible in the current frame but its track is maintained (e.g., behind a building, then re‑enters). Typically set based on detection confidence or human annotation.

   • truncated (boolean): True when the bounding box is cut by the image boundary.

   • uncertain (boolean): low‑confidence detection or ambiguous track.

   • motion_type (optional): moving, stopped, parked, etc.

5. Outputs a track file (e.g., MOTChallenge format, CSV, JSON) with per‑frame entries: frame_id, track_id, x1, y1, x2, y2, confidence, class, occluded, invisible, truncated, and any other custom attributes.

Key Results & Benefits

• Long‑term tracking across hundreds or thousands of frames, even through complete occlusion or invisibility (via ReID).

• Per‑frame attribute richness enables training of robust tracking models that understand occlusion and truncation.

• Multiple vehicle classes tracked simultaneously (cars, trucks, buses, motorcycles, bicycles, etc.).

• Consistent track IDs – the same vehicle keeps its ID across the entire sequence, enabling trajectory analysis.

• High scalability – the pipeline can process hours of video with hundreds of vehicles.

• Output formats compatible with standard tracking benchmarks (MOTChallenge, KITTI, Waymo Open Dataset).

Key Technologies Used

• Object detection – YOLOv8 / Faster R‑CNN / DETR for vehicle bounding boxes and class labels.

• Multi‑object tracking – SORT, DeepSORT, ByteTrack, or transformer‑based trackers.

• Re‑identification (ReID) – appearance feature extractor (ResNet, OSNet) trained on vehicle images.

• Occlusion & truncation estimation – via bounding box geometry, overlap with other detections, and image boundary checks.

• Invisible state handling – track maintenance using Kalman prediction when no detection is associated for a few frames.

• Temporal smoothing – to reduce ID switches and jitter.