AYAOSHIMA.MPYCPP
Back to Blog

Advanced Object Tracking with SSD MobileNet V3 and Kalman Filter

Introduction

Real-time object tracking is a fundamental challenge in computer vision, essential for applications such as surveillance, autonomous systems, and human-computer interaction. This article details my implementation of an object tracking pipeline using:

  • SSD MobileNet V3 for real-time object detection.
  • Kalman Filter for smooth and predictive tracking.
  • OpenCV + TensorFlow for integration and visualization.

The goal is to reduce detection noise, handle occlusions, and enable persistent tracking across frames.

Implementation Workflow

My pipeline integrates real-time object detection with state estimation via Kalman filtering. Below is a breakdown of the architecture:

  1. Object Detection (SSD MobileNet V3)
    • Loaded the pre-trained SSD MobileNet V3 (COCO-trained) model using TensorFlow.
    • Performed real-time inference on video frames using optimized TensorFlow graph execution.
    • Extracted bounding boxes, confidence scores, and class labels.
  2. Tracking Initialization
    • Used the object centroids from SSD MobileNet as initial measurements.
    • Mapped bounding box coordinates to state vectors for Kalman filtering.
  3. Kalman Filtering for State Estimation
    • Designed a constant velocity motion model for tracking.
    • Defined state vectors: Position (x, y), velocity (vx, vy), and acceleration (ax, ay).
    • Implemented Predict and Update Steps:
      • Prediction: Estimates next object position based on previous motion.
      • Update: Adjusts prediction using SSD MobileNet’s detection.
  4. Bounding Box Association (IoU + Hungarian Algorithm)
    • Used Intersection over Union (IoU) to match detected objects with existing trackers.
    • Applied the Hungarian Algorithm for optimal multi-object assignment.
  5. Visualization & Optimization
    • Drew smoothed bounding boxes over time for consistent tracking.
    • Used OpenCV to overlay tracking IDs and confidence scores.
    • Optimized performance with batch processing and TensorFlow Lite (TFLite).

Optimization & Key Challenges

To ensure smooth and efficient object tracking, I tackled the following challenges:

  • Handling Occlusions:
    • Used Kalman Filter to predict object motion even when detections were lost.
    • Implemented an adaptive threshold to reassign lost objects when they reappear.
  • Reducing False Positives:
    • Applied confidence thresholding to ignore weak detections.
    • Incorporated Non-Maximum Suppression (NMS) to remove duplicate bounding boxes.
  • Improving Speed & Latency:
    • Converted model to TFLite for mobile-friendly execution.
    • Ran inference on GPU using TensorRT optimization.

Reference: Handwritten Notes

Below are my handwritten notes on **Kalman Filter state estimation, covariance updates, and measurement corrections** used in this project. These illustrate the mathematical foundation behind the object tracking pipeline.

State Vector & Prediction

Covers state transition matrix, predicted state vector, and covariance propagation.

Kalman Gain & Update Equations

Describes Kalman Gain computation, uncertainty reduction, and update covariance matrix.

Measurement & Correction

Explains measurement mapping, residual calculation, and final correction step.