The notebook can be found here: https://colab.research.google.com/drive/1npzkUZijAVJWhkhBYbzSFOigD468_YAT#scrollTo=DTSpNgqPavqK

Change Detection on Video with SLIC Superpixels

This project implements an approach for change detection between consecutive video frames by leveraging SLIC superpixel segmentation instead of noisy, slow pixel-by-pixel comparisons.

The core idea is to first segment each image into meaningful regions (superpixels) and then compare these regions' features and positions to determine if a change has occurred.

---

How It Works

The system compares two consecutive frames, $t$ and $t+1$, through a four-step pipeline:

1. Segmentation with SLIC

The SLIC (Simple Linear Iterative Clustering) algorithm is applied to each frame to cluster similar, nearby pixels into superpixels.

• Gaussian Blurring: A slight Gaussian blur ($\sigma=1$) is applied before segmentation to reduce noise and smooth small texture details, improving superpixel quality.
• Parameters:
  • n_segments: The desired number of superpixels. The value affects the size and precision of regions.
  • compactness ($m$): Balances color similarity ($d_{lab}$) against spatial proximity ($d_{xy}$), controlling the shape of the superpixels. $$D = \sqrt{d_{lab}^2 + m \times d_{xy}^2}$$

2. Feature Extraction

For every superpixel in a frame, a 4-dimensional feature vector is computed, along with its centroid (spatial center).

3. Superpixel Matching & Change Detection

A superpixel in frame $t$ is matched to a superpixel in frame $t+1$ based on a two-stage distance check:

1. Spatial Proximity Filtering: Only candidates in $t+1$ whose centroid is within a maximum spatial distance of the superpixel's centroid in $t$ are considered.
2. Feature Distance Matching: Among the spatial candidates, the best match is the one with the minimum feature distance.

• Change Criteria: A superpixel in $t$ is declared "changed" if:
  • It has no spatially close candidates in $t+1$.
  • The feature distance to its best match is greater than a threshold.
• New Object Detection (Extension): Superpixels in $t+1$ that were never matched to any superpixel from $t$ are considered "new regions" (e.g., a new object entering the scene) and are also flagged as changed.

4. Visualization

A binary mask is created from all "changed" superpixels (including "new regions"). The changes are then visualized on both frames $t$ and $t+1$ by:

1. Displaying the boundaries of all superpixels.
2. Filling the detected changed regions with a semi-transparent red overlay.

---

⚠️ Limitations & Future Work

Current Flaws

• Fast Motion: If an object moves significantly such that its centroid displacement exceeds max_spatial_dist, the matching will fail, and it will be flagged as a change/new object.
• Sudden Appearance/Disappearance: While partially handled by the "new region" detection, sudden object appearance/disappearance can still create segmentation mismatches.
• Segmentation Jitter: Small changes in the SLIC superpixel boundaries between frames can occasionally lead to mismatches, especially with object deformation.

Suggested Enhancement (Future Work)

Integrate Optical Flow:

The current method assumes minimal movement between frames. A significant improvement would be to use Optical Flow (e.g., Lucas-Kanade) to estimate the motion of the pixels between frames $t$ and $t+1$. This could:

1. Predict Motion: Move the centroid of a superpixel in $t$ according to the flow to better predict its location in $t+1$.
2. Improve Matching: Perform the spatial proximity check and feature matching on the predicted location, allowing for successful matching even with significant object motion.