NSER-IBVS: Efficient Self-Supervised Neuro-Analytic Visual Servoing for Real-time Quadrotor Control

Sebastian Mocanu, Sebastian-Ion Nae, Mihai-Eugen Barbu, Marius Leordeanu

A self-supervised neuro-analytical framework for vision-based quadrotor control. This repository implements NSER-IBVS (Numerically Stable Efficient Reduced Image-Based Visual Servoing) with a teacher-student architecture that enables autonomous drone positioning without fiducial markers or GPS.

Table of Contents

• Overview
• Features
• Quick Start
• Documentation
• Pre-trained Models
• Hardware Requirements
• Results
• Citation

Overview

This framework introduces a cost-efficient approach to visual servoing where a compact 1.7M parameter student network learns automatically from an analytical NSER-IBVS teacher. The system achieves:

• 11x faster inference compared to the teacher pipeline (540 FPS vs 48 FPS)
• Marker-free operation using learned visual features
• Simulation-to-reality transfer with minimal real-world fine-tuning
• Real-time onboard deployment capability

The teacher employs a two-stage neural network pipeline (YOLOv11 for segmentation + U-Net mask splitter for target orientation estimation) combined with a numerically stable IBVS control law. Through knowledge distillation, the student learns to directly output velocity commands from RGB images.

Features

• NSER-IBVS Teacher: Solves numerical instabilities in classical IBVS through reduced equations and stable keypoint ordering
• Two-Stage Segmentation: YOLOv11 Nano (2.84M params) + Mask Splitter (1.94M params) for robust anterior-posterior vehicle segmentation
• Lightweight Student: Direct end-to-end control from camera images without explicit visual servoing computation
• Digital Twin: Parrot Sphinx simulator with custom UE4 bunker environment for training data generation
• Modular Design: Separate packages for detection, IBVS control, PID filtering, and neural network inference

Repository Structure

nser-ibvs-drone/
├── nser_ibvs_drone/          # Main package
│   ├── controllers/          # Drone control implementations
│   ├── detection/            # YOLO and mask splitter engines
│   ├── distiled_network/     # Student network architecture and engines
│   ├── evaluation/           # Results evaluation with inference, flops and memory analysis
│   ├── ibvs/                 # NSER-IBVS control law and math functions
│   ├── pid/                  # PID controllers and filters
│   ├── processors/           # Frame processing pipelines
│   ├── simulator/            # Simulator experiments suite running pipeline
│   ├── utils/                # Project absolute paths generation and camera parameters loading
│   └── visualization/        # Plotting and analysis tools
├── assets/
│   ├── camera_parameters/    # Intrinsic matrices for different setups
│   ├── environment/          # Custom UE4 environment configs and mesh models
│   └── reference/            # Goal pose images and keypoints
├── config/
│   ├── pid/                  # PID tuning parameters
│   └── simulator/            # Experiment scenario configurations
├── external/                 # Git submodules
│   ├── drone_base/           # Architecture and boilerplate for drone communication
│   ├── drone_sim_runner/     # Simulation orchestration
│   └── mask_splitter/        # Mask splitting network and annotation tool
├── models/                   # Pre-trained model weights
├── runnable/                 # Entry point scripts
├── scripts/                  # Shell scripts for simulator environment setup
├── student_train_pipeline/   # Student network training code
└── docs/                     # Documentation and implementation details

Quick Start

Installation

# Clone with submodules
git clone --recursive https://github.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone.git
cd nser-ibvs-drone

# Create virtual environment
python3 -m venv ./venv
source venv/bin/activate

# Install dependencies
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

Update submodules

git submodule update --remote --merge

For detailed installation instructions including Parrot Sphinx setup, see the Installation Guide.

Running Experiments

Simulator (requires Parrot Sphinx)

# Run automated test suite across all 8 starting poses
python runnable/run_simulator_experiments.py \
    --sphinx_bunker_base_dir=/path/to/UnrealApp.sh \
    --target_runs=5

# Run with student network
python runnable/run_simulator_experiments.py \
    --sphinx_bunker_base_dir=/path/to/UnrealApp.sh \
    --target_runs=5 \
    --is_student

Real-World

python runnable/run_ibvs_splitter.py --is_real_world --experiment_name="real-ibvs-test"
python runnable/run_student.py --is_real_world --experiment_name="real-student-test"

For additional details see running-experiments.md.

Documentation

• Installation Guide - Complete setup instructions
• Architecture Overview - System design and components
• Running Experiments - Detailed experiment and runnable/ scripts guide
• Configuration Reference - All configuration options
• Simulator Launcher Scripts - Simulator launching instructions

Pre-trained Models

Pre-trained models are available in the models/ directory:

Model	Parameters	Description
29_05_best__yolo11n-seg_sim_car_bunker__all.pt	2.84M	YOLOv11 Nano segmentation (simulator)
real-yolo-car-full-segmentation.pt	2.84M	YOLOv11 Nano segmentation (real-world)
mask_splitter-epoch_10-dropout_0-low_x2-and-high_x0_quality_early_stop.pt	1.94M	Mask splitter (simulator)
mask_splitter-epoch_10-dropout_0-_x2_real_early_stop.pt	1.94M	Mask splitter (real-world)
student_model_sim_on_real_world_distribution.pth	1.7M	Student network (sim + real normalization)
student_real_pretrained_augX3_80_runs.pth	1.7M	Student network (real-world fine-tuned)

Final trained models are also available on hugging face brittleru/nser-ibvs-drone.

Hardware Requirements

Simulation:

• OS: Ubuntu 22.04/24.04 or Debian 11
• GPU: NVIDIA with CUDA support (recommended)
• RAM: 8GB minimum, 32GB recommended
• Storage: ~30GB

Real-World:

• Parrot Anafi 4K drone
• Standard laptop with WiFi and GPU preferably
• Indoor environment with Lambertian floor surface (to avoid drone drifting)

Results

To reproduce the results check runnable/ directory and running-experiments.md for additional details.

Performance comparison between teacher (NSER-IBVS) and student networks:

Metric	Teacher	Student
Inference Speed	48.3 FPS	540.8 FPS
Mean Error (Sim)	29.76 px	14.26 px
IoU (Sim)	0.522	0.752
Mean Error (Real)	29.96 px	33.33 px
IoU (Real)	0.627	0.591

The student achieves 11x faster inference while maintaining comparable accuracy, making it suitable for real-time onboard deployment.

Trajectories teacher vs student from two starting points:

Front View Trajectories

Up View Trajectories

Control commands and error evolutions over time:

Real-World Flight - Teacher (IBVS)

Real-World Flight - Student

Digital-Twin Flight - Teacher (IBVS)

Digital-Twin Flight - Student

Altitude and speed over time. Trajectory of the drone from the starting point to the goal, and drone flight states data:

Real-world vs Digital-twin distribution of commands, linear and angular (yaw) velocities, over all experiments comparison:

Citation

If you find this work useful, please consider citing:

@InProceedings{Mocanu_2025_ICCV,
    author    = {Mocanu, Sebastian and Nae, Sebastian-Ion and Barbu, Mihai-Eugen and Leordeanu, Marius},
    title     = {Efficient Self-Supervised Neuro-Analytic Visual Servoing for Real-time Quadrotor Control},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
    month     = {October},
    year      = {2025},
    pages     = {1744-1753}
}