🧤 Dataglove: Real-Time Object Recognition with the VMG30 Sensor Glove

This project enables real-time object recognition using the VMG30 sensorized glove by Virtual Realities.
Sensor data from the glove is processed using a deep learning model that classifies hand gestures and object interactions in real time.

This repository has also been a playground for experimenting with continual learning techniques, allowing the model to incrementally learn new object classes without forgetting previously learned ones.

---

Neural Network Model: A-CBLN

At the core of the system is an A-CBLN model:

Attention-based Convolutional Bidirectional LSTM Network
Inspired by the PLOS ONE paper.

---

Repository Structure

ros2_ws/
├── src/
│ ├── dataglove/ # ROS 2 nodes: data acquisition + real-timeclassification
│ ├── dataglove_msg/ # Custom ROS 2 messages for glove data
│ └── nn/ # Neural network code: training, evaluation, continual learning
├── datasets/ # datasets gathered by me
├── presentation/ # slides to present the project
├── images # picture of glove, object used, and graphs

• dataglove/
  ROS 2 nodes for:

  • Reading data from the VMG30 glove
  • Publishing it as ROS messages
  • Running real-time classification

• dataglove_msg/
  Custom message definitions used to structure glove sensor data in ROS 2.

• nn/
  Python code for:

  • Training, validating, and testing the neural network
  • Implementing continual learning methods like:
    • Learning Without Forgetting (LwF)
    • Dark Experience Replay (DER)
  • Preprocessing, model saving/loading, and experiment tracking

---

Build & Run Instructions

Make sure you have sourced your ROS 2 environment (and Python virtual environment, if used).

Build the ROS 2 Packages

colcon build --packages-select dataglove dataglove_msg
source install/setup.bash

Run the System

Start the glove data publisher:

ros2 run dataglove glove

Start the real-time classifier:

ros2 run dataglove real_time_classifier

---

Training & Continual Learning

To train a new model on recorded glove data:

cd src/nn/
python3 train.py
cp checkpoint.pt ../dataglove/dataglove

Make sure to install the required Python dependencies:

pip install -r requirements.txt

The nn/ directory also includes: Continual learning strategies like:

• lwf.py for Learning Without Forgetting

• der.py for Dark Experience Replay

Learning Whithout Forgetting: see paper ArXive
Dark Experience Replay: see ArXive