A hand gesture letter speller application so kids can learn how to spell their name using the dutch gesture language alphabet.
The NGT speller helps children learn to spell the names of their deaf classmates and in general allows them to learn how to spell full words in the Dutch gesture language. Further developed version from tool from Julian. This version makes use of collected frames of a gesture, where the keypoints of the hand are extracted and transformed to npy files on which a LSTM-model is trained. The model is used to find the spelled letter in the application
The project is mainly build upon google's mediapipe package (https://ai.google.dev/edge/mediapipe/solutions/vision/gesture_recognizer), by making use of keypoints of the hand extracted with mediapipe, you can create your own dataset (or train on the data in the folder sequences) and make use of the application.
The following python version is needed;
Python 3.10.19To run the full application, so also train the model and collect the data (the hand gestures required for a the spelling of a given letter) two python environments are needed, the packages/requirements for the environments are in the env folder, with the two names of the environment.txt's as requirements for said environment. If you do not want to train the model you can skip the train environment and just download the packages for the live environment and go straight to main_app.py
After installing the environment:
NOLAI_gesture_liveyou can collect data with collect_data.py and run the main application main_app.py.
The environment for training is:
NOLAI_gesture_trainIn this environment the you can train the model with train_model.py and convert the model afterwards into an ONNX model with convert_onnx.py, so it is compatible with the previous environment and no dependency issues can take place.
After collecting data and training the model the main_app.py can be ran, or you can the tests in the test folder (these were some experiments that eventually became the main application).
The frontend was build using PyQt6, using the "housestyle" of NOLAI. The PyQt package is included in the NOLAI_gesture_live environment.
Data was collected by my own hand gestures and by two people who are very familiar with spelling in the dutch sign language. One of them being a sing language translator. Most letters needed about 50 samples to be correctly captured by the model. The letters with a moving component: the H, J, U, X and Z needed around 120 samples to be actually captured. Some letters also needed some more samples due to their similarity; the N and the M for instance where trained with 70 samples.
Important note: some letters have a slightly different orientation. This is due to these letters being either very similar to one another; so the decision was taken to change the orientation a bit so the keypoints could be seperated for the letter in a better way. This is the case for the T which now has a left orientation instead of a frontal one (to avoid conflicts with the F). It is also the case that the translator spelled the letter R upright instead of how the letter is shown in the reference picture in the application, thus these do not match.
-Mediapipe very good package for keypoint extraction of the hand, additionally mediapipe is very useful for both poses and facial keypoints recognition (however out of the scope of this project) -Possible further usecases might be in the realm of gesture languages in general, mediapipe allows for great recognition of hand keypoints, so if there in the future might be a co-implementation project regarding NGT or something, this prototype might be a great starting point to further investigate usage for more broad gestures (e.g. actual words)