Gesdraw - Hand Gesture Drawing Application

Note: This is an older project. While we’ve added basic macOS compatibility and refreshed dependencies, some features may not work perfectly on modern systems (e.g., screen capture permissions, camera initialization, or voice components). See Troubleshooting for common fixes.

A hand gesture-powered drawing application that allows you to draw using your webcam and hand movements. The app supports three different drawing modes and includes voice commands for hands-free operation.

Features

• Three Drawing Modes:

  • New Canvas: Draw on a blank canvas
  • On Screen: Draw over a live screenshot of your screen
  • Existing File: Draw over a user-selected image

• Hand Gesture Controls:

  • Two fingers up: Selection mode (choose colors, tools)
  • One finger up: Drawing mode
  • Gesture-based toolbar with color selection, eraser, clear, save, and send options

• Voice Commands:

  • Color selection: "red", "blue", "green", "eraser"
  • Actions: "save", "send", "clear"
  • Brush/eraser thickness: "brush thickness 20", "eraser thickness 50"

• Export Options:

  • Save drawings as images
  • Send drawings to Word documents with timestamps

Installation

1. Clone or download this repository
2. Install the required dependencies:

pip install -r requirements.txt

Note for macOS users: You may need to install pyaudio separately if you encounter issues:

brew install portaudio
pip install pyaudio

Usage

1. Run the main application:

python MainScreen.py

2. Choose your drawing mode:

   • New Canvas: Start with a blank canvas
   • On Screen: Draw over your current screen
   • Existing File: Select an image file to draw over

3. Drawing Controls:

   • Hold up two fingers to enter selection mode and choose tools/colors from the top toolbar
   • Hold up one finger to draw
   • Use voice commands by selecting the voice assistant option in the toolbar

4. Keyboard Shortcuts:

   • Press s to save the current drawing
   • Press q to quit the drawing mode

System Requirements

• Python 3.7+
• Webcam
• Microphone (for voice commands)
• macOS, Windows, or Linux

File Structure

• MainScreen.py - Main GUI launcher
• VirtualPainterNew.py - New canvas drawing mode
• VirtualPainterScreen.py - Screen overlay drawing mode
• VirtualPainterFile.py - File overlay drawing mode
• HandTrackingModule1.py - Hand tracking functionality
• assistant.py - Voice command processing
• DocSave.py - Word document export functionality
• guiimages/ - UI button images
• images3/ - Toolbar overlay images
• SavedImages/ - Exported drawings
• SavedDocuments/ - Generated Word documents
• tempimgs/ - Temporary images for document export

Troubleshooting

macOS Issues

• If voice commands don't work, ensure microphone permissions are granted
• If you get audio errors, try installing pyaudio via Homebrew first

Camera Issues

• IMPORTANT: Grant camera permissions in System Preferences > Security & Privacy > Camera
• Ensure your webcam is not being used by another application
• The app expects a 1280x720 resolution; lower resolutions may cause issues
• If you see "OpenCV: not authorized to capture video", you need to grant camera permissions

Voice Recognition Issues

• Speak clearly and ensure a quiet environment
• Check that your microphone is working and permissions are granted

Python Version Compatibility

• This app works with Python 3.12 (Anaconda recommended)
• Mediapipe 0.10.21 is used for compatibility with Python 3.12
• If you encounter numpy version conflicts, they shouldn't affect core functionality

Dependencies

See requirements.txt for the complete list of required Python packages.

License

This project is open source. Feel free to modify and distribute as needed.