Natural language control for macOS powered by FunctionGemma and Ollama. Control system settings, automate tasks, and manage apps using plain English. Runs 100% locally—no cloud or Internet required to run this.
- 🗣️ Natural Language Interface - Control your Mac using plain English
- 🔒 100% Local - No data leaves your machine, powered by Ollama
- 🤖 FunctionGemma Integration - Google's efficient function-calling model
- 🖥️ 45 macOS Functions - System control, GUI automation, app integration
- 🌐 Web Interface - Beautiful Streamlit UI for easy access
- ⚡ Fast & Lightweight - Runs on Apple Silicon and Intel Macs
- Volume and brightness adjustment
- Dark/Light mode switching
- Do Not Disturb toggle
- Screenshot capture
- Mouse clicks, drags, and position tracking
- Keyboard typing and shortcuts
- Screen scrolling
- Open/close applications
- Create reminders
- Control Music playback
- Access calendar events
- Send iMessages
- Mathematical calculations
- Statistical analysis
- QR code generation
- Wi-Fi network scanning
- Text file analysis
User Input (Natural Language)
↓
FunctionGemma via Ollama
↓
Function Mapping & Parameter Extraction
↓
macOS Function Execution (AppleScript + Python APIs)
↓
Result Processing & Response Generation
- macOS 11.0+ (Big Sur or later)
- Python 3.12+
- Ollama installed and running
- FunctionGemma model
Download ollama application from here: https://ollama.com/
# Pull FunctionGemma model
ollama run functiongemma:270mgit clone https://github.com/krupagaliya/FunctionMac.git
cd FunctionMac# Create virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate
# Install dependencies
pip install -r requirements.txtSome features require macOS permissions:
-
System Settings → Privacy & Security → Accessibility
- Add Terminal or your Python IDE
-
System Settings → Privacy & Security → Automation
- Allow control of System Events
-
System Settings → Privacy & Security → Screen Recording
- Required for screenshots
python test_system.pyLaunch the Streamlit web app:
streamlit run streamlit_app.pyIt will show the Localhost URL and something like http://localhost:8504 open it in your browser to access:
- Function browser by category
- Direct function execution
- Quick action buttons
- Real-time results display
Start the AI assistant in your terminal:
python macos_ai_assistant.pyExample conversation:
🗣️ You: Take a screenshot and save it to desktop
🤖 Assistant: I've taken a screenshot and saved it to your desktop
🗣️ You: Set volume to 50 percent
🤖 Assistant: Volume has been set to 50%
🗣️ You: Turn on dark mode
🤖 Assistant: Appearance has been set to Dark mode
🗣️ You: What's my battery level?
🤖 Assistant: Your battery is at 85% and currently charging
from macos_ai_assistant import MacOSAIAssistant
# Initialize the assistant
assistant = MacOSAIAssistant()
# Process a query
result = assistant.process_query("Set brightness to 50%")
if result['success']:
print(f"Response: {result['final_response']}")
print(f"Functions called: {[fc['function'] for fc in result['function_calls']]}")
else:
print(f"Error: {result['error']}")"Set volume to 50 percent"
"Mute the volume"
"Turn on dark mode"
"Set brightness to 80%"
"Enable Do Not Disturb"
"Show me battery information"
"What's my CPU usage?"
"Get my screen resolution"
"Where is my mouse cursor?"
"What time is it?"
"Take a screenshot"
"Find all PDF files in Downloads"
"Copy this to clipboard: Hello World"
"What's in my clipboard?"
"Open Safari"
"Open Calculator"
"List running applications"
"Quit Finder"
"Click at coordinates 500, 300"
"Type 'Hello World'"
"Send keyboard shortcut cmd+c"
"Scroll down 5 clicks"
"Create a reminder: Buy groceries"
"Show my calendar events"
"Play music"
"Pause the music"
"Calculate sqrt(16) + sin(pi/2) * 10"
"Analyze these numbers: 10, 20, 30, 40, 50"
"Generate a QR code for https://github.com"
"Scan for Wi-Fi networks"
FunctionMac/
├── macos_ai_assistant.py # Main AI assistant orchestrator
├── macos_functions.py # Core macOS function implementations (40+ functions)
├── streamlit_app.py # Web interface
├── test_system.py # System verification tests
├── comprehensive_function_test.py # Full function test suite
├── requirements.txt # Python dependencies
├── README.md # Documentation
├── LICENSE # MIT License
- Model:
functiongemma:270m(lightweight, fast) - Function Definitions: Automatically exposed as tools
- Parameter Extraction: NL → structured function calls
| Method | Use Case |
|---|---|
| AppleScript | System Events, app control |
| subprocess | Shell commands, system tools |
| PyAutoGUI | Mouse/keyboard automation |
| psutil | System information |
| ctypes | Low-level macOS APIs |
Click to expand full function list
System Information
get_battery_info,get_system_info,get_cpu_usageget_memory_usage,get_disk_usage,get_screen_infoget_current_datetime
Audio Control
get_volume,set_volume,mute_volume,unmute_volume
Display Control
get_brightness,set_brightnessget_current_theme,change_theme
Focus Mode
enable_do_not_disturb,disable_do_not_disturb
Clipboard
copy_to_clipboard,get_clipboard_content,paste_from_clipboard
File Operations
take_screenshot,find_files,open_file,open_folderprocess_text_file
Applications
open_application,quit_application,list_running_applications
GUI Automation
click_at_coordinates,drag_mouse,get_mouse_positiontype_text,send_keyboard_shortcut,scroll_screen
App Integration
create_reminder,get_calendar_eventscontrol_music,send_message
Data Processing
calculate_expression,analyze_datagenerate_qr_code,get_wifi_networks
Make sure Ollama is running
# Ollama has functiongemma:270m model
ollama ls
# Test the model
ollama run functiongemma:270m "Hello"If functions fail with permission errors:
- Open System Settings → Privacy & Security
- Grant permissions to Terminal/IDE for:
- Accessibility
- Automation
- Screen Recording (for screenshots)
- Full Disk Access (for file operations)
I am working on expanding FunctionMac with advanced capabilities and fine-tuning the AI model.
🧠 Model Improvement Fine-tuning FunctionGemma: Specifically training the model on macOS-specific function calls to improve accuracy and parameter extraction for complex queries.
🚧 Upcoming Feature Categories
- Window & Space Management (Split view, stage manager)
- Network & Connectivity (Wi-Fi, Bluetooth toggle)
- Quick Actions & Shortcuts (Time savers)
- Smart Clipboard & Text Processing (History, formatting)
- Timer & Productivity (Pomodoro, alarms)
- Audio & Media Control (Enhanced playback)
- File Operations (Compression, advanced search)
- System Maintenance (Cache cleaning, updates)
- Accessibility & Display (Zoom, contrast)
- Smart Automation (Multi-step workflows)
Contributions are welcome! Here's how to get started:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Add tests for new functionality
- Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- Model Accuracy: FunctionGemma (270m) may occasionally misinterpret complex or ambiguous natural language commands
- Best Practice: Use specific, clear commands (e.g., "set volume to 50%" instead of "make it louder")
- Multi-step Commands: Break complex requests into individual commands for better results
💡 I am actively working on fine-tuning FunctionGemma to improve accuracy. See Roadmap. Contributions are welcome!
This project is licensed under the MIT License - see the LICENSE file for details.
- Ollama - Local LLM runtime
- Google FunctionGemma - Function calling model
- Streamlit - Web interface framework
- Issues: GitHub Issues
- Discussions: GitHub Discussions
Made with ❤️ for the macOS community, powered by Google's FunctionGemma model