This project extends the TAP AI Frappe application with a powerful, conversational AI layer. It provides a single, robust API endpoint that can understand user questions and intelligently route them to the best tool - a curated knowledge bank, a direct database query, a semantic vector search, or a direct LLM fallback - to provide accurate, context-aware answers.
The system is designed for multi-turn conversations, automatically managing chat history to understand follow-up questions. It features asynchronous processing via RabbitMQ workers, voice input/output support, and dynamic configuration management for seamless integration with TAP LMS.
Current deployment topology:
- AI application server:
ai.evalix.xyz(hosts TAP AI code and workers) - Remote database server:
data.evalix.xyz(PostgreSQL)
- Project Overview
- Core Architecture
- System Workflow
- Complete Codebase Structure
- Dependencies
- Installation
- Configuration
- One-Time Setup
- Testing
- API Documentation
- Worker System
- Core File Descriptions
- Telegram Bot Demo
- Deployment Guide
- Troubleshooting
TAP AI is a conversational AI engine built on top of the Frappe framework. It intelligently routes user queries to specialized execution engines:
- Knowledge Bank Tool: For curated TAP responses, greetings, short support phrases, and high-confidence conversational matches
- Text-to-SQL Engine: For factual, database-specific queries
- Vector RAG Engine: For conceptual, semantic, and summarization queries
- Direct LLM Tool: For open-ended conversation when no knowledge-bank entry fits
- RabbitMQ Worker Architecture: Asynchronous processing for scalability
- Voice Processing: STT β LLM β TTS pipeline for voice queries
Key Features:
- Intelligent routing using LLMs
- Multi-turn conversation support with history management
- Hybrid query execution (Knowledge Bank + SQL + Vector Search + Direct LLM)
- Automatic fallback mechanisms with confidence thresholds
- Telegram bot integration
- Rate limiting and authentication built-in
- Voice input/output support via Telegram
- Asynchronous processing with RabbitMQ
- Dynamic configuration for TAP LMS integration
- Admin-controlled DocType exclusion system
- Added a hybrid Knowledge Bank verifier: the system now probes the best KB candidate and asks the LLM to verify whether the candidate appropriately answers the user's query; the LLM either returns the KB response (optionally lightly personalized) or generates a fresh answer. This reduces false positives (e.g., distinguishing "who are you" vs "how are you").
- Router now uses the hybrid verifier for
knowledge_bankrouted queries. - DocType event hooks invalidate the KB cache on insert/update/delete to keep the KB context fresh.
- A verifier LLM cache is used to reduce latency for repeated verification queries (TTL configurable).
Technology Stack:
- Backend: Python 3.10+
- Framework: Frappe (ERPNext)
- LLM: OpenAI GPT models
- Vector DB: Pinecone
- Database: Remote PostgreSQL (
data.evalix.xyz) - Message Queue: RabbitMQ (Pika)
- Caching: Redis
- Web Framework: Flask (for Telegram webhooks)
- ORM: SQLAlchemy
Language Composition:
- Python: 107,850 bytes (99%)
- JavaScript: 564 bytes (1%)
The system's intelligence lies in its central router, which acts as a decision-making brain. When a query is received, it follows this flow:
- Intelligent Routing: An LLM analyzes the user's query to determine its intent.
- Tool Selection:
- For short, curated conversational intents that match the TAP response bank, it selects the Knowledge Bank Tool.
- For factual, specific questions (e.g., "list all...", "how many..."), it selects the Text-to-SQL Engine.
- For conceptual, open-ended, or summarization questions (e.g., "summarize...", "explain..."), it selects the Vector RAG Engine.
- For open-ended supportive conversation that does not fit the knowledge bank, it selects the Direct LLM Tool.
- Execution & Fallback: The chosen tool executes the query. If the knowledge bank misses or returns a low-confidence match, the system falls back to the Direct LLM tool. If SQL fails to produce a satisfactory answer, the system automatically falls back to the Vector RAG engine as a safety net.
- Answer Synthesis: The retrieved data or direct response is returned as a final, human-readable answer.
graph TD
subgraph "User Input"
User[User Query]
end
subgraph "API Layer"
QueryAPI["api/query.py<br><b>Unified Query API (Text + Voice)</b>"]
end
subgraph "Message Queue"
RabbitMQ["RabbitMQ<br>Message Broker"]
end
subgraph "Worker Processes"
STTWorker["workers/stt_worker.py<br><b>Speech-to-Text</b>"]
LLMWorker["workers/llm_worker.py<br><b>LLM Router</b>"]
TTSWorker["workers/tts_worker.py<br><b>Text-to-Speech</b>"]
end
subgraph "Services"
Router["services/router.py<br><b>Intelligent Router</b>"]
KB["services/direct_response_bank.py<br><b>Knowledge Bank</b>"]
SQL["services/sql_answerer.py<br><b>SQL Engine</b>"]
RAG["services/rag_answerer.py<br><b>RAG Engine</b>"]
Direct["services/direct_answerer.py<br><b>Direct LLM</b>"]
end
subgraph "Data Layer"
PostgresDB[(Remote PostgreSQL<br>data.evalix.xyz)]
PineconeDB[(Pinecone<br>Vector DB)]
end
User -->|Text or Voice| QueryAPI
QueryAPI -->|Request| RabbitMQ
RabbitMQ -->|audio_stt_queue| STTWorker
RabbitMQ -->|text_query_queue| LLMWorker
RabbitMQ -->|audio_tts_queue| TTSWorker
STTWorker -->|Transcribed Text| RabbitMQ
LLMWorker -->|Route Query| Router
Router -->|Curated Match| KB
KB -->|High confidence| Router
KB -->|Miss / low confidence| Direct
Router -->|Factual| SQL
Router -->|Conceptual| RAG
Router -->|Open-ended support| Direct
SQL -->|SQL Query| PostgresDB
RAG -->|Vector Search| PineconeDB
LLMWorker -->|Answer| TTSWorker
TTSWorker -->|Audio File| PostgresDB
The robustness of the system comes from the specialized design of each engine.
This engine excels at factual queries because it builds an "intelligent schema" before prompting the LLM.
graph TD
A[User Query] --> B["1. Inspect Live Frappe Metadata"]
B --> C["2. Create Rich Schema Prompt"]
C --> D{LLM: Generate SQL}
D --> E[Remote PostgreSQL data.evalix.xyz]
E --> F[Structured Data Rows]
This engine excels at conceptual queries by retrieving semantically relevant documents.
This tool handles short, high-confidence conversational intents like greetings, acknowledgements, simple help requests, identity questions, and other curated TAP response patterns.
graph TD
A[User Query] --> B[Normalize and score against curated entries]
B --> C{Confidence threshold met?}
C -->|Yes| D[Return stored TAP response]
C -->|No| E[Fallback to Direct LLM]
graph TD
A[User Query + Chat History] --> B{LLM: Refine Query}
B --> C["1. Select DocTypes"]
C --> D["2. Semantic Search"]
D --> E["3. Fetch Full Text"]
E --> F[Rich Context Chunks]
tap_ai/
βββ __init__.py # Package initialization
βββ hooks.py # Frappe hooks for app lifecycle
βββ modules.txt # Module declaration
βββ patches.txt # Database migration patches
β
βββ api/ # REST API Endpoints
β βββ __init__.py
β βββ query.py # Unified query endpoint (text + voice, async via RabbitMQ)
β βββ result.py # Unified result polling endpoint (with optional server-side wait)
β βββ voice_query.py # Backward-compatible wrapper alias for unified query
β βββ voice_result.py # Backward-compatible wrapper alias for unified result
β
βββ services/ # Core execution engines
β βββ __init__.py
β βββ router.py # Intelligent router (brain of system)
β βββ sql_answerer.py # Text-to-SQL engine
β βββ rag_answerer.py # Vector RAG engine
β βββ doctype_selector.py # DocType selection for RAG
β βββ pinecone_store.py # Pinecone vector database integration
β βββ pinecone_index.py # Pinecone index lifecycle
β βββ ratelimit.py # API rate limiting utility
β
βββ workers/ # RabbitMQ Background Workers
β βββ llm_worker.py # Main LLM routing worker
β βββ stt_worker.py # Speech-to-Text worker (Whisper)
β βββ tts_worker.py # Text-to-Speech worker (OpenAI TTS)
β
βββ schema/ # Database schema generation
β βββ __init__.py
β βββ generate_schema.py # Schema generator script
β βββ tap_ai_schema.json # Generated schema file
β
βββ infra/ # Infrastructure utilities
β βββ __init__.py
β βββ config.py # Centralized config loader
β βββ sql_catalog.py # Schema catalog loader
β
βββ utils/ # Utility functions
β βββ __init__.py
β βββ dynamic_config.py # Dynamic config for TAP LMS integration
β βββ remote_db.py # Remote PostgreSQL connection helpers
β βββ mq.py # RabbitMQ publisher utility
β
βββ config/ # Frappe app configuration
β βββ __init__.py
β
βββ public/ # Static assets
β βββ .gitkeep
β
βββ templates/ # Frappe templates
β βββ __init__.py
β βββ pages/
β
βββ tap_ai/ # Additional modules (if any)
# Root-level files
βββ README.md # This file
βββ requirements.txt # Python dependencies
βββ pyproject.toml # Project metadata & build config
βββ license.txt # License information
βββ .env # Local environment variables (do not commit secrets)
βββ .gitignore # Git ignore rules
βββ .vscode/ # VS Code workspace settings
βββ .eslintrc # ESLint configuration
βββ .editorconfig # Editor configuration
βββ .pre-commit-config.yaml # Pre-commit hooks
βββ __init__.py # Root package init
βββ telegram_webhook.py # Telegram bot bridge script
psycopg2-binary(or equivalent) - PostgreSQL database driver for remote DB accesssqlalchemy>=2.0.32- SQL toolkit and ORMsqlalchemy-utils>=0.41.2- SQLAlchemy utility functions
openai>=1.40.0- OpenAI API client (GPT, Whisper, TTS)langchain>=0.3.0- LLM frameworklangchain-community>=0.3.0- LangChain integrationslangchain-openai>=0.1.17- LangChain OpenAI integrationtiktoken>=0.7.0- Token counting for OpenAI
pinecone- Pinecone vector database client
pika- RabbitMQ client for async processing
numpy>=1.26.4- Numerical computing
redis>=5.0.8- Redis client for caching and rate limiting
python-dotenv>=1.0.1- Environment variable loadingpydantic>=2.8.2- Data validationloguru>=0.7.2- Enhanced loggingtenacity>=9.0.0- Retry library
Flask- Web framework for webhookspython-telegram-bot- Telegram bot libraryrequests- HTTP client library
Note: Telegram integration dependencies are only required for
telegram_webhook.pyand are not included inrequirements.txtby default.
pytest>=8.3.2- Testing frameworkhttpx>=0.27.2- Async HTTP client
Frappe~=15.0+- Installed via bench (not in requirements.txt)
- Python 3.10+
- Frappe bench installed
- Remote PostgreSQL server reachable (
data.evalix.xyz) - RabbitMQ broker running
- Redis server running
- Pinecone account (for Vector RAG)
- OpenAI API key
# Get the app
bench get-app tap_ai https://github.com/theapprenticeproject/Ai.git
# Install on site
bench --site <site-name> install-app tap_ai# Install all required packages
bench pip install -r apps/tap_ai/requirements.txt
# Or install key packages individually
bench pip install langchain-openai pinecone psycopg2-binary pika redis# RabbitMQ (macOS)
brew install rabbitmq
# RabbitMQ (Ubuntu)
sudo apt-get install rabbitmq-server
# Redis (macOS)
brew install redis
# Redis (Ubuntu)
sudo apt-get install redis-server
# Start services
brew services start rabbitmq-server
brew services start redis-servercd apps/tap_ai
pre-commit installEdit your site's site_config.json file and add:
{
"openai_api_key": "sk-your-openai-key-here",
"primary_llm_model": "gpt-4o-mini",
"embedding_model": "text-embedding-3-small",
"pinecone_api_key": "pcn-your-pinecone-key-here",
"pinecone_index": "tap-ai-byo",
"rabbitmq_url": "amqp://guest:guest@localhost:5672/",
"redis_host": "localhost",
"redis_port": 6379,
"redis_db": 0,
"max_context_length": 2048,
"vector_search_k": 5,
"max_response_tokens": 500
}| Key | Type | Purpose | Default |
|---|---|---|---|
openai_api_key |
string | OpenAI API authentication | Required |
primary_llm_model |
string | Primary LLM for routing | gpt-4o-mini |
embedding_model |
string | Model for embeddings | text-embedding-3-small |
pinecone_api_key |
string | Pinecone authentication | Required |
pinecone_index |
string | Pinecone index name | tap-ai-byo |
rabbitmq_url |
string | RabbitMQ connection URL | amqp://guest:guest@localhost:5672/ |
redis_host |
string | Redis hostname | localhost |
redis_port |
int | Redis port | 6379 |
redis_db |
int | Redis database number | 0 |
max_context_length |
int | Max LLM context tokens | 2048 |
vector_search_k |
int | Top-K vectors for RAG | 5 |
max_response_tokens |
int | Max response tokens | 500 |
Create .env file in frappe-bench:
OPENAI_API_KEY=sk-your-key
PINECONE_API_KEY=pcn-your-key
RABBITMQ_URL=amqp://guest:guest@localhost:5672/Note: A local
.envfile is included for convenience. Do not store production secrets in source control.
bench execute tap_ai.schema.generate_schema.cliThis creates tap_ai_schema.json needed by SQL and RAG engines.
bench execute tap_ai.services.pinecone_index.cli_ensure_indexbench execute tap_ai.services.pinecone_store.cli_upsert_all# Unified query: text
curl -X POST "http://localhost:8000/api/method/tap_ai.api.query.query" \
-H "Content-Type: application/json" \
-d '{"q": "List all courses", "user_id": "test_user"}'
# Response
{"request_id": "REQ_a1b2c3d4"}
# Poll unified result (auto long-poll defaults)
curl "http://localhost:8000/api/method/tap_ai.api.result.result?request_id=REQ_a1b2c3d4"# Unified query: voice
curl -X POST "http://localhost:8000/api/method/tap_ai.api.query.query" \
-H "Content-Type: application/json" \
-d '{"audio_url": "https://example.com/audio.mp3", "user_id": "test_user"}'
# Response
{"request_id": "VREQ_x1y2z3w4"}
# Poll unified result with explicit wait override
curl "http://localhost:8000/api/method/tap_ai.api.result.result?request_id=VREQ_x1y2z3w4&wait_seconds=25&poll_interval_ms=500"In separate terminal windows:
# Worker 1: LLM Worker
cd frappe-bench
bench execute tap_ai.workers.llm_worker.start
# Worker 2: STT Worker
bench execute tap_ai.workers.stt_worker.start
# Worker 3: TTS Worker
bench execute tap_ai.workers.tts_worker.startPOST /api/method/tap_ai.api.query.query
Request body:
{
"q": "Your question here (text mode)",
"user_id": "unique_user_identifier"
}or
{
"audio_url": "https://example.com/audio.mp3 (voice mode)",
"user_id": "unique_user_identifier"
}Response:
{
"request_id": "REQ_abc12345"
}GET /api/method/tap_ai.api.result.result?request_id=REQ_abc12345
Optional query params:
wait_seconds(0-55)poll_interval_ms(100-2000)
If omitted, TAP AI auto-tunes defaults by mode:
- text:
wait_seconds=8,poll_interval_ms=300 - voice:
wait_seconds=25,poll_interval_ms=500
Response (pending):
{
"status": "processing"
}Response (success):
{
"status": "success",
"answer": "The answer to your question...",
"query": "Your question",
"history": [...],
"metadata": {...}
}Primary endpoint:
POST /api/method/tap_ai.api.query.query
Backward-compatible alias:
POST /api/method/tap_ai.api.voice_query.voice_query
Request body:
{
"audio_url": "https://example.com/audio.mp3",
"user_id": "unique_user_identifier"
}Response:
{
"request_id": "VREQ_xyz98765"
}Primary endpoint:
GET /api/method/tap_ai.api.result.result?request_id=VREQ_xyz98765
Backward-compatible alias:
GET /api/method/tap_ai.api.voice_result.voice_result?request_id=VREQ_xyz98765
Response (processing):
{
"status": "processing"
}Response (success):
{
"status": "success",
"transcribed_text": "What is the first course?",
"answer_text": "The first course is...",
"audio_url": "/files/output_file.mp3",
"language": "en"
}Note:
voice_resultalias may returnstatus: "processing"while STT, LLM, and TTS jobs complete in the background. Poll until the final status issuccess.
The system uses RabbitMQ for asynchronous processing. Three workers handle different tasks:
- Pulls text queries from
text_query_queue - Runs the router to choose between SQL and RAG
- Manages conversation history
- Routes voice queries to TTS worker
- Updates request status in Redis cache
Start with:
bench execute tap_ai.workers.llm_worker.start- Pulls voice requests from
audio_stt_queue - Downloads audio from provided URL
- Uses Whisper API to transcribe
- Detects language of transcription
- Routes transcribed text to LLM worker
Start with:
bench execute tap_ai.workers.stt_worker.start- Pulls synthesization jobs from
audio_tts_queue - Uses OpenAI TTS to generate speech
- Saves audio file to Frappe File Manager
- Returns audio URL and marks request as complete
Start with:
bench execute tap_ai.workers.tts_worker.starttap_ai/api/query.py
- Unified text + voice query entry point
- Rate limiting check
- Publishes to RabbitMQ
text_query_queue(text) oraudio_stt_queue(voice) - Returns request_id for polling
tap_ai/api/result.py
- Unified result endpoint for text and voice
- Supports short server-side waiting to reduce coarse flow wait-node dependence
- Retrieves from Redis cache
tap_ai/api/voice_query.py
- Backward-compatible wrapper for unified query endpoint
tap_ai/api/voice_result.py
- Backward-compatible wrapper for unified result endpoint
tap_ai/services/router.py
- Central query routing logic
- Chooses between SQL and RAG engines
- Manages fallback logic
- Handles chat history
tap_ai/services/sql_answerer.py
- Generates SQL from natural language
- Builds intelligent schema for LLM
- Executes queries against remote PostgreSQL
- Returns structured data
tap_ai/services/rag_answerer.py
- Retrieves semantically similar documents
- Refines queries with chat history
- Synthesizes answers from context
- Handles multi-turn conversations
tap_ai/services/doctype_selector.py
- Selects relevant DocTypes for RAG
- Reduces search space
- Improves retrieval accuracy
tap_ai/services/pinecone_store.py
- Manages Pinecone interactions
- Upserts documents with embeddings
- Performs semantic search
tap_ai/services/ratelimit.py
- Enforces API rate limits
- Uses Redis for distributed counting
- Tracks requests per user
tap_ai/workers/llm_worker.py
- Main processing worker
- Routes queries through the dual-engine system
- Manages conversation context
- Bridges text and voice pipelines
tap_ai/workers/stt_worker.py
- Speech-to-Text processing
- Audio download and handling
- Language detection
- Whisper API integration
tap_ai/workers/tts_worker.py
- Text-to-Speech synthesis
- OpenAI TTS integration
- Frappe File Manager integration
- Audio file management
tap_ai/utils/dynamic_config.py
- Decouples TAP AI from TAP LMS schema changes
- Handles dynamic DocType mapping
- Manages user profiles with enrollment data
- Singleton pattern for configuration caching
- Validation and context resolution rules
tap_ai/utils/mq.py
- RabbitMQ publisher
- Queue declaration and management
- Persistent message delivery
tap_ai/infra/config.py
- Centralized configuration loader
- Frappe integration with fallbacks
- Works both inside Frappe and standalone
- Service status validation
tap_ai/schema/generate_schema.py
- Dynamically discovers all Frappe DocTypes
- Builds intelligent schema for SQL queries
- Supports admin-controlled exclusions
- Auto-detects joins and relationships
User β Telegram β Ngrok β telegram_webhook.py β Frappe API β AI Engine
- Telegram account
- Ngrok installed and authenticated
- Frappe bench running
- Search for
@BotFatheron Telegram - Send
/newbot - Follow instructions
- Copy the bot token (e.g.,
123456:ABC-DEF1234)
ngrok config add-authtoken <your-ngrok-token>
ngrok http 5000Copy the HTTPS forwarding URL (e.g., https://random-string.ngrok-free.app)
# Install dependencies
bench pip install Flask python-telegram-bot requests
# Edit telegram_webhook.py and set:
# - TELEGRAM_BOT_TOKEN
# - FRAPPE_API_URL
# - FRAPPE_API_KEY
# - FRAPPE_API_SECRET
# - OPENAI_API_KEY
# Run the bridge
python apps/tap_ai/telegram_webhook.pycurl -F "url=https://<NGROK_URL>/webhook" \
"https://api.telegram.org/bot<BOT_TOKEN>/setWebhook"Open Telegram and start a conversation with your bot!
# Terminal 1: Frappe
bench start
# Terminal 2: LLM Worker
bench execute tap_ai.workers.llm_worker.start
# Terminal 3: STT Worker
bench execute tap_ai.workers.stt_worker.start
# Terminal 4: TTS Worker
bench execute tap_ai.workers.tts_worker.start
# Terminal 5: Ngrok (optional for Telegram)
ngrok http 5000Use Supervisor or systemd for worker management:
# /etc/supervisor/conf.d/tap-ai-workers.conf
[program:tap-ai-llm]
command=bench execute tap_ai.workers.llm_worker.start
directory=/opt/frappe-bench
autostart=true
autorestart=true
[program:tap-ai-stt]
command=bench execute tap_ai.workers.stt_worker.start
directory=/opt/frappe-bench
autostart=true
autorestart=true
[program:tap-ai-tts]
command=bench execute tap_ai.workers.tts_worker.start
directory=/opt/frappe-bench
autostart=true
autorestart=true# Check site_config.json
cat sites/<site-name>/site_config.json | grep openai_api_key
# Or check env vars
echo $OPENAI_API_KEY# Check if RabbitMQ is running
brew services list | grep rabbitmq
# Or check status
rabbitmqctl status
# Start if not running
brew services start rabbitmq-server# Recreate index
bench execute tap_ai.services.pinecone_index.cli_ensure_index
# Upsert data
bench execute tap_ai.services.pinecone_store.cli_upsert_all# Check RabbitMQ queues
rabbitmqctl list_queues
# Check Redis connection
redis-cli PING
# Check Frappe logs
tail -f frappe-bench/logs/frappe.logThis project is licensed under the terms specified in license.txt.
Last Updated: 2026-03-18
Version: 2.0.0
Author: Anish Aman
Repository: theapprenticeproject/Ai