|
| 1 | +# QueryForge |
| 2 | + |
| 3 | +Production-grade natural language to SQL engine for financial data — built from first principles with LLMOps discipline. |
| 4 | + |
| 5 | +QueryForge is the production-grade version of what Databricks Genie, Snowflake Cortex Analyst, and Azure AI Foundry's NL2SQL feature are commercializing. It converts natural language questions into validated SQL across multiple financial data schemas, with prompt versioning via MLflow, automated RAGAS evaluation on every GitHub commit, query result validation, and a feedback loop for continuous improvement. |
| 6 | + |
| 7 | +Inspired by the text-to-SQL POC built at SMBC on real banking data. |
| 8 | + |
| 9 | +## Architecture |
| 10 | + |
| 11 | +``` |
| 12 | + +---------------------+ |
| 13 | + | Natural Language | |
| 14 | + | Question | |
| 15 | + +----------+----------+ |
| 16 | + | |
| 17 | + +----------v----------+ |
| 18 | + | FastAPI | |
| 19 | + | /generate-sql | |
| 20 | + | /execute | |
| 21 | + | /feedback | |
| 22 | + +----------+----------+ |
| 23 | + | |
| 24 | + +----------------+----------------+ |
| 25 | + | | |
| 26 | + +----------v----------+ +----------v----------+ |
| 27 | + | Claude API | | PostgreSQL | |
| 28 | + | Chain-of-thought | | Financial schemas: | |
| 29 | + | reasoning + SQL gen | | - accounts | |
| 30 | + +----------+----------+ | - transactions | |
| 31 | + | | - risk_metrics | |
| 32 | + | | - model_inventory | |
| 33 | + +----------v----------+ +---------------------+ |
| 34 | + | Query Validation | |
| 35 | + +----------+----------+ |
| 36 | + | |
| 37 | + +----------v----------+ |
| 38 | + | RAGAS Evaluation | |
| 39 | + | Faithfulness | |
| 40 | + | Answer relevancy | |
| 41 | + | Context recall | |
| 42 | + +----------+----------+ |
| 43 | + | |
| 44 | + +----------v----------+ |
| 45 | + | MLflow Tracking | |
| 46 | + +---------------------+ |
| 47 | +``` |
| 48 | + |
| 49 | +## LLMOps Layer (MLflow + Databricks CE) |
| 50 | + |
| 51 | +- **MLflow Prompt Registry** — Version-controlled prompt templates (system prompt, schema context, few-shot examples) |
| 52 | +- **MLflow Experiments** — Every evaluation run logged with SQL accuracy, execution rate, RAGAS scores |
| 53 | +- **MLflow Model Registry** — `production` vs `staging` prompt versions, with promotion gates |
| 54 | +- **Databricks CE Notebooks** — Schema introspection, Gold layer sample queries for few-shot context |
| 55 | + |
| 56 | +## Inference + Evaluation Layer (Docker) |
| 57 | + |
| 58 | +- **FastAPI** — `/generate-sql`, `/execute`, `/feedback` endpoints |
| 59 | +- **PostgreSQL** — Sample financial database (accounts, transactions, risk_metrics, model_inventory schemas) |
| 60 | +- **Claude API** — SQL generation with chain-of-thought reasoning before outputting SQL |
| 61 | +- **RAGAS** — Automated evaluation: faithfulness, answer relevancy, context recall |
| 62 | +- **GitHub Actions** — Evaluation pipeline runs on every PR, blocks merge if accuracy drops |
| 63 | + |
| 64 | +## Getting Started |
| 65 | + |
| 66 | +### Prerequisites |
| 67 | + |
| 68 | +- Docker and Docker Compose |
| 69 | +- Anthropic API key |
| 70 | +- Python 3.11+ |
| 71 | + |
| 72 | +### Setup |
| 73 | + |
| 74 | +```bash |
| 75 | +# Clone the repo |
| 76 | +git clone https://github.com/SQLicious/QueryForge.git |
| 77 | +cd QueryForge |
| 78 | + |
| 79 | +# Set environment variables |
| 80 | +cp .env.example .env |
| 81 | +# Add your ANTHROPIC_API_KEY to .env |
| 82 | + |
| 83 | +# Start services |
| 84 | +docker-compose up -d |
| 85 | + |
| 86 | +# Run evaluation |
| 87 | +python scripts/evaluate.py |
| 88 | +``` |
| 89 | + |
| 90 | +## Project Structure |
| 91 | + |
| 92 | +``` |
| 93 | +QueryForge/ |
| 94 | +├── src/ # Application source code |
| 95 | +├── tests/ # Test suite |
| 96 | +├── scripts/ # Utility and evaluation scripts |
| 97 | +├── data/ # Schema definitions and sample data |
| 98 | +├── docker-compose.yml # Service orchestration |
| 99 | +├── requirements.txt # Python dependencies |
| 100 | +└── .github/workflows/ # CI/CD pipelines |
| 101 | +``` |
| 102 | + |
| 103 | +## License |
| 104 | + |
| 105 | +MIT |
0 commit comments