Skip to content

Commit 65c8ecd

Browse files
pavanjavapavanjava
committed
refactored architecture for more programatic usage
1 parent 733ab97 commit 65c8ecd

6 files changed

Lines changed: 496 additions & 79 deletions

File tree

‎README.md‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
[![PyPI version](https://img.shields.io/pypi/v/qql-cli?color=blue&label=PyPI)](https://pypi.org/project/qql-cli/)
66
[![Python 3.12+](https://img.shields.io/pypi/pyversions/qql-cli)](https://pypi.org/project/qql-cli/)
77
[![MIT License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
8-
[![Tests](https://img.shields.io/badge/tests-500%20passing-brightgreen)](tests/)
8+
[![Tests](https://img.shields.io/badge/tests-549%20passing-brightgreen)](tests/)
99

1010
Write `INSERT`, `SELECT`, `SEARCH`, `SCROLL`, `RECOMMEND`, `UPDATE`, `DELETE`, and `CREATE COLLECTION` statements instead of Python SDK calls. Supports hybrid dense+sparse vector search, grouped search (GROUP BY), cross-encoder reranking, quantization (scalar, turbo, binary, product), SQL-style `WHERE` filters, script execution, and collection dump/restore.
1111

@@ -50,6 +50,18 @@ Your query string
5050

5151
When you run `INSERT`, the `text` field is automatically converted into a dense vector using [Fastembed](https://github.com/qdrant/fastembed). In **hybrid mode** (`USING HYBRID`), a sparse BM25 vector is also generated alongside the dense vector, and searches use Qdrant's Reciprocal Rank Fusion (RRF) by default to merge the results of both retrieval methods. You can switch hybrid search to DBSF with `FUSION 'dbsf'`.
5252

53+
QQL also exposes a **programmatic API** for use inside Python applications — no CLI required:
54+
55+
```python
56+
from qql import Connection
57+
58+
with Connection("http://localhost:6333") as conn:
59+
conn.run_query("INSERT INTO COLLECTION notes VALUES {'text': 'Qdrant is fast'}")
60+
result = conn.run_query("SEARCH notes SIMILAR TO 'vector database' LIMIT 5")
61+
for hit in result.data:
62+
print(hit["score"], hit["payload"])
63+
```
64+
5365
---
5466

5567
## Installation
@@ -86,7 +98,7 @@ Full documentation lives in the [`docs/`](docs/) folder and at **[pavanjava.gith
8698
| [WHERE Filters](docs/filters.md) | Full SQL-style filter operators |
8799
| [Collections & Quantization](docs/collections.md) | SHOW, CREATE, DROP, QUANTIZE (scalar/turbo/binary/product), CREATE INDEX, UPDATE VECTOR, UPDATE PAYLOAD |
88100
| [Scripts: EXECUTE / DUMP](docs/scripts.md) | Script files, collection backup/restore |
89-
| [Programmatic Usage](docs/programmatic.md) | Use QQL as a Python library |
101+
| [Programmatic Usage](docs/programmatic.md) | Use QQL as a Python library via `Connection` or `run_query()` |
90102
| [Reference: Models / Config / Errors](docs/reference.md) | Embedding models, config file, error reference |
91103

92104
---

‎docs/programmatic.md‎

Lines changed: 172 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -4,98 +4,203 @@ QQL can be used as a Python library without the CLI.
44

55
---
66

7-
## `run_query()` — high-level API
7+
## `Connection` — Primary API
8+
9+
`Connection` is the recommended way to use QQL programmatically. It opens a
10+
single connection to Qdrant once and reuses it for every `run_query()` call —
11+
more efficient than the legacy `run_query()` function, which creates a new
12+
client on every invocation.
13+
14+
### Basic usage
815

916
```python
10-
from qql import run_query
17+
from qql import Connection
18+
19+
conn = Connection("http://localhost:6333")
1120

1221
# Insert a document (dense-only)
13-
result = run_query(
14-
"INSERT INTO COLLECTION notes VALUES {'text': 'hello world', 'author': 'alice', 'year': 2024}",
15-
url="http://localhost:6333",
22+
result = conn.run_query(
23+
"INSERT INTO COLLECTION notes VALUES {'text': 'hello world', 'author': 'alice', 'year': 2024}"
1624
)
1725
print(result.message) # "Inserted 1 point [<id>]"
18-
print(result.data) # {"id": 1001 or "<uuid>", "collection": "notes"}
26+
print(result.data) # {"id": "<uuid>", "collection": "notes"}
1927

20-
# Insert with hybrid vectors
21-
result = run_query(
22-
"INSERT INTO COLLECTION notes VALUES {'text': 'hello world'} USING HYBRID",
23-
url="http://localhost:6333",
24-
)
25-
print(result.message) # "Inserted 1 point [<id>] (hybrid)"
26-
27-
# Dense search with WHERE filter
28-
result = run_query(
29-
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 WHERE year >= 2023 AND author != 'bot'",
30-
url="http://localhost:6333",
28+
# Search
29+
result = conn.run_query(
30+
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 WHERE year >= 2023"
3131
)
3232
for hit in result.data:
3333
print(hit["score"], hit["payload"])
3434

35-
# Hybrid search with WHERE filter
36-
result = run_query(
37-
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 USING HYBRID WHERE year >= 2023",
38-
url="http://localhost:6333",
39-
)
40-
for hit in result.data:
41-
print(hit["score"], hit["payload"])
35+
conn.close()
36+
```
4237

43-
# Scroll / pagination
44-
result = run_query(
45-
"SCROLL FROM notes LIMIT 2",
46-
url="http://localhost:6333",
47-
)
48-
for point in result.data["points"]:
49-
print(point["id"], point["payload"])
50-
print(result.data["next_offset"])
38+
### Context manager (preferred)
5139

52-
# Bulk insert (all records embedded and upserted in one call)
53-
result = run_query(
54-
"""INSERT BULK INTO COLLECTION notes VALUES [
55-
{'id': 1, 'text': 'first document', 'year': 2023},
56-
{'id': 2, 'text': 'second document', 'year': 2024}
57-
]""",
58-
url="http://localhost:6333",
59-
)
60-
print(result.message) # "Inserted 2 points"
40+
The context manager guarantees the HTTP connection pool is released even if an
41+
exception occurs:
6142

62-
# Recommend similar points using known IDs as positive examples
63-
result = run_query(
64-
"RECOMMEND FROM notes POSITIVE IDS (1, 2) NEGATIVE IDS (3) LIMIT 5",
65-
url="http://localhost:6333",
66-
)
67-
for hit in result.data:
68-
print(hit["score"], hit["payload"])
43+
```python
44+
from qql import Connection
6945

70-
# Retrieve a point by ID
71-
result = run_query(
72-
"SELECT * FROM notes WHERE id = 1",
73-
url="http://localhost:6333",
74-
)
75-
print(result.data) # {"id": "1", "payload": {...}}
46+
with Connection("http://localhost:6333") as conn:
47+
# All queries share the same connection
48+
conn.run_query(
49+
"INSERT INTO COLLECTION notes VALUES {'text': 'hello world'} USING HYBRID"
50+
)
51+
result = conn.run_query(
52+
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 USING HYBRID WHERE year >= 2023"
53+
)
54+
for hit in result.data:
55+
print(hit["score"], hit["payload"])
56+
```
57+
58+
### Qdrant Cloud
59+
60+
```python
61+
from qql import Connection
62+
63+
with Connection("https://<your-cluster>.qdrant.io", secret="<your-api-key>") as conn:
64+
result = conn.run_query("SHOW COLLECTIONS")
65+
print(result.data)
66+
```
67+
68+
### Custom embedding model
7669

77-
# Delete by filter
70+
```python
71+
from qql import Connection
72+
73+
with Connection(
74+
"http://localhost:6333",
75+
default_model="BAAI/bge-base-en-v1.5",
76+
) as conn:
77+
conn.run_query(
78+
"INSERT INTO COLLECTION articles VALUES {'text': 'Attention is all you need'}"
79+
)
80+
```
81+
82+
### All statement examples
83+
84+
```python
85+
from qql import Connection
86+
87+
with Connection("http://localhost:6333") as conn:
88+
89+
# Hybrid insert
90+
conn.run_query(
91+
"INSERT INTO COLLECTION notes VALUES {'text': 'hello world'} USING HYBRID"
92+
)
93+
94+
# Dense search with WHERE filter
95+
result = conn.run_query(
96+
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 WHERE year >= 2023 AND author != 'bot'"
97+
)
98+
for hit in result.data:
99+
print(hit["score"], hit["payload"])
100+
101+
# Hybrid search
102+
result = conn.run_query(
103+
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 USING HYBRID WHERE year >= 2023"
104+
)
105+
106+
# Scroll / pagination
107+
result = conn.run_query("SCROLL FROM notes LIMIT 2")
108+
for point in result.data["points"]:
109+
print(point["id"], point["payload"])
110+
next_cursor = result.data["next_offset"] # str | int | None
111+
112+
# Continue pagination
113+
if next_cursor is not None:
114+
result = conn.run_query(f"SCROLL FROM notes AFTER '{next_cursor}' LIMIT 2")
115+
116+
# Bulk insert
117+
result = conn.run_query(
118+
"""INSERT BULK INTO COLLECTION notes VALUES [
119+
{'id': 1, 'text': 'first document', 'year': 2023},
120+
{'id': 2, 'text': 'second document', 'year': 2024}
121+
]"""
122+
)
123+
print(result.message) # "Inserted 2 points"
124+
125+
# Recommend similar points
126+
result = conn.run_query(
127+
"RECOMMEND FROM notes POSITIVE IDS (1, 2) NEGATIVE IDS (3) LIMIT 5"
128+
)
129+
for hit in result.data:
130+
print(hit["score"], hit["payload"])
131+
132+
# Retrieve a point by ID
133+
result = conn.run_query("SELECT * FROM notes WHERE id = 1")
134+
print(result.data) # {"id": "1", "payload": {...}}
135+
136+
# Delete by filter
137+
conn.run_query("DELETE FROM notes WHERE year < 2023")
138+
139+
# Inspect collection diagnostics
140+
result = conn.run_query("SHOW COLLECTION notes")
141+
print(result.data["topology"]) # "dense" or "hybrid"
142+
print(result.data["vectors"]) # {"": {...}} or {"dense": {...}}
143+
print(result.data["payload_schema"]) # field index info, or None
144+
```
145+
146+
### `Connection` parameters
147+
148+
| Parameter | Type | Default | Description |
149+
|---|---|---|---|
150+
| `url` | `str` | `"http://localhost:6333"` | Qdrant instance URL |
151+
| `secret` | `str \| None` | `None` | API key; `None` for unauthenticated |
152+
| `default_model` | `str \| None` | `None` → `sentence-transformers/all-MiniLM-L6-v2` | Dense embedding model used when no `USING MODEL` clause is given |
153+
154+
### Power-user: `executor` property
155+
156+
For low-level access to the pipeline, use `conn.executor` directly:
157+
158+
```python
159+
from qql import Connection
160+
from qql.lexer import Lexer
161+
from qql.parser import Parser
162+
163+
with Connection("http://localhost:6333") as conn:
164+
tokens = Lexer().tokenize("SEARCH docs SIMILAR TO 'hello' LIMIT 5")
165+
node = Parser(tokens).parse()
166+
result = conn.executor.execute(node)
167+
```
168+
169+
---
170+
171+
## `run_query()` — Legacy one-shot API
172+
173+
> **Note:** `run_query()` is kept for backward compatibility. It creates a new
174+
> `Connection` (and therefore a new `QdrantClient`) on every call. For
175+
> workloads that issue more than one query, use `Connection` instead.
176+
177+
```python
178+
from qql import run_query
179+
180+
# Insert a document
78181
result = run_query(
79-
"DELETE FROM notes WHERE year < 2023",
182+
"INSERT INTO COLLECTION notes VALUES {'text': 'hello world', 'author': 'alice', 'year': 2024}",
80183
url="http://localhost:6333",
81184
)
82-
print(result.message) # "Deleted N point(s)"
185+
print(result.message)
83186

84-
# Inspect collection diagnostics
187+
# Search
85188
result = run_query(
86-
"SHOW COLLECTION notes",
189+
"SEARCH notes SIMILAR TO 'hello' LIMIT 5 WHERE year >= 2023",
87190
url="http://localhost:6333",
88191
)
89-
print(result.data["topology"]) # "dense" or "hybrid"
90-
print(result.data["vectors"]) # {"": {...}} or {"dense": {...}, ...}
91-
print(result.data["payload_schema"]) # {"field": {"type": "keyword", ...}, ...} or None
192+
for hit in result.data:
193+
print(hit["score"], hit["payload"])
92194
```
93195

196+
`run_query()` accepts the same `url`, `secret`, and `default_model` parameters
197+
as `Connection.__init__()`.
198+
94199
---
95200

96201
## Low-level pipeline API
97202

98-
For more control, use the pipeline directly:
203+
For full control, use the Lexer → Parser → Executor pipeline directly:
99204

100205
```python
101206
from qdrant_client import QdrantClient
@@ -117,9 +222,12 @@ for hit in result.data:
117222
print(hit["score"], hit["payload"])
118223
```
119224

225+
This is equivalent to what `Connection` does internally, giving you full
226+
control over the client lifecycle and config.
227+
120228
---
121229

122-
## ExecutionResult
230+
## `ExecutionResult`
123231

124232
All operations return an `ExecutionResult`:
125233

‎docs/reference.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,9 +133,10 @@ qql/
133133
├── pyproject.toml # Package config; installs the `qql` CLI command
134134
├── src/
135135
│ └── qql/
136-
│ ├── __init__.py # Public API: run_query()
136+
│ ├── __init__.py # Public API: Connection, run_query()
137137
│ ├── cli.py # CLI entry point: connect, disconnect, execute, dump, REPL
138138
│ ├── config.py # QQLConfig dataclass + ~/.qql/config.json I/O
139+
│ ├── connection.py # Connection class — stateful programmatic API
139140
│ ├── exceptions.py # QQLError, QQLSyntaxError, QQLRuntimeError
140141
│ ├── lexer.py # Tokenizer: string → List[Token]
141142
│ ├── ast_nodes.py # Frozen dataclasses for each statement and filter type
@@ -148,6 +149,7 @@ qql/
148149
├── test_lexer.py # Tokenizer unit tests
149150
├── test_parser.py # Parser unit tests
150151
├── test_executor.py # Executor unit tests (mocked Qdrant client)
152+
├── test_connection.py # Connection class unit tests (mocked Qdrant client)
151153
├── test_script.py # Script runner unit tests
152154
└── test_dumper.py # Dumper unit tests
153155
```
@@ -162,7 +164,7 @@ Tests do not require a running Qdrant instance — the Qdrant client is mocked.
162164
pytest tests/ -v
163165
```
164166

165-
Expected output: **500 tests passing**.
167+
Expected output: **549 tests passing**.
166168

167169
---
168170

‎src/qql/__init__.py‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,15 @@
66
__version__ = "0.0.0+unknown"
77

88
from .config import DEFAULT_MODEL, QQLConfig, load_config
9+
from .connection import Connection
910
from .exceptions import QQLError, QQLRuntimeError, QQLSyntaxError
1011
from .executor import ExecutionResult, Executor
1112
from .lexer import Lexer
1213
from .parser import Parser
1314

1415
__all__ = [
1516
"__version__",
17+
"Connection",
1618
"QQLConfig",
1719
"QQLError",
1820
"QQLRuntimeError",
@@ -32,15 +34,13 @@ def run_query(
3234
secret: str | None = None,
3335
default_model: str | None = None,
3436
) -> ExecutionResult:
35-
"""Convenience function for programmatic use."""
36-
from qdrant_client import QdrantClient
37+
"""One-shot convenience function kept for backward compatibility.
3738
38-
cfg = QQLConfig(
39-
url=url,
40-
secret=secret,
41-
default_model=default_model or DEFAULT_MODEL,
42-
)
43-
client = QdrantClient(url=url, api_key=secret)
44-
tokens = Lexer().tokenize(query)
45-
node = Parser(tokens).parse()
46-
return Executor(client, cfg).execute(node)
39+
Creates a :class:`Connection`, runs one query, and closes the connection.
40+
For workloads that issue multiple queries, prefer :class:`Connection`
41+
directly — it reuses a single client across all calls::
42+
43+
with Connection(url, secret=secret) as conn:
44+
result = conn.run_query(query)
45+
"""
46+
return Connection(url=url, secret=secret, default_model=default_model).run_query(query)

0 commit comments

Comments
 (0)