Merge branch 'main' into feat/pr25-merge-fix

# Conflicts:
#	README.md
#	docs/getting-started.md
#	docs/index.html
#	docs/reference.md
#	docs/search.md
#	src/qql/ast_nodes.py
#	src/qql/cli.py
#	src/qql/executor.py
#	src/qql/parser.py
#	src/qql/script.py
#	tests/test_executor.py
#	tests/test_lexer.py
#	tests/test_parser.py
#	tests/test_script.py

Author: srimon12

README.md

@@ -5,9 +5,9 @@
 [![PyPI version](https://img.shields.io/pypi/v/qql-cli?color=blue&label=PyPI)](https://pypi.org/project/qql-cli/)
 [![Python 3.12+](https://img.shields.io/pypi/pyversions/qql-cli)](https://pypi.org/project/qql-cli/)
 [![MIT License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-375%20passing-brightgreen)](tests/)
+[![Tests](https://img.shields.io/badge/tests-405%20passing-brightgreen)](tests/)
 
-Write `INSERT`, `SELECT`, `SEARCH`, `RECOMMEND`, `DELETE`, and `CREATE COLLECTION` statements instead of Python SDK calls. Supports hybrid dense+sparse vector search, cross-encoder reranking, quantization (scalar, turbo, binary, product), SQL-style `WHERE` filters, script execution, and collection dump/restore.
+Write `INSERT`, `SELECT`, `SEARCH`, `SCROLL`, `RECOMMEND`, `DELETE`, and `CREATE COLLECTION` statements instead of Python SDK calls. Supports hybrid dense+sparse vector search, cross-encoder reranking, quantization (scalar, turbo, binary, product), SQL-style `WHERE` filters, script execution, and collection dump/restore.
 
 ```
 qql> INSERT INTO COLLECTION notes VALUES {'text': 'Qdrant is a vector database', 'author': 'alice', 'year': 2024}
@@ -48,7 +48,7 @@ Your query string
   Qdrant instance
 ```
 
-When you run `INSERT`, the `text` field is automatically converted into a dense vector using [Fastembed](https://github.com/qdrant/fastembed). In **hybrid mode** (`USING HYBRID`), a sparse BM25 vector is also generated alongside the dense vector, and searches use Qdrant's Reciprocal Rank Fusion (RRF) to merge the results of both retrieval methods.
+When you run `INSERT`, the `text` field is automatically converted into a dense vector using [Fastembed](https://github.com/qdrant/fastembed). In **hybrid mode** (`USING HYBRID`), a sparse BM25 vector is also generated alongside the dense vector, and searches use Qdrant's Reciprocal Rank Fusion (RRF) by default to merge the results of both retrieval methods. You can switch hybrid search to DBSF with `FUSION 'dbsf'`.
 
 ---
 
@@ -82,7 +82,7 @@ Full documentation lives in the [`docs/`](docs/) folder and at **[pavanjava.gith
 |---|---|
 | [Getting Started](docs/getting-started.md) | Installation, connecting, first queries |
 | [INSERT / INSERT BULK](docs/insert.md) | Adding documents, batch inserts, payload types |
-| [SEARCH / SELECT / RECOMMEND / Hybrid / RERANK](docs/search.md) | Semantic search, point retrieval, hybrid, reranking, recommendations |
+| [SEARCH / SELECT / SCROLL / RECOMMEND / Hybrid / RERANK](docs/search.md) | Semantic search, point retrieval, pagination, hybrid, reranking, recommendations |
 | [WHERE Filters](docs/filters.md) | Full SQL-style filter operators |
 | [Collections & Quantization](docs/collections.md) | CREATE, DROP, QUANTIZE (scalar/turbo/binary/product), CREATE INDEX |
 | [Scripts: EXECUTE / DUMP](docs/scripts.md) | Script files, collection backup/restore |
@@ -102,8 +102,14 @@ INSERT BULK INTO COLLECTION articles VALUES [{'text': '...'}, {'text': '...'}]
 SEARCH articles SIMILAR TO 'query' LIMIT 10
 SEARCH articles SIMILAR TO 'query' LIMIT 10 WHERE year >= 2020
 SEARCH articles SIMILAR TO 'query' LIMIT 10 USING HYBRID
+SEARCH articles SIMILAR TO 'query' LIMIT 10 USING HYBRID FUSION 'dbsf'
 SEARCH articles SIMILAR TO 'query' LIMIT 10 USING HYBRID RERANK
 
+-- Scroll
+SCROLL FROM articles LIMIT 50
+SCROLL FROM articles WHERE year >= 2024 LIMIT 50
+SCROLL FROM articles AFTER 'cursor-id' LIMIT 50
+
 -- Recommend
 RECOMMEND FROM articles POSITIVE IDS (1001, 1002) LIMIT 5
 
@@ -140,7 +146,7 @@ Tests do not require a running Qdrant instance — the Qdrant client is mocked.
 pytest tests/ -v
 ```
 
-Expected: **375 tests passing**.
+Expected: **405 tests passing**.
 
 ---
 

docs/getting-started.md

@@ -24,7 +24,7 @@ Your query string
   Qdrant instance
 ```
 
-When you run `INSERT`, the `text` field is automatically converted into a dense vector using [Fastembed](https://github.com/qdrant/fastembed). In **hybrid mode** (`USING HYBRID`), a sparse BM25 vector is also generated alongside the dense vector, and searches use Qdrant's Reciprocal Rank Fusion (RRF) to merge the results of both retrieval methods.
+When you run `INSERT`, the `text` field is automatically converted into a dense vector using [Fastembed](https://github.com/qdrant/fastembed). In **hybrid mode** (`USING HYBRID`), a sparse BM25 vector is also generated alongside the dense vector, and searches use Qdrant's Reciprocal Rank Fusion (RRF) by default to merge the results of both retrieval methods. You can override that with `FUSION 'dbsf'` on hybrid searches.
 
 ---
 
@@ -138,6 +138,9 @@ SEARCH notes SIMILAR TO 'vector storage engines' LIMIT 3
 -- Filter results
 SEARCH notes SIMILAR TO 'vector databases' LIMIT 5 WHERE year >= 2023
 
+-- Browse with pagination
+SCROLL FROM notes LIMIT 10
+
 -- List all collections
 SHOW COLLECTIONS
 
@@ -150,7 +153,7 @@ SELECT * FROM notes WHERE id = 1
 ## Next Steps
 
 - [INSERT / INSERT BULK](insert.md) — adding documents
-- [SEARCH / SELECT / RECOMMEND / Hybrid / RERANK](search.md) — querying
+- [SEARCH / SELECT / SCROLL / RECOMMEND / Hybrid / RERANK](search.md) — querying
 - [WHERE Filters](filters.md) — payload filtering
 - [Collections & Quantization](collections.md) — managing collections
 - [Scripts: EXECUTE / DUMP](scripts.md) — automating with script files

docs/index.html

@@ -114,7 +114,7 @@ <h1>QQL</h1>
     <a href="https://pypi.org/project/qql-cli/"><img src="https://img.shields.io/pypi/v/qql-cli?color=blue&label=PyPI" alt="PyPI version" /></a>
     <a href="https://pypi.org/project/qql-cli/"><img src="https://img.shields.io/pypi/pyversions/qql-cli" alt="Python versions" /></a>
     <a href="https://github.com/pavanjava/qql/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License" /></a>
-    <a href="https://github.com/pavanjava/qql/actions"><img src="https://img.shields.io/badge/tests-375%20passing-brightgreen" alt="375 tests" /></a>
+    <a href="https://github.com/pavanjava/qql/actions"><img src="https://img.shields.io/badge/tests-405%20passing-brightgreen" alt="405 tests" /></a>
   </div>
 
   <pre><span class="cmt"># Install</span>
@@ -148,8 +148,8 @@ <h3>INSERT / INSERT BULK</h3>
       <p>Adding documents, batch inserts, payload types</p>
     </a>
     <a class="card" href="search">
-      <h3>SEARCH / SELECT / RECOMMEND</h3>
-      <p>Semantic search, point retrieval, hybrid search, reranking, recommendations</p>
+      <h3>SEARCH / SELECT / SCROLL / RECOMMEND</h3>
+      <p>Semantic search, point retrieval, pagination, hybrid search, reranking, recommendations</p>
     </a>
     <a class="card" href="filters">
       <h3>WHERE Filters</h3>

docs/programmatic.md

@@ -40,6 +40,15 @@ result = run_query(
 for hit in result.data:
     print(hit["score"], hit["payload"])
 
+# Scroll / pagination
+result = run_query(
+    "SCROLL FROM notes LIMIT 2",
+    url="http://localhost:6333",
+)
+for point in result.data["points"]:
+    print(point["id"], point["payload"])
+print(result.data["next_offset"])
+
 # Bulk insert (all records embedded and upserted in one call)
 result = run_query(
     """INSERT BULK INTO COLLECTION notes VALUES [
@@ -120,6 +129,7 @@ class ExecutionResult:
 | INSERT BULK | `None` (count in `result.message`) |
 | SELECT | `{"id": str, "payload": dict}` or `None` when not found |
 | SEARCH | `[{"id": str, "score": float, "payload": dict}, ...]` |
+| SCROLL | `{"points": [{"id": str, "payload": dict}, ...], "next_offset": str \| None}` |
 | RECOMMEND | `[{"id": str, "score": float, "payload": dict}, ...]` |
 | SHOW COLLECTIONS | `["name1", "name2", ...]` |
 | CREATE COLLECTION | `None` |

docs/reference.md

@@ -36,6 +36,9 @@ SEARCH docs SIMILAR TO 'hello' LIMIT 5 USING MODEL 'BAAI/bge-small-en-v1.5'
 -- Hybrid with custom dense model
 SEARCH docs SIMILAR TO 'hello' LIMIT 5 USING HYBRID DENSE MODEL 'BAAI/bge-base-en-v1.5'
 
+-- Hybrid with explicit fusion strategy
+SEARCH docs SIMILAR TO 'hello' LIMIT 5 USING HYBRID FUSION 'dbsf'
+
 -- Hybrid with both custom
 SEARCH docs SIMILAR TO 'hello' LIMIT 5
   USING HYBRID DENSE MODEL 'BAAI/bge-base-en-v1.5' SPARSE MODEL 'prithivida/Splade_PP_en_v1'
@@ -159,7 +162,7 @@ Tests do not require a running Qdrant instance — the Qdrant client is mocked.
 pytest tests/ -v
 ```
 
-Expected output: **375 tests passing**.
+Expected output: **405 tests passing**.
 
 ---
 
@@ -171,13 +174,14 @@ Expected output: **375 tests passing**.
 | `Connection failed: ...` | Qdrant unreachable at given URL | Check that Qdrant is running and the URL is correct |
 | `INSERT requires a 'text' field in VALUES` | `text` key missing from the VALUES dict | Add `'text': '...'` to your dict |
 | `Vector dimension mismatch: collection '...' expects X dims, but model produces Y dims` | Model used in INSERT differs from the one used to create the collection | Use `USING MODEL` to specify the same model as the collection was created with |
-| `Collection '...' does not exist` | SEARCH / SELECT / DROP / DELETE on a non-existent collection | Check name spelling or run `SHOW COLLECTIONS` |
+| `Collection '...' does not exist` | SEARCH / SCROLL / SELECT / DROP / DELETE on a non-existent collection | Check name spelling or run `SHOW COLLECTIONS` |
 | `Unexpected token '...'; expected a QQL statement keyword` | Unrecognized statement | Check the query syntax and supported statement list |
 | `SELECT requires a string or integer point id, got '...'` | `SELECT` used with a non-ID filter value | Use `SELECT * FROM <collection> WHERE id = '<id>'` or an integer ID |
 | `Unterminated string literal (at position N)` | A string is missing its closing quote | Close the string with a matching `'` or `"` |
 | `Unexpected character '@' (at position N)` | A character not part of QQL syntax | Remove or quote the offending character |
 | `Expected a filter operator after field '...'` | Unknown operator in WHERE clause | Use one of: `=`, `!=`, `>`, `>=`, `<`, `<=`, `IN`, `NOT IN`, `BETWEEN`, `IS NULL`, `IS NOT NULL`, `IS EMPTY`, `IS NOT EMPTY`, `MATCH` |
 | `Expected ')' ...` | Unclosed parenthesis in WHERE clause | Add the missing `)` to close the group |
 | `Qdrant error during SEARCH: ...` | Hybrid search on a non-hybrid collection, or wrong vector names | Ensure the collection was created with `HYBRID` before using `USING HYBRID` in INSERT/SEARCH |
+| `Qdrant error during SCROLL: ...` | Qdrant rejected scroll request | Verify collection state, filter, and cursor (`AFTER`) value |
 | `Unknown index type '...'` | Invalid schema type in CREATE INDEX | Use one of: `keyword`, `integer`, `float`, `bool`, `text`, `geo`, `datetime` |
 | `Qdrant error during CREATE INDEX: ...` | Qdrant rejected the index creation | Check field name and collection state |

docs/search.md

@@ -1,4 +1,4 @@
-# SEARCH, SELECT, RECOMMEND, Hybrid Search & Reranking
+# SEARCH, SELECT, SCROLL, RECOMMEND, Hybrid Search & Reranking
 
 ---
 
@@ -14,7 +14,7 @@ SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n>
 SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> USING MODEL '<model_name>'
 SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> [USING MODEL '<model>'] WHERE <filter>
 SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> USING HYBRID
-SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> USING HYBRID [DENSE MODEL '<model>'] [SPARSE MODEL '<model>'] [WHERE <filter>]
+SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> USING HYBRID [FUSION 'rrf|dbsf'] [DENSE MODEL '<model>'] [SPARSE MODEL '<model>'] [WHERE <filter>]
 SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> USING SPARSE [MODEL '<sparse_model>']
 SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> EXACT
 SEARCH <collection_name> SIMILAR TO '<query_text>' LIMIT <n> [USING ...] [WHERE <filter>] [RERANK] WITH { hnsw_ef: <n>, exact: true|false, acorn: true|false }
@@ -33,7 +33,7 @@ Search only papers published after 2020:
 SEARCH articles SIMILAR TO 'deep learning' LIMIT 10 WHERE year > 2020
 ```
 
-Hybrid search (combines dense semantic + sparse BM25 keyword retrieval via RRF):
+Hybrid search (combines dense semantic + sparse BM25 keyword retrieval via RRF by default):
 ```sql
 SEARCH articles SIMILAR TO 'attention mechanism' LIMIT 10 USING HYBRID
 ```
@@ -120,15 +120,41 @@ SEARCH articles SIMILAR TO 'RAG' LIMIT 10 WHERE tag = 'li' WITH { acorn: true }
 
 ---
 
+## SCROLL — pagination / browsing
+
+Use `SCROLL` to iterate through points in a collection page by page.
+
+**Syntax:**
+```sql
+SCROLL FROM <collection_name> LIMIT <n>
+SCROLL FROM <collection_name> WHERE <filter> LIMIT <n>
+SCROLL FROM <collection_name> AFTER '<point_id>' LIMIT <n>
+SCROLL FROM <collection_name> WHERE <filter> AFTER <point_id> LIMIT <n>
+```
+
+**Examples:**
+```sql
+SCROLL FROM articles LIMIT 50
+SCROLL FROM articles WHERE year >= 2024 LIMIT 50
+SCROLL FROM articles AFTER 'cursor-id' LIMIT 50
+```
+
+**Behavior:**
+- Returns points in ID order with payloads.
+- Returns a `next_offset` cursor when more points are available.
+- Use `AFTER <next_offset>` to fetch the next page.
+
+---
+
 ## Hybrid Search (USING HYBRID)
 
-Hybrid search combines **dense semantic vectors** and **sparse BM25 keyword vectors** in a single query and merges the results with Qdrant's **Reciprocal Rank Fusion (RRF)** algorithm. This typically outperforms either method alone.
+Hybrid search combines **dense semantic vectors** and **sparse BM25 keyword vectors** in a single query. By default QQL merges the two result sets with Qdrant's **Reciprocal Rank Fusion (RRF)** algorithm, and you can optionally switch to **DBSF** with a `FUSION` clause.
 
 ### How it works internally
 
 1. Both a dense vector (`TextEmbedding`) and a sparse BM25 vector (`SparseTextEmbedding`) are generated from your query text.
 2. Qdrant fetches the top candidates from each index independently (`prefetch limit = LIMIT × 4`).
-3. The two result lists are merged using RRF — a rank-based fusion that does not require score normalization.
+3. The two result lists are merged using the selected fusion strategy (`RRF` by default, or `DBSF` when requested).
 4. The final top-N results are returned.
 
 ### Step 1: Create a hybrid collection
@@ -161,6 +187,9 @@ SEARCH articles SIMILAR TO 'transformer architecture' LIMIT 10 USING HYBRID
 -- Hybrid search with a WHERE filter
 SEARCH articles SIMILAR TO 'attention' LIMIT 10 USING HYBRID WHERE year >= 2017
 
+-- Hybrid with DBSF fusion
+SEARCH articles SIMILAR TO 'hybrid retrieval' LIMIT 10 USING HYBRID FUSION 'dbsf'
+
 -- Hybrid with custom dense model
 SEARCH articles SIMILAR TO 'embeddings' LIMIT 5
   USING HYBRID DENSE MODEL 'BAAI/bge-base-en-v1.5'
@@ -176,6 +205,7 @@ SEARCH articles SIMILAR TO 'sparse retrieval' LIMIT 5
 |---|---|
 | Dense model | configured default (`sentence-transformers/all-MiniLM-L6-v2`) |
 | Sparse model | `Qdrant/bm25` |
+| Fusion | `rrf` |
 
 ### Dense vs. hybrid — when to use which
 

src/qql/ast_nodes.py

@@ -186,13 +186,22 @@ class SelectStmt:
     point_id: str | int
 
 
+@dataclass(frozen=True)
+class ScrollStmt:
+    collection: str
+    limit: int
+    query_filter: FilterExpr | None = None
+    after: str | int | None = None
+
+
 @dataclass(frozen=True)
 class SearchStmt:
     collection: str
     query_text: str
     limit: int
     model: str | None               # dense model; None → use config default
     hybrid: bool = False            # if True, use prefetch+RRF hybrid search
+    fusion: str | None = None       # hybrid fusion strategy; None → default rrf
     sparse_only: bool = False       # if True, query only the sparse vector (no dense)
     sparse_model: str | None = None # sparse model for hybrid/sparse-only; None → SparseEmbedder.DEFAULT_MODEL
     query_filter: FilterExpr | None = None  # optional WHERE clause; default keeps existing tests valid
@@ -232,6 +241,7 @@ class DeleteStmt:
     | DropCollectionStmt
     | ShowCollectionsStmt
     | SelectStmt
+    | ScrollStmt
     | SearchStmt
     | RecommendStmt
     | DeleteStmt

src/qql/cli.py

@@ -49,13 +49,18 @@
   [yellow]SHOW COLLECTIONS[/yellow]
       List all collections in the connected Qdrant instance.
 
+  [yellow]SCROLL FROM[/yellow] <name> [yellow]LIMIT[/yellow] <n>
+      Paginate points by ID order.
+      Optional: [yellow]WHERE[/yellow] <filter>
+      Optional: [yellow]AFTER[/yellow] '<id>'|<int>
+
   [yellow]SELECT * FROM[/yellow] <name> [yellow]WHERE id =[/yellow] '<id>'|<int>
       Retrieve a single point by its ID and return its payload.
 
   [yellow]SEARCH[/yellow] <name> [yellow]SIMILAR TO[/yellow] '<text>' [yellow]LIMIT[/yellow] <n>
       Semantic search by vector similarity.
       Optional: [yellow]USING MODEL[/yellow] '<model>'
-      Optional: [yellow]USING HYBRID[/yellow] [DENSE MODEL '<model>'] [SPARSE MODEL '<model>']
+      Optional: [yellow]USING HYBRID[/yellow] [FUSION 'rrf|dbsf'] [DENSE MODEL '<model>'] [SPARSE MODEL '<model>']
       Optional: [yellow]USING SPARSE[/yellow] [MODEL '<model>']   sparse-vector-only search
       Optional: [yellow]WHERE[/yellow] <filter>   (e.g. WHERE year > 2020 AND status = 'ok')
       Optional: [yellow]RERANK[/yellow] [MODEL '<model>']   rerank results with a cross-encoder
@@ -403,6 +408,20 @@ def _run_and_print(executor: Executor, query: str) -> None:
         console.print(table)
         return
 
+    # Pretty-print scroll results
+    if isinstance(result.data, dict) and "points" in result.data and "next_offset" in result.data:
+        points = result.data["points"]
+        if points:
+            table = Table(show_header=True, header_style="bold cyan")
+            table.add_column("ID")
+            table.add_column("Payload")
+            for point in points:
+                table.add_row(point["id"], str(point["payload"]))
+            console.print(table)
+        if result.data["next_offset"] is not None:
+            console.print(f"[dim]next_offset: {result.data['next_offset']}[/dim]")
+        return
+
     # Pretty-print SELECT result
     if isinstance(result.data, dict) and "id" in result.data and "payload" in result.data:
         table = Table(show_header=True, header_style="bold cyan")