Skip to content

working with script files is here #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pavanjava merged 1 commit into from
Apr 19, 2026
Merged

working with script files is here #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 156 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ qql> SEARCH notes SIMILAR TO 'vector databases' LIMIT 5 USING HYBRID RERANK
- [The QQL Shell](#the-qql-shell)
- [All QQL Operations](#all-qql-operations)
- [INSERT — add a point](#insert--add-a-point)
- [INSERT BULK — batch insert](#insert-bulk--batch-insert-multiple-points)
- [SEARCH — find similar points](#search--find-similar-points)
- [Query-Time Search Params (`EXACT`, `WITH`)](#query-time-search-params-exact-with)
- [WHERE Clause Filters](#where-clause-filters)
Expand All @@ -44,6 +45,9 @@ qql> SEARCH notes SIMILAR TO 'vector databases' LIMIT 5 USING HYBRID RERANK
- [CREATE COLLECTION — create a collection](#create-collection--create-a-collection)
- [DROP COLLECTION — delete a collection](#drop-collection--delete-a-collection)
- [DELETE — remove a point](#delete--remove-a-point)
- [Script Files](#script-files)
- [EXECUTE — run a script file](#execute--run-a-qql-script-file)
- [DUMP COLLECTION — export to script](#dump-collection--export-collection-to-a-qql-script-file)
- [Embedding Models](#embedding-models)
- [Value Types in Dictionaries](#value-types-in-dictionaries)
- [Configuration File](#configuration-file)
Expand Down Expand Up @@ -838,6 +842,158 @@ To find a point's ID, run a SEARCH first and copy the ID from the results table.

---

## Script Files

QQL supports reading from and writing to `.qql` script files, making it easy to automate bulk operations, seed databases, and back up collections.

---

### EXECUTE — run a .qql script file

Execute a file containing multiple QQL statements in sequence. Each statement is parsed and executed in order. `--` comments are stripped before parsing.

**CLI usage:**
```bash
qql execute /path/to/script.qql

# Stop on first error instead of continuing through all statements
qql execute /path/to/script.qql --stop-on-error
```

**In-shell usage (inside the QQL REPL):**
```
qql> EXECUTE /path/to/script.qql
qql> \e /path/to/script.qql
```

**Script format:**

```sql
-- This is a comment — the entire line is ignored
-- ============================================================
-- QQL Script — populate articles collection
-- ============================================================

-- Step 1: create the collection
CREATE COLLECTION articles

-- Step 2: bulk insert records
INSERT BULK INTO COLLECTION articles VALUES [
{'text': 'Neural networks learn representations', 'year': 2023},
{'text': 'Attention mechanisms in transformers', 'year': 2024}
]

-- Step 3: verify
SHOW COLLECTIONS
```

**Rules:**
- `--` to end-of-line is a comment and is ignored (inline or full-line)
- Statements can span multiple lines (e.g. `INSERT BULK ... VALUES [...]`)
- Blank lines between statements are ignored
- By default all statements run even if one fails; use `--stop-on-error` to halt early

**Example output:**
```
Executing: /path/to/script.qql

[1/3] CREATE COLLECTION articles
✓ Collection 'articles' created (384-dimensional vectors, cosine distance)
[2/3] INSERT BULK INTO COLLECTION articles VALUES [ …
✓ Inserted 2 points
[3/3] SHOW COLLECTIONS
✓ 1 collection(s) found

Done. 3/3 statement(s) succeeded.
```

---

### DUMP COLLECTION — export collection to a .qql script file

Export every point in a collection to a `.qql` script file. The generated file is valid QQL — it can be re-imported with `qql execute` to restore or migrate the collection. Points are written in batches of 50 as `INSERT BULK` statements.

**CLI usage:**
```bash
qql dump <collection_name> <output.qql>
```

**In-shell usage (inside the QQL REPL):**
```
qql> DUMP COLLECTION <name> <output.qql>
```

**Example:**
```bash
qql dump medical_records /tmp/medical_records.qql
```

```
Dumping: 'medical_records' → /tmp/medical_records.qql

Collection type : hybrid (dense + sparse)
Points : 41
Batches : 1 (50 points/batch)

[1/1] wrote 41 point(s)

Done. 41 point(s) written.
```

**Generated file structure:**
```sql
-- ============================================================
-- QQL Dump — collection: medical_records
-- Generated : 2026-04-19 14:32:11
-- Points : 41
-- Type : hybrid (dense + sparse)
-- Note : Re-importing re-embeds all text using the
-- configured model (see: qql connect).
-- ============================================================

CREATE COLLECTION medical_records HYBRID

-- Batch 1 / 1 (records 1–41)
INSERT BULK INTO COLLECTION medical_records VALUES [
{
'text': 'Alzheimers disease is characterized by...',
'title': 'Alzheimers Disease Overview',
'department': 'neurology',
'year': 2023,
'peer_reviewed': true
},
...
] USING HYBRID

-- ============================================================
-- End of dump
-- Written : 41
-- Skipped : 0 (no 'text' field)
-- ============================================================
```

**Round-trip workflow — backup and restore:**
```bash
# 1. Dump the collection
qql dump medical_records backup.qql

# 2. Drop it
qql> DROP COLLECTION medical_records

# 3. Restore from the dump
qql execute backup.qql
```

**Rules and notes:**
- Points without a `'text'` payload field are **skipped** (counted in the footer comment).
- Hybrid collections produce `CREATE COLLECTION <name> HYBRID` and `INSERT BULK ... USING HYBRID` statements.
- Dense collections produce plain `CREATE COLLECTION <name>` and `INSERT BULK` statements.
- All payload value types are preserved: strings, integers, floats, booleans (`true`/`false`), `null`, lists, and nested dicts.
- Re-importing re-embeds all text using your currently configured model — use the same model as the original collection to preserve semantic accuracy.
- Parent directories of the output path are created automatically.

---

## Embedding Models

QQL uses [Fastembed](https://github.com/qdrant/fastembed) to convert text into vectors locally — no external API call is needed.
Expand Down
168 changes: 168 additions & 0 deletions src/qql/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,15 @@
[yellow]DELETE FROM[/yellow] <name> [yellow]WHERE id =[/yellow] '<id>'
Delete a point by its ID.

Script files (in-shell):
[yellow]EXECUTE[/yellow] <path> or [yellow]\\e[/yellow] <path>
Run a .qql script file. Statements are executed in order.
Lines starting with [yellow]--[/yellow] are treated as comments and ignored.

[yellow]DUMP[/yellow] <name> <output.qql> or [yellow]DUMP COLLECTION[/yellow] <name> <output.qql>
Export all points in a collection to a .qql script file.
The file can be re-imported with EXECUTE.

Keyboard shortcuts:
← → arrows move cursor within the current line
↑ ↓ arrows scroll through command history
Expand Down Expand Up @@ -119,6 +128,109 @@ def disconnect() -> None:
console.print("Disconnected. Config removed.")


# ── execute ────────────────────────────────────────────────────────────────────

@main.command()
@click.argument("file", type=click.Path(exists=True, readable=True))
@click.option(
"--stop-on-error",
is_flag=True,
default=False,
help="Halt execution on the first statement error (default: continue all).",
)
def execute(file: str, stop_on_error: bool) -> None:
"""Execute a .qql script file against the connected Qdrant instance.

Lines beginning with -- are treated as comments and skipped.
Each QQL statement is executed in order and its result is printed.
"""
from qdrant_client import QdrantClient

cfg = load_config()
if cfg is None:
err_console.print(
"[bold red]Not connected.[/bold red] "
"Run: [bold]qql connect --url <url>[/bold]"
)
sys.exit(1)

try:
client = QdrantClient(url=cfg.url, api_key=cfg.secret)
client.get_collections()
except Exception as e:
err_console.print(f"[bold red]Connection failed:[/bold red] {e}")
sys.exit(1)

from .executor import Executor
from .script import run_script

executor = Executor(client, cfg)
console.print(f"[bold cyan]Executing:[/bold cyan] {file}\n")

ok, fail = run_script(file, executor, console, err_console, stop_on_error)
total = ok + fail

if fail == 0:
console.print(
f"\n[bold green]Done.[/bold green] "
f"{total}/{total} statement(s) succeeded."
)
else:
console.print(
f"\n[bold yellow]Done.[/bold yellow] "
f"{ok}/{total} succeeded, [bold red]{fail} failed[/bold red]."
)
sys.exit(1)


# ── dump ───────────────────────────────────────────────────────────────────────

@main.command()
@click.argument("collection")
@click.argument("output", type=click.Path())
def dump(collection: str, output: str) -> None:
"""Dump a collection to a .qql script file.

OUTPUT is the path for the generated .qql file.
The file contains CREATE COLLECTION + INSERT BULK statements and can be
re-imported with: qql execute <output>
"""
from qdrant_client import QdrantClient

cfg = load_config()
if cfg is None:
err_console.print(
"[bold red]Not connected.[/bold red] "
"Run: [bold]qql connect --url <url>[/bold]"
)
sys.exit(1)

try:
client = QdrantClient(url=cfg.url, api_key=cfg.secret)
client.get_collections()
except Exception as e:
err_console.print(f"[bold red]Connection failed:[/bold red] {e}")
sys.exit(1)

from .dumper import dump_collection

console.print(
f"[bold cyan]Dumping:[/bold cyan] '{collection}' → {output}\n"
)
written, skipped = dump_collection(collection, output, client, console, err_console)

if written == 0 and skipped == 0:
# collection not found — error already printed by dump_collection
sys.exit(1)

console.print(
f"\n[bold green]Done.[/bold green] "
f"{written} point(s) written"
+ (f", [yellow]{skipped} skipped[/yellow] (no 'text' field)" if skipped else "")
+ f"."
)


# ── REPL ───────────────────────────────────────────────────────────────────────

def _launch_repl(cfg: QQLConfig) -> None:
Expand Down Expand Up @@ -161,6 +273,62 @@ def _launch_repl(cfg: QQLConfig) -> None:
console.print(HELP_TEXT)
continue

# ── EXECUTE <path> / \e <path> — run a .qql script file ──────────
if low.startswith("execute ") or low.startswith("\\e "):
script_path = query.split(None, 1)[1].strip()
from .script import run_script
ok, fail = run_script(script_path, executor, console, err_console)
total = ok + fail
if fail == 0:
console.print(
f"[bold green]Done.[/bold green] "
f"{total}/{total} statement(s) succeeded."
)
else:
console.print(
f"[bold yellow]Done.[/bold yellow] "
f"{ok}/{total} succeeded, [bold red]{fail} failed[/bold red]."
)
continue

# ── DUMP [COLLECTION] <name> <file> — export collection to .qql ──
# Accepts both:
# DUMP COLLECTION <name> <output.qql>
# DUMP <name> <output.qql>
if low.startswith("dump "):
parts = query.split(None, 3) # up to 4 tokens
if len(parts) >= 2 and parts[1].lower() == "collection":
# DUMP COLLECTION <name> <file>
if len(parts) < 4:
err_console.print(
"[bold red]Usage:[/bold red] DUMP COLLECTION <name> <output.qql>"
)
continue
coll_name, out_path = parts[2], parts[3]
else:
# DUMP <name> <file>
if len(parts) < 3:
err_console.print(
"[bold red]Usage:[/bold red] DUMP <name> <output.qql>"
)
continue
coll_name, out_path = parts[1], parts[2]
from .dumper import dump_collection
console.print(
f"[bold cyan]Dumping:[/bold cyan] '{coll_name}' → {out_path}\n"
)
written, skipped = dump_collection(
coll_name, out_path, client, console, err_console
)
if written > 0 or skipped == 0:
console.print(
f"[bold green]Done.[/bold green] "
f"{written} point(s) written"
+ (f", [yellow]{skipped} skipped[/yellow] (no 'text' field)" if skipped else "")
+ "."
)
continue

_run_and_print(executor, query)


Expand Down
Loading
Loading