Skip to content

docs: add roadmap and contributor guides #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srimon12 wants to merge 1 commit into from
Closed

docs: add roadmap and contributor guides #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .github/RFC_TEMPLATE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# RFC: <Short Title>

- **Status:** Draft / Proposed / Accepted / Rejected / Implemented
- **Author:** @github-username
- **Target:** Python / Go / Both
- **Created:** YYYY-MM-DD

## Summary

Describe the proposed change in one paragraph.

## Motivation

What problem does this solve?

- Current pain:
- Who is affected:
- Why QQL should expose this:

## Proposed Syntax

```sql
-- Minimal example
NEW SYNTAX ...

-- Example with optional clauses
NEW SYNTAX ... WHERE ... WITH { ... }
```

## Qdrant Mapping

| QQL syntax | Qdrant API/model |
|---|---|
| `...` | `...` |

If Qdrant does not directly support the behavior, explain why QQL should still add it.

## Output

Human-readable output:

```text
...
```

JSON output:

```json
{
"success": true,
"message": "...",
"data": {}
}
```

## Compatibility

- Does this break existing QQL scripts?
- Does this affect JSON output contracts?
- Should Python and Go match?
- Can one implementation ship first?

## Implementation Plan

1. Lexer/parser/AST changes
2. Executor changes
3. Tests
4. Documentation

## Alternatives

List simpler or competing designs and why they were not chosen.

## Open Questions

- Question 1
- Question 2

## References

- Qdrant docs:
- Related issues:
- Related PRs:
92 changes: 92 additions & 0 deletions ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# QQL Roadmap

> **Status:** Draft for maintainer and community discussion
> **Scope:** Public direction for the Python `qql-cli` project, with companion notes for the Go implementation where parity matters
> **Principle:** Keep QQL small, readable, and close to Qdrant's real API surface

QQL is a SQL-like language and CLI for common Qdrant workflows. The near-term goal is not to cover every Qdrant feature. The goal is to make everyday vector database work easier to read, script, test, and share.

## Current Position

The Python implementation already supports the core workflow:

| Area | Status |
|---|---|
| Collection create/drop/list | Supported |
| Payload indexes | Supported |
| Insert and bulk insert | Supported |
| Dense search | Supported |
| Hybrid dense+sparse search | Supported |
| Sparse-only search | Supported |
| WHERE filters | Supported |
| Recommend by example IDs | Supported |
| Query-time search params | Supported |
| Reranking | Supported in Python |
| Delete by ID or filter | Supported |
| Script execution and dump/restore | Supported |
| Programmatic Python API | Supported through `run_query()` |

The Go implementation is developed separately. It should aim for language and behavior parity where practical, but this Python repository should not block on Go work before improving its own CLI and documentation.

## Near-Term Priorities

These are the best candidates for small, useful contributions. Each one should have tests and documentation before being considered complete.

| Priority | Feature | Why it matters | Suggested syntax |
|---|---|---|---|
| P0 | Get point by ID | Basic inspection is currently missing | `GET FROM <collection> WHERE id = '<id>'` |
| P0 | Scroll points | Needed for real datasets and exports | `SCROLL <collection> LIMIT 100` |
| P0 | Search pagination | Needed for browsing result sets | `SEARCH ... LIMIT 10 OFFSET 20` |
| P1 | Count points | Useful for validation and scripts | `COUNT <collection> WHERE <filter>` |
| P1 | Describe collection | Improves introspection and debugging | `DESCRIBE COLLECTION <name>` |
| P1 | Update payload | Avoids full reinsert for metadata changes | `UPDATE <collection> SET {...} WHERE id = '<id>'` |
| P1 | Delete payload keys | Removes fields without deleting points | `DELETE PAYLOAD field FROM <collection> WHERE id = '<id>'` |

## Later Ideas

These are worth exploring, but they should not distract from the smaller parity gaps above.

| Area | Possible work |
|---|---|
| Retrieval quality | MMR, score boosting, named vector search, batch search |
| Collection configuration | Distance selection, HNSW config, quantization, on-disk payload |
| Developer experience | Connection profiles, clearer JSON contracts, better error messages |
| Ecosystem | Syntax highlighting, examples, tutorials, migration guides |
| Operations | Collection aliases, snapshots, backup/restore workflows |

## Contribution Process

Use an RFC when a change affects syntax, CLI behavior, or JSON output. Small documentation fixes, tests, and bug fixes do not need an RFC.

Good roadmap issues should include:

- the Qdrant API being exposed
- the proposed QQL syntax
- expected human-readable output
- expected JSON output
- Python tests required
- Go parity notes, if relevant

## Documentation Goals

The documentation should stay practical:

- README: quick start and common usage
- `docs/syntax/`: compact syntax reference
- `docs/COMPATIBILITY.md`: checked feature matrix
- `docs/CONTRIBUTING.md`: contributor workflow
- `docs/RFCS/`: proposed and accepted syntax decisions
- `docs/TUTORIALS/`: runnable examples as they are added
- `docs/MIGRATING/`: focused migration notes

## Success Criteria

QQL is moving in the right direction when:

- users can inspect, insert, search, recommend, update, count, and export without dropping to raw SDK calls for common cases
- syntax changes are discussed before implementation
- docs describe what is implemented today, not only what is planned
- Python and Go differences are visible and intentional
- contributors can find small, well-scoped issues

This roadmap is intentionally modest. It should be revised as maintainers and contributors agree on scope.
50 changes: 50 additions & 0 deletions SHARED.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Cross-Repository Documentation Notes

QQL has a Python implementation (`qql-cli`) and a companion Go implementation (`qql-go`). Some documentation is useful to keep conceptually aligned across both projects, but each repository remains responsible for its own implementation details.

This file is a coordination guide, not a hard synchronization system.

## Shared in Spirit

These documents should use the same terminology and avoid contradicting each other across implementations:

| File | Purpose |
|---|---|
| `ROADMAP.md` | Project direction and priority areas |
| `.github/RFC_TEMPLATE.md` | Template for syntax and behavior proposals |
| `.github/ISSUE_LABELS.md` | Suggested issue label taxonomy |
| `docs/CONTRIBUTING.md` | Contributor workflow |
| `docs/SYNTAX_GUIDELINES.md` | How to add or change QQL syntax |
| `docs/COMPATIBILITY.md` | Feature matrix across Qdrant, Python, and Go |
| `docs/RFCS/README.md` | RFC process overview |

## Repository-Specific

These files should normally stay different:

| Python `qql` | Go `qql-go` | Why |
|---|---|---|
| `README.md` | `README.md` | Different install, command, and release details |
| `pyproject.toml` | `go.mod` | Different package managers |
| `src/qql/` | Go source tree | Different implementations |
| `tests/` | Go tests | Different test frameworks |
| release notes | release notes | Different version history |

## Update Guidance

When a change affects the QQL language rather than one implementation:

1. Update the local documentation.
2. Note whether the behavior is Python-only, Go-only, or shared.
3. If the companion implementation is affected, open or link a tracking issue there.
4. Avoid blocking one implementation's documentation on the other unless the feature requires true lockstep behavior.

## Long-Term Options

If cross-repo drift becomes painful, consider one of these later:

- a small `qql-spec` repository for syntax and compatibility docs
- a CI check that compares selected docs between repos
- release notes that explicitly call out Python and Go parity gaps

For now, keep the process lightweight and accurate.
138 changes: 138 additions & 0 deletions docs/COMPATIBILITY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# QQL / Qdrant Compatibility Matrix

> Tracks the current Python `qql-cli` surface and known companion status for `qql-go`.
> Last checked: 2026-04-28.

This document should describe implemented behavior conservatively. If a feature is planned but not implemented, keep it marked as missing until tests exist.

## Legend

| Symbol | Meaning |
|---|---|
| Supported | Implemented and covered by normal usage/tests |
| Partial | Implemented with known limits |
| Missing | Not currently exposed by QQL |
| Planned | Roadmap or RFC candidate |
| Unknown | Needs verification in that implementation |

## Collection Management

| Feature | Qdrant API | Python `qql-cli` | Go `qql-go` | Notes |
|---|---|---|---|---|
| Create collection | `create_collection` | Supported | Supported | Dense collection |
| Create hybrid collection | `create_collection` with sparse vectors | Supported | Supported | `CREATE COLLECTION ... HYBRID` |
| Create with custom distance | Vector params | Missing | Missing | Currently cosine-only in Python |
| Create with custom HNSW | `hnsw_config` | Missing | Missing | Roadmap candidate |
| Create with quantization | `quantization_config` | Missing | Missing | Roadmap candidate |
| Create with on-disk payload | `on_disk_payload` | Missing | Missing | Roadmap candidate |
| Create with multivectors | `multivector_config` | Missing | Missing | Advanced roadmap candidate |
| Drop collection | `delete_collection` | Supported | Supported | `DROP COLLECTION` |
| List collections | `get_collections` | Supported | Supported | `SHOW COLLECTIONS` |
| Collection info | `get_collection` | Missing | Missing | Proposed as `DESCRIBE COLLECTION` |
| Collection aliases | Alias APIs | Missing | Missing | Later idea |
| Collection snapshots | Snapshot APIs | Missing | Missing | Later idea |

## Points / Documents

| Feature | Qdrant API | Python `qql-cli` | Go `qql-go` | Notes |
|---|---|---|---|---|
| Insert point | `upsert` | Supported | Supported | Requires a `text` field for embedding |
| Insert bulk | `upsert` | Supported | Supported | `INSERT BULK` |
| Explicit point ID on insert | `upsert` | Supported | Supported | Integer or UUID string |
| Get point by ID | `retrieve` | Missing | Missing | Near-term roadmap candidate |
| Update payload | `set_payload` | Missing | Missing | Near-term roadmap candidate |
| Delete point by ID | `delete` | Supported | Supported | `DELETE ... WHERE id = ...` |
| Delete points by filter | `delete` with filter selector | Supported | Supported | Python parser/executor support non-ID filters |
| Delete payload keys | `delete_payload` | Missing | Missing | Near-term roadmap candidate |
| Count points | `count` | Missing | Missing | Near-term roadmap candidate |
| Scroll points | `scroll` | Missing | Missing | Near-term roadmap candidate |

## Search

| Feature | Qdrant API | Python `qql-cli` | Go `qql-go` | Notes |
|---|---|---|---|---|
| Dense search | `query_points` | Supported | Supported | Default mode |
| Hybrid search | `query_points` + RRF | Supported | Supported | `USING HYBRID` |
| Sparse-only search | `query_points` sparse vector | Supported | Supported | `USING SPARSE` |
| Exact search | `SearchParams.exact` | Supported | Supported | `EXACT` or `WITH { exact: true }` |
| HNSW ef tuning | `SearchParams.hnsw_ef` | Supported | Supported | `WITH { hnsw_ef: N }` |
| ACORN filtered search | `SearchParams.acorn` | Supported | Supported | Depends on Qdrant support |
| Search with filters | `Filter` | Supported | Supported | `WHERE` clause |
| Search pagination | `offset` | Missing | Missing | Near-term roadmap candidate |
| Batch search | Batch/query APIs | Missing | Missing | Later idea |
| MMR diversity | Query diversity controls | Missing | Missing | Later idea |
| Score boosting | Formula/rescore APIs | Missing | Missing | Later idea |
| Multivector search | Multivector query | Missing | Missing | Later idea |
| Rerank | Cross-encoder / inference | Supported | Partial | Python uses local Fastembed cross-encoder; Go behavior should be checked against `qql-go` docs |
| Relevance feedback | Feedback query | Missing | Missing | Later idea |

## Recommend

| Feature | Qdrant API | Python `qql-cli` | Go `qql-go` | Notes |
|---|---|---|---|---|
| Recommend by examples | Recommend query | Supported | Supported | `RECOMMEND FROM` |
| Positive/negative IDs | Recommend input | Supported | Supported | |
| Strategy selection | `RecommendStrategy` | Supported | Supported | `average_vector`, `best_score`, `sum_scores` |
| Cross-collection lookup | `lookup_from` | Supported | Supported | |
| Named vector usage | `using` | Supported | Supported | |
| Offset | `offset` | Supported | Supported | |
| Score threshold | `score_threshold` | Supported | Supported | |
| Filtered recommend | `Filter` | Supported | Supported | `WHERE` clause |

## Payload Indexes

| Feature | Qdrant API | Python `qql-cli` | Go `qql-go` | Notes |
|---|---|---|---|---|
| Keyword index | `create_payload_index` | Supported | Supported | |
| Integer index | `create_payload_index` | Supported | Supported | Python syntax uses `TYPE integer` |
| Float index | `create_payload_index` | Supported | Supported | |
| Bool index | `create_payload_index` | Supported | Supported | |
| Text index | `create_payload_index` | Supported | Partial | Go support should be verified |
| Geo index | `create_payload_index` | Supported | Missing | Python maps `TYPE geo` |
| Datetime index | `create_payload_index` | Supported | Missing | Python maps `TYPE datetime` |

## Filtering

| Feature | Qdrant model | Python `qql-cli` | Go `qql-go` | Notes |
|---|---|---|---|---|
| Equality | `MatchValue` | Supported | Supported | `=` |
| Inequality | `must_not` + `MatchValue` | Supported | Supported | `!=` |
| Range | `Range` | Supported | Supported | `>`, `<`, `>=`, `<=` |
| Between | `Range` | Supported | Supported | Inclusive |
| In list | `MatchAny` | Supported | Supported | `IN (...)` |
| Not in list | `MatchExcept` | Supported | Supported | `NOT IN (...)` |
| Is null | `IsNull` | Supported | Supported | |
| Is empty | `IsEmpty` | Supported | Supported | |
| Full-text match | `MatchText` | Supported | Supported | `MATCH` |
| Match any term | `MatchTextAny` | Supported | Supported | `MATCH ANY` |
| Match phrase | `MatchPhrase` | Supported | Supported | `MATCH PHRASE` |
| Logical operators | `must`, `should`, `must_not` | Supported | Supported | `AND`, `OR`, `NOT` |
| Nested fields | Payload key paths | Supported | Supported | Dot notation |
| Nested array access | Payload key paths | Partial | Partial | Keep examples conservative until integration-tested |

## Version Notes

| Implementation | Current version in this repo/docs | Notes |
|---|---|---|
| Python `qql-cli` | `1.4.0` | Source of truth for this repository |
| Go `qql-go` | `0.1.x` | Companion implementation; verify exact behavior in the Go repo before release claims |

## Known Gaps

| Gap | Impact | Suggested next step |
|---|---|---|
| No `GET` statement | Hard to inspect one point from the CLI | Add RFC or issue |
| No `SCROLL` statement | Hard to page/export large collections through QQL syntax | Add RFC or issue |
| No `COUNT` statement | Hard to validate scripts and filters | Add RFC or issue |
| No `DESCRIBE COLLECTION` | Users must drop to SDK/Qdrant UI for collection metadata | Add RFC or issue |
| No payload update syntax | Metadata updates require SDK calls or full reinsert | Add RFC or issue |
| Limited custom collection configuration | Advanced users need SDK for distance/HNSW/quantization | Define minimal syntax before implementing |

## Maintenance Rule

When changing QQL behavior:

1. Update this matrix in the same PR.
2. Link or mention tests that prove the status.
3. Mark companion implementation status as `Unknown` rather than guessing.
4. Avoid future-tense claims unless there is an accepted RFC or linked issue.
Loading
Loading