vector search over Notes — query instances by relevance to natural language

[matt-beanland]

Description

Diffo.Provider.Note already carries text — free-form annotation that parties write against instances. The next step is making that text searchable by meaning, not just by exact match or substring. Given a natural-language query, return the instances whose notes are most relevant, ranked by similarity, with the matching note excerpts surfaced as data.

Two layers:

Note-level vector search — embed each note's text on create/update; vector index the embeddings; a read action that takes a query text, embeds it, and returns notes ranked by similarity.

Instance-level relevance — aggregate note relevance up to the annotated instance. An instance's relevance score is some combination (max? sum? weighted) of the relevance of its notes (and possibly notes on instances related to it). Query instances by their relevance to a natural-language question.

Why it matters

Free-text notes are the most under-leveraged surface in any operational system. They're written casually by parties (engineers, support agents, installers, customers) and contain the lived experience of the domain — the things the schema doesn't say. Today they're searchable by contains/2 if you're lucky.

With vector search, the choreography gets a new alignment dimension: text-relevance. A customer reporting "intermittent dropouts after rain" can land on instances whose notes describe similar patterns, even when the wording is different. An engineer designing a new service can find prior services with similar constraints expressed naturally. A repairer can find prior repairs that resemble the symptom.

This also makes the system honest about what it knows — the rich free-text knowledge that parties carry becomes queryable as a first-class part of the graph, not a separate text-search silo. And it fits the choreography model: each instance carries its own annotations, search emerges from text relevance across the graph, no central index outside the framework.

What we'd find useful

# note-level
{:ok, notes} =
  Diffo.Provider.Note
  |> Ash.Query.for_read(:find_similar, %{query: "intermittent packet loss after rain", limit: 20})
  |> Ash.read()

# instance-level — instances ranked by note relevance
{:ok, instances} =
  DslAccess
  |> Ash.Query.for_read(:find_by_note_relevance, %{
    query: "intermittent packet loss after rain",
    limit: 10
  })
  |> Ash.read()

# combined with other filters
{:ok, instances} =
  DslAccess
  |> Ash.Query.filter(places_contains_geo: ^customer_lat_lng_within(:km, 5))
  |> Ash.Query.for_read(:find_by_note_relevance, %{query: "..."})
  |> Ash.read()

The relevance scores should be available alongside the records — collapsible to a ranked list of IDs (per #42), or kept rich (instances + scores + matching note excerpts) when the consumer wants the why.

A possible direction

Several pieces, each likely a separate yarn upstream:

1. Embedding-on-write hook on Diffo.Provider.Note — an embedding attribute populated by a configurable embed function on :create and :update. Local model (bumblebee/ollama) or hosted (OpenAI/Cohere) — consumer's choice via config.
2. Vector index in AshNeo4j — Neo4j 5+ has native vector indexes via db.index.vector.queryNodes. AshNeo4j needs to expose this either as an index declaration on the resource or as a query primitive that produces a similarity-ordered result.
3. Instance-level aggregate — a calculation or aggregate on Instance that scores it against a query by aggregating the relevance of its (and related?) notes. Probably a new Diffo extension or just an Ash calculation pattern.
4. A query primitive — find_by_note_relevance(query, opts) action shape that takes the natural-language query and returns ranked records. Could be a Diffo-provided action that any BaseInstance resource gets.

The exemplar work would be to wire one embedding model end-to-end on Note, prove the Neo4j vector index round-trips through AshNeo4j, and prototype an instance-relevance ranking. The deltas become yarns for AshNeo4j (vector index + query expressions) and Diffo (the embedding hook and the instance-aggregate pattern).

Related:

• #3, #41, #42 — the answer-shape family handles "found instances + scores" naturally; collapse to ranked-IDs for the simple case.
• #38, #39 — feasibility + note-relevance compose: "which servable shelves have notes suggesting they handle weather conditions like the customer's reported issue."
• #40 — note-relevance is a metric of value (a quality score, a fit score) that the preference machinery can weigh.

[matt-beanland]

WWZD reframing — note-relevance as a calculation that returns a score, not a special action. Compose with every other Ash filter/sort using standard machinery:

DslAccess
|> Ash.Query.filter(note_relevance(query: "intermittent dropouts after rain") > 0.5)
|> Ash.Query.filter(places_contains_geo: ^customer_lat_lng_within(:km, 5))
|> Ash.Query.sort(note_relevance: :desc)
|> Ash.read()

The vector search lives inside the note_relevance calculation — AshNeo4j translates the expression to Neo4j's db.index.vector.queryNodes when it can. The calculation takes the query text as an argument; the score it returns is just a number, filterable and sortable like any other.

This makes semantic search a value metric (per #40) — relevance is a dimension that the preference/worldview machinery can weigh alongside cost, distance, capacity, fairness-ledger. Customer-centric worldview weights relevance high when the customer asks "find something like this"; network-balanced worldview ignores it; the same primitive serves both.

It also stays honest about cardinality. note_relevance returns a number for every record in the candidate set. If the consumer wants only the top-N, they sort and limit. If they want a Pareto frontier across relevance + cost + distance, they compose multiple metric calculations. No special "search action" — just calculations as expressions over the graph.

That's the shape change for this issue: from "a query primitive" to "a calculation that's a value metric." Same primitive that #10/#32 use for live bring-up, just sourced from vector similarity rather than graph traversal.

[matt-beanland]

Now we are using AshAi for map #44 we should first explore its capabilities. I think it supports vector but is somehow opinionated about AshPostgres.

[matt-beanland]

Quick discovery while merging #44:

ash_ai's vectorize do ... end macro and embed-on-write machinery exists and is well-shaped — but the transformer hard-couples to AshPostgres / pgvector at compile time:

# deps/ash_ai/lib/ash_ai/transformers/vectorize.ex:30
if Ash.Resource.Info.data_layer(dsl) != AshPostgres.DataLayer do
  raise "AshAi vectorization only currently supports AshPostgres"
end

So the declarative shape we want is there, but our target (Diffo.Provider.Note on AshNeo4j.DataLayer) raises at compile.

Three observations:

• The shape is right. Declarative vectorize do + named embedding model + strategy (sync/oban) + similarity calculations is exactly what this issue wants. No design rethink needed.
• Neo4j has the substrate. Native vector indexes (CREATE VECTOR INDEX, db.index.vector.queryNodes) ship in Neo4j 5+. The underlying capability exists.
• The bridge is the contribution. ash_ai needs to dispatch to the data layer for vector storage/queries instead of assuming pgvector, OR AshNeo4j needs to integrate following the pattern AshPostgres uses.

Path forward: fork ash_ai and contribute AshNeo4j data-layer support upstream. Land it in ash_ai itself rather than working around it locally. Once merged, this issue becomes a thin wrapper — add vectorize do full_text do text fn note -> note.text end end embedding_model ... end to Diffo.Provider.Note and the semantic-relevance calculation pattern from the WWZD reframing falls out.

Order of work:

1. Fork ash_ai, add AshNeo4j.DataLayer branch alongside AshPostgres in transformers/vectorize.ex and wherever else pgvector specifics leak (storage layout, query expressions, index creation).
2. Confirm round-trip on a small test resource using AshNeo4j + Neo4j vector index.
3. Open PR to ash_ai upstream.
4. Once merged (or local-pin pending merge), wire vectorize on Diffo.Provider.Note — that may need a yarn with diffo since Note is in the closed Provider domain, or we wrap Note's text indexing in a related resource on our side.
5. Add the note_relevance(query: ...) calculation on Instance (per the WWZD reframing earlier in this thread).

Marking this as blocked on the upstream contribution rather than chasing a workaround — the snapshot-mirror or raw-Cypher options both compromise the exemplar lesson.

[matt-beanland]

Direction resolved after the Josh-input + path comparison: semantic search on Notes lives in diffo as an action.

Shape:

# Diffo.Provider.Note (or paired resource)
read :search_by_relevance do
  argument :query, :string, allow_nil?: false
  argument :limit, :integer, default: 20
  # uses ash_neo4j vector primitives directly (blocked on diffo-dev/ash_neo4j#74)
  prepare ...  # embed query → vector index lookup → sort by similarity
end

# the consuming domain
tool :search_notes_by_relevance, Diffo.Provider.Note, :search_by_relevance

Why this shape:

• Action is the bridge to MCP. ash_ai's tool exposure works for any Ash action; we don't need ash_ai's vectorize do macro at all. The action's arguments generate the JSON Schema; the same authorisation gates apply; one tool entry surfaces it to MCP clients.
• Stays inside the diffo discipline. Notes are a diffo primitive; semantic search over them is a diffo concern. Lives close to the rest of the bring-up / value-metric / choreography machinery.
• Ships with diffo for all consumers. Anyone using diffo gets semantic search out of the box once this lands; non-diffo ash_neo4j projects aren't blocked on it.
• Doesn't fight the ash_ai data-layer-independence conversation. That's a parallel architectural conversation Josh has invited; we don't pre-build for it or assume its shape. If ash_ai grows data-layer independence later, we can revisit whether the diffo action layer collapses into ash_ai's vectorize — but until then, action-as-tool is enough.

Blockers reduce to one: diffo-dev/ash_neo4j#74 (vector index support in the data layer). The ash_ai blocker we previously named is no longer in scope — we don't go through ash_ai for this; we go directly through ash_neo4j's vector primitives.

Where the diffo action lives: probably on Diffo.Provider.Note itself rather than a wrapping resource in a consumer domain. Notes already belong to diffo's closed Provider domain; semantic search over them is naturally diffo's responsibility too. That conversation belongs in a diffo ticket once the AshNeo4j primitives land.

The WWZD reframing (relevance as a calculation composable with other Ash filters) still holds — once the basic action ships, we can layer on a note_relevance calculation for the composable case. The action is the simple-question shape; the calculation is the composable shape; same machinery underneath.

[matt-beanland]

Sharpening — no DSL required. Just vanilla Ash:

1. Embed-on-write change — added to Diffo.Provider.Note's :create and :update actions:

   change Diffo.Provider.Changes.EmbedNoteText  # generates embedding, stores in attribute

2. Search-by-relevance read action — on Diffo.Provider.Note:

   read :search_by_relevance do
     argument :query, :string, allow_nil?: false
     argument :limit, :integer, default: 20
     prepare Diffo.Provider.Preparations.VectorSearchNotes
   end

Both are vanilla Ash. No vectorize do macro to add, no DSL extension on the diffo side, no compile-time transformer. The same kit every other diffo action uses. The embedding model is configurable (env or runtime config) — could default to a local model (bumblebee) or be set to a hosted one.

This keeps the surface area tiny:

• Two changes/preparations (or one of each — EmbedNoteText and VectorSearchNotes)
• One new attribute on Note (:embedding, vector type once diffo-dev/ash_neo4j#74 lands)
• One new action

Consuming domains expose the action via their existing tools do block — tool :search_notes_by_relevance, Diffo.Provider.Note, :search_by_relevance — and the MCP surface picks it up automatically. No new DSL anywhere in the stack.