Skip to content

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525

Open
indietyp wants to merge 3 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres
Open

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525
indietyp wants to merge 3 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres

Conversation

@indietyp
Copy link
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Prepares the MIR execution pipeline for consumption by the postgres compiler. The execution analysis now produces a complete IslandGraph (not just a flat island list), the placement solver and island placement accept external allocators, and the traversal system gains the TraversalPathBitMap and as_symbol() APIs that the SQL generator needs. Also adds a backend switch cost to the terminator placement so cross-backend transitions are no longer free.

🔍 What does this change?

Execution analysis (pass/execution/mod.rs):

  • run() becomes run_in() with an explicit allocator parameter, returning ExecutionAnalysisResidual (assignment + island graph). The island graph is now constructed as part of the analysis rather than left to the caller.
  • Adds run_all_in() which runs the execution analysis over all graph-read bodies in a DefIdSlice.

Traversal system (traversal/mod.rs, traversal/entity.rs):

  • TraversalPathBitMap: per-vertex-type collection of TraversalPathBitSets with pointwise lattice operations. The postgres compiler uses this to track which paths each island accesses across all vertex types.
  • TraversalPath::as_symbol(): returns a static symbol for each path variant, used as SQL column aliases so the interpreter can locate result columns by name.
  • TraversalPathBitSet::vertex(): returns the vertex type for a bitset.
  • EntityPath::as_symbol() and EntityPath::column_name(): per-path SQL identifiers.

Terminator placement (terminator_placement/mod.rs):

  • TransMatrix gains AddAssign for element-wise saturating addition.
  • Adds backend_switch_cost() which encodes a fixed overhead for cross-backend transitions (Postgres to Interpreter: 8, Interpreter to Embedding: 4, etc.). Previously cross-backend transitions had zero inherent cost, so empty blocks were arbitrarily assigned to the interpreter even when staying on postgres was free.

Placement solver (placement/solve/):

  • PlacementSolver::run() becomes run_in() with allocator parameter.
  • CSP solver internal allocations use the provided allocator.

Island graph (island/graph/mod.rs):

  • IslandGraph::new_in() now takes an allocator for its output storage.

Pretty printer (pretty/text.rs):

  • TextFormatAnnotations gains annotate_basic_block() and BasicBlockAnnotation associated type, plus a blanket impl for &mut T.

Builder (builder/rvalue.rs):

  • Adds RValueBuilder::opaque_entity_uuid() convenience constructor for the common pattern of extracting an entity's UUID.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The backend_switch_cost() values (8, 4, etc.) are hand-tuned heuristics. They correctly prevent the solver from arbitrarily switching backends for empty blocks, but a proper cost model would derive these from measured overhead.

🛡 What tests cover this?

  • Updated execution pass end-to-end tests (execution/tests.rs) including entity_uuid_equality, mixed_postgres_embedding_interpreter, projection_and_apply_splits
  • Updated statement placement tests for interpret and postgres backends (eq_opaque_entity_uuid)
  • Updated terminator placement and island tests
  • Placement solver tests updated for allocator API

❓ How to test this?

cargo nextest run -p hashql-mir

feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@cursor
Copy link

cursor bot commented Mar 8, 2026

PR Summary

Medium Risk
Touches core MIR execution-placement logic (solver APIs, island graph construction, and terminator transition costs), which can change target assignment behavior and query compilation outcomes across backends.

Overview
Updates the MIR execution pipeline to be allocator-parameterized and to return richer outputs: ExecutionAnalysis::run_in now yields an ExecutionAnalysisResidual containing both per-block target assignment and a fully-built IslandGraph, plus a new run_all_in helper to analyze all graph-read bodies.

Extends execution planning data structures for Postgres compilation: islands/graphs gain new query helpers (members/contains/find), traversal paths gain as_symbol() and a new per-vertex TraversalPathBitMap lattice, and the MIR builder/body! macro add support for opaque aggregates (including ~ unary in docs).

Changes placement behavior by introducing a fixed backend switch overhead into terminator transition costs (via TransMatrix +=), and refactors the placement solver and island placement APIs to accept external allocators; tests/snapshots are updated and expanded (notably EntityUuid opaque equality) to reflect the new planning outputs and cost model.

Written by Cursor Bugbot for commit 8d113b2. This will update automatically on new commits. Configure here.

@vercel
Copy link

vercel bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment Mar 20, 2026 8:19am
petrinaut Ready Ready Preview Mar 20, 2026 8:19am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview Mar 20, 2026 8:19am
hashdotdesign-tokens Ignored Ignored Preview Mar 20, 2026 8:19am

@augmentcode
Copy link

augmentcode bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: Prepares the HashQL MIR execution pipeline for Postgres compilation by returning a full island dependency graph and exposing traversal/placement APIs needed by the SQL generator.
Key Changes:

  • Execution analysis now takes an explicit allocator (run_in) and returns ExecutionAnalysisResidual (assignment + IslandGraph); adds run_all_in for batch analysis.
  • Traversal adds TraversalPathBitMap, TraversalPath::as_symbol(), and EntityPath::as_symbol()/column_name() for stable SQL column aliases.
  • Placement solver and island placement APIs accept external allocators and route CSP internal allocations through the provided bump allocator.
  • Terminator placement introduces a fixed backend_switch_cost() so cross-backend transitions have non-zero overhead (avoids “free” backend flips).
  • Island graph gains helper queries (members, contains, find) and is now built/resolved within execution analysis.
  • Pretty printer supports per-basic-block annotations; MIR builder gains an opaque aggregate constructor and new tests cover opaque UUID equality.
Tests: Updates/extends execution, statement placement, terminator placement, and allocator-related solver snapshots (incl. entity_uuid_equality). Notes: Backend switch costs are heuristics intended to be refined by a measured cost model.

🤖 Was this summary useful? React with 👍 or 👎

Copy link

@augmentcode augmentcode bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@codecov
Copy link

codecov bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 64.24581% with 128 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.98%. Comparing base (3ba579f) to head (8d113b2).

Files with missing lines Patch % Lines
...cal/hashql/mir/src/pass/execution/traversal/mod.rs 0.00% 44 Missing ⚠️
libs/@local/hashql/mir/src/pretty/text.rs 21.62% 28 Missing and 1 partial ⚠️
.../hashql/mir/src/pass/execution/traversal/entity.rs 0.00% 28 Missing ⚠️
libs/@local/hashql/mir/src/pass/execution/mod.rs 44.00% 14 Missing ⚠️
.../hashql/mir/src/pass/execution/island/graph/mod.rs 35.29% 11 Missing ⚠️
.../mir/src/pass/execution/placement/solve/csp/mod.rs 90.00% 0 Missing and 1 partial ⚠️
libs/@local/hashql/mir/src/pass/execution/tests.rs 99.07% 1 Missing ⚠️
Additional details and impacted files
@@                                           Coverage Diff                                            @@
##           bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query    #8525       +/-   ##
========================================================================================================
+ Coverage                                                                63.39%   83.98%   +20.58%     
========================================================================================================
  Files                                                                      763      326      -437     
  Lines                                                                    89217    51060    -38157     
  Branches                                                                  2916     1299     -1617     
========================================================================================================
- Hits                                                                     56563    42882    -13681     
+ Misses                                                                   32107     7763    -24344     
+ Partials                                                                   547      415      -132     
Flag Coverage Δ
apps.hash-api ?
local.hash-isomorphic-utils ?
rust.hash-graph-api ?
rust.hash-graph-postgres-store ?
rust.hashql-compiletest 29.69% <ø> (ø)
rust.hashql-eval ?
rust.hashql-hir ?
rust.hashql-mir 92.08% <64.24%> (-0.37%) ⬇️
rust.hashql-syntax-jexpr ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 56 skipped benchmarks1


Comparing bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres (8d113b2) with bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (3ba579f)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 6ef6d13 to ef9b858 Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5d95ba7 to 1aa0f1c Compare March 8, 2026 16:15
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 1aa0f1c to 87024df Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from a91293c to 43775ba Compare March 8, 2026 17:03
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 87024df to 269a31f Compare March 8, 2026 17:03
@vercel vercel bot temporarily deployed to Preview – petrinaut March 8, 2026 17:24 Inactive
@graphite-app graphite-app bot requested review from a team March 8, 2026 17:25
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from db87959 to 8d113b2 Compare March 20, 2026 08:11
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 43775ba to 3ba579f Compare March 20, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants