calculations and aggregates

[matt-beanland]

now ash_neo4j supports calculations and aggregates in the datalayer lets try them out. Idea is to see what we can and can't do, and either improve nbn and/or access, and/or raise bug reports/feature requests on diffo/ash_neo4j.

diffo should refactor the assigner and could extend provider dsl for live fields within characteristics.

But in the interim we can create new fields on the extended instance, etc, and use these when the characteristics are read by through the extended instance etc. Would also mean we need custom Jason for serialisation.

[matt-beanland]

Lets build in nbn domain:
resource, aggregate, type, description
Cvc, avc_count, integer, count of how many Avc are assigned to the Cvc
Cvc, downstream_subscription_ratio, ratio of sum of Avc downstream bandwidth over Cvc bandwidth

NniGroup, nni_count, integer, count of how many Nni are assigned to the NniGroup
NniGroup, cvc_count, integer, count of how many Cvcs are assigned to the Cvc
NniGroup, avc_count, integer, count of how many Avcs are assigned to the NniGroup indirectly via Cvc
NniGroup, utilisation, ratio of sum of Cvc bandwidth over NniGroup bandwidth

[matt-beanland]

Ash aggregates are now defined and tested across the NBN domain resources, taking advantage of ash_neo4j's aggregate support including over embedded typed structs.

CVC: characteristic_count, svlan_assigned? (via Relationship characteristic), cvc_char_value (deserialized CvcValue struct), avc_count
AVC: characteristic_count, cvlan_assigned? (via Relationship characteristic), avc_char_value (deserialized AvcValue struct), cvc_name (lifted via Relationship source), svlan (four-hop lift: AVC → Relationship → CVC → Relationship → Characteristic)
NniGroup: nni_group_char_value (deserialized NniGroupValue struct), nni_count
NbnEthernet: installer_name (lifted from Party via PartyRef using party_refs.role filter on destination)

Key discovery: source and target on Relationship nodes are the keys to multi-hop traversal through the graph. The four-hop svlan aggregate on AVC is the deepest path proven working.
Deferred

RSP instance_count and has_instances — pending ash_neo4j #219 (indirect relationships) and a new diffo DSL for party inheritance via relationship path
svlan_assigned? and cvlan_assigned? as exists — pending diffo #125 (Ash.Type.Dynamic path traversal) or Ash core join_filter support

Upstream tickets raised

ash_neo4j #251 — ? in aggregate names generates invalid Cypher aliases (fixed on dev)
ash_neo4j #252 — aggregate filters silently dropped (fixed on dev)
ash_neo4j #253 — (related fix, dev)
diffo #125 — Diffo.Type.Dynamic.value opaque to Ash ref resolution; path to Ash.Type.Dynamic PR
diffo (new) — party inheritance via relationship path in instances DSL
Ash (to discuss with Zach) — join_filter for intermediate node filtering in multi-hop aggregates

[matt-beanland]

Yarning reframed this into a concrete use case — turtles all the way up.

Reify with shelf bringing up its cards:

Shelf has a :slots pool, each slot assigns to a Card. Today the shelf's characteristic just carries shelf-identity (device_name, family, model, technology). Reified, it would also bring up its cards:

"shelf": {
  "device_name": "QDONC-1001",
  "family": "ISAM",
  "cards": [
    {"slot": 1, "card": {...card 1's brought-up view...}},
    {"slot": 2, "card": {...}}
  ],
  "total_ports": 96
}

Two ash_neo4j primitives in one use case:

• Calculation — cards is a calc that traverses the :slots AssignmentRelationships in slot order (using Diffo.Provider.Calculations.FieldFromAssignment or a custom calc that returns the assigned card records sorted by slot).
• Aggregate — total_ports sums (last - first + 1) across each assigned card's :ports pool, an ash_neo4j aggregate across the assignment graph.

This is the "bring up" pattern from #10 at scale. #10 is turtles all the way down — each slot expects what's below. This is turtles all the way up — each parent composes what's below into its own view. Same primitives, mirrored direction. The shelf brings up its cards; the cards bring up their shelf (per #10); both views derive from the same graph without snapshot/refresh.

For diffo: the "diffo should refactor the assigner and could extend provider DSL for live fields within characteristics" line in the original body is now upstream as diffo#170 (Define/Relate/Assign change modules) and #169 (typed-char surfacing in JSON). What remains here is us using ash_neo4j calculations + aggregates against the assignment graph — and reporting what works, what doesn't, and whether a Diffo DSL extension for live characteristic fields would make this cleaner than the current :value calculation pattern.

Narrowed scope: explore ash_neo4j calculations and aggregates by reifying shelf.cards and shelf.total_ports. Capture what works and what surfaces as a yarn for diffo / ash_neo4j.

[matt-beanland]

Landed on 10-shelf-on-path alongside #10. Turtles-up working end-to-end.

What landed:

• DiffoExample.Calculations.ReverseInheritedCharacteristic — dual of InheritedCharacteristic. Traverses OUTGOING AssignmentRelationship records (this instance as source) optionally filtered by alias, then reads the typed characteristic on each assignee. "Insanity is hereditary — you get it from your kids."
• Shelf has :cards calc — brings up every card's typed characteristic value, sorted by slot number (via the value field on the assignment record).
• Shelf has :total_ports calc — sums (last - first + 1) across each card's :ports AssignableCharacteristic. Multi-hop aggregate via a small custom calc (DiffoExample.Access.Calculations.ShelfTotalPorts).

Test in test/access/shelf_test.exs "shelf brings up its cards and total_ports":

{:ok, shelf_with_brought_up} = Ash.load(shelf, [:cards, :total_ports])

assert [
         %{family: :ISAM, model: "EBLT48", technology: :adsl2Plus},
         %{family: :ISAM, model: "EBLT24", technology: :adsl2Plus}
       ] = shelf_with_brought_up.cards

assert shelf_with_brought_up.total_ports == 72   # 48 + 24

Findings about ash_neo4j aggregates and calculations (the original ask of this issue):

• Plain calculations work cleanly. Ash.Resource.Calculation modules can do their own Neo4j queries (via Ash.Query against AshNeo4j-backed resources) and compose with Ash.load. We use this pattern for InheritedCharacteristic, ReverseInheritedCharacteristic, and ShelfTotalPorts.
• Aggregates over multi-hop assignment traversals aren't directly expressible as Ash aggregates today — they need a custom calc. total_ports is structurally an aggregate (sum across the cards of card.ports.last - card.ports.first + 1), but the path from shelf to ports passes through AssignmentRelationship and AssignableCharacteristic, which isn't a direct Ash has_many relationship. A future ash_neo4j primitive that aggregates over assignment-graph traversals natively would be a clean upgrade.

Upstreamed alongside #10 as diffo#172 — proposes both inherited_characteristic and reverse_inherited_characteristic DSL declarations.

The ShelfTotalPorts calc stays domain-specific for now; a more general "sum pool capacity of assignees" primitive could later live in diffo or ash_neo4j once the pattern proves to repeat.

[matt-beanland]

Resource shape landed on the 10-shelf-on-path branch alongside #10. Shelf brings up its assigned cards via DiffoExample.Calculations.ReverseInheritedCharacteristic (the dual — insanity is hereditary, you get it from your kids) and aggregates port capacity via ShelfTotalPorts. Tested end-to-end (cards in slot order, total_ports = 72 across two cards). JSON rendering and DSL macros upstreamed to diffo#172 and diffo#173. Findings on ash_neo4j aggregates: plain calcs work cleanly; multi-hop aggregates over assignment-graph traversals need a custom calc today, worth its own ash_neo4j primitive eventually.