Skip to content

Phase 1: Session-Scoped Graph Metadata, Typed Snapshots, and Edge Ownership Refactor#227

Open
griffinmilsap wants to merge 7 commits intodevfrom
feature/session-metadata
Open

Phase 1: Session-Scoped Graph Metadata, Typed Snapshots, and Edge Ownership Refactor#227
griffinmilsap wants to merge 7 commits intodevfrom
feature/session-metadata

Conversation

@griffinmilsap
Copy link
Collaborator

Summary

This PR introduces phase-1 support for high-level graph metadata in GraphServer while preserving existing low-level graph APIs.

Core outcomes:

  • session-scoped metadata lifecycle,
  • typed graph snapshots,
  • DAG-native edge ownership,
  • and metadata capture for units/collections/streams/tasks with dynamic settings opt-in.

Why

We need GraphServer to become a useful source of truth for both topology and high-level graph structure:

  • better graph rendering and profiling,
  • session-bound cleanup of high-level metadata,
  • future state/settings querying workflows.

What Changed

1) GraphContext Session Control Plane

  • Replaced ad-hoc session read locking with a dedicated session IO task and command queue.
  • GraphContext.connect/disconnect now require an active session context.
  • register_metadata and snapshot operate through session commands.

2) Session Metadata + Snapshot Typing

  • Added structured metadata dataclasses in src/ezmsg/core/graphmeta.py:
    • GraphMetadata
    • ComponentMetadata, CollectionMetadata, UnitMetadata
    • StreamMetadata, InputStreamMetadata, OutputStreamMetadata
    • TaskMetadata, DynamicSettingsMetadata
  • Added typed snapshot dataclasses:
    • Edge
    • SnapshotSession
    • GraphSnapshot
  • GraphContext.snapshot() now returns GraphSnapshot.

3) Session Transport

  • Session metadata and snapshots are serialized via pickle.
  • Metadata registration validates payload type on the server (GraphMetadata).

4) Initial Settings Payload

  • initial_settings now includes:
    • pickled settings bytes (or None when pickling fails),
    • and a readable representation.
  • Readable representation:
    • dataclasses.asdict(...) for dataclass settings (recursive),
    • otherwise repr(...).

5) Edge Ownership Model

  • Moved edge ownership into DAG (edge_owners) to avoid mirrored ownership state in GraphServer.
  • Added owner-aware add/remove semantics with topology-change return values.
  • None owner represents persistent edges.

6) Session-as-Client Unification

  • SessionInfo moved to netprotocol and handled as a standard graph client type.
  • Session disconnect now reliably drops:
    • session-owned edges,
    • session metadata.

7) Backend Metadata Collection

  • Extracted reusable crawl_components(...).
  • Built metadata as typed dataclasses.
  • Dynamic settings detection remains opt-in via INPUT_SETTINGS inlet.

Behavior Notes

  • Low-level GraphService.connect/disconnect remains available for persistent graph edges.
  • Session-scoped metadata is automatically removed when a session disconnects.

Caveats

Pickle-based metadata/snapshot transport assumes compatible class definitions between communicating environments. GraphServer maintains pickled byte representation of initial settings dataclass, but does not attempt to unpickle because of this. Text/Dict-based representations of initial settings are present to allow for visualization/introspection even if server does not have compatible class definitions in its virtual environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant