Skip to content

feat: load sessions from all Copilot CLI storage locations#82

Open
spboyer wants to merge 3 commits into
jayparikh:mainfrom
spboyer:feat/multi-source-sessions
Open

feat: load sessions from all Copilot CLI storage locations#82
spboyer wants to merge 3 commits into
jayparikh:mainfrom
spboyer:feat/multi-source-sessions

Conversation

@spboyer
Copy link
Copy Markdown
Contributor

@spboyer spboyer commented Apr 13, 2026

Summary

Enables AGENTVIZ to load sessions from all three Copilot CLI storage locations, not just local JSONL files.

New session sources:

  1. Session store enrichment -- reads ~/.copilot/session-store.db and ~/.copilot/data.db to enrich discovered sessions with: full summary, model name, reasoning effort, custom title (from /rename), repo, branch, turn count. Surfaces store-only sessions that have events.jsonl but weren't found by the file scanner.

  2. Shared session discovery -- finds copilot-session-*.md files in cwd (from /share file) and secret GitHub gists (from /share gist via gh gist list).

  3. Shared markdown parser -- new format parser that converts Copilot CLI /share markdown into normalized events. Supports drag-and-drop of .md files. Auto-detected by parseSession.ts.

Files changed (9)

  • routes/sessionStoreReader.js (new) -- reads session-store.db + data.db with graceful fallback
  • routes/sharedSessions.js (new) -- discovers shared markdown files and gists
  • routes/sessions.js -- integrates both new sources into /api/sessions, adds /api/session/shared endpoint
  • src/lib/sharedSessionParser.js (new) -- client-side markdown-to-events parser
  • src/lib/parseSession.ts -- detects shared-md format, routes to new parser
  • src/lib/sessionTypes.ts -- adds shared-md to SessionFormat type
  • 3 new test files (27 tests)

Tests

All 590 tests pass (567 existing + 23 new).

References

Closes #81

@jayparikh
Copy link
Copy Markdown
Owner

Thanks for pushing this — loading sessions from the additional Copilot CLI storage locations closes a real gap in AGENTVIZ, and the added functionality here is valuable. I pulled the branch locally and it is green (build, typecheck, test), but I still see one merge-blocking integration gap plus two follow-up fixes.

1. Shared sessions are discoverable but still not openable from the app (blocker)

The backend now exposes shared sessions via /api/sessions and adds /api/session/shared, but the client open flow is still wired for the old path -> /api/session?path=... contract.

What is broken

  • Shared local markdown files (source: "shared-file", format: "shared-md") currently fall back to /api/session?path=..., which rejects them because:
    • the path is outside the existing allowlist, and
    • /api/session only serves .json / .jsonl
  • Shared gists (source: "shared-gist") are never fetched at all, because the current open flow assumes discovered sessions always have a path

Please fix this end-to-end

  1. In the app open flow (src/App.jsx / src/hooks/useDiscoveredSessions.js), stop treating all discovered sessions as path-only. Pass the full discovered entry into fetchSessionContent(...) so it can branch on source / format.
  2. In fetchSessionContent(...), route like this:
    • source === "manifest" -> existing manifest behavior
    • source === "shared-file" -> GET /api/session/shared?path=...
    • source === "shared-gist" -> GET /api/session/shared?gist=...
    • everything else -> existing GET /api/session?path=...
  3. Normalize the shared-session response so afterLoad(...) receives the raw markdown text, not a JSON wrapper. Either:
    • make /api/session/shared return text/plain raw markdown like /api/session, or
    • keep the JSON response but have the client extract and return raw
  4. Add regression coverage proving that both:
    • a discovered shared-file entry can be opened from the landing UI, and
    • a discovered shared-gist entry can be opened from the landing UI

2. /api/session/shared?path=... needs the same path restrictions as the other session endpoint (warning)

Right now the new shared-file endpoint will read any .md file reachable on disk.

Please tighten this

  • Restrict local shared-file reads to the same trust boundary as discovery, i.e. the directory scanned by findSharedSessionFiles() (process.cwd())
  • Use realpathSync(...) + path.relative(...) and reject:
    • paths outside cwd
    • traversal attempts
    • non-.md files

Please add route tests for

  • allowed: shared markdown file under cwd
  • rejected: markdown file outside cwd

3. Avoid synchronous gh gist list work on every /api/sessions poll (warning)

findSharedSessionGists() currently does synchronous CLI/network work inside the /api/sessions request path. Since the client polls that endpoint repeatedly, one slow gh call can block the server.

Please change this so the request path is not doing blocking network I/O every time

Acceptable fixes would be:

  • cache gist results with a TTL and refresh them separately, or
  • move gist discovery to an async path that does not block the main request handler

What already looks good

  • the shared markdown parser itself is wired into parseSession.ts
  • drag-and-drop .md parsing looks fine
  • the current issue is the discovered-session integration path, not the parser core

Once the open flow is fixed, please also update the docs for the new session sources if this feature is staying in the PR.

@spboyer spboyer force-pushed the feat/multi-source-sessions branch from 60b07de to aff587f Compare May 8, 2026 00:36
@spboyer
Copy link
Copy Markdown
Contributor Author

spboyer commented May 9, 2026

Fixed in 4373648 -- all three points addressed.

1. Blocker (shared sessions openable): fetchSessionContent now branches on source: shared-file goes to /api/session/shared?path= and extracts .raw; shared-gist goes to /api/session/shared?gist= and extracts .raw; everything else uses existing /api/session?path=. App.jsx passes the full entry (not bare path) to fetchSessionContent, widens condition to include entry.gistId so gist entries (no path) are openable, and copies gistId into discoveredOnly.

2. Warning (path restriction): Exported isAllowedSharedPath() using isPathInsideRoot. Applied after realpathSync in the handler: files outside process.cwd() return 403. Four new tests: inside-cwd allows, outside-cwd rejects, sibling-prefix-of-cwd rejects, null-cwd returns false.

3. Warning (gist cache): Module-level _gistCache with 5-minute TTL in sharedSessions.js. findSharedSessionGists() returns cached results immediately on repeated /api/sessions polls; execFileSync only runs when cache is cold or stale.

786/786 tests pass.

@spboyer spboyer force-pushed the feat/multi-source-sessions branch from 4373648 to 34530d2 Compare May 15, 2026 22:54
@spboyer spboyer force-pushed the feat/multi-source-sessions branch from 34530d2 to 7b0c181 Compare May 25, 2026 12:01
spboyer and others added 3 commits May 26, 2026 08:02
Enriches session discovery with metadata from session-store.db and
data.db (model, reasoning effort, full summary, custom names).
Discovers shared sessions from /share file (local markdown) and
/share gist (GitHub gists). Adds shared-md format parser that
converts Copilot CLI shared session markdown into normalized events.

New files:
- routes/sessionStoreReader.js -- reads session-store.db + data.db
- routes/sharedSessions.js -- discovers shared markdown + gists
- src/lib/sharedSessionParser.js -- client-side markdown parser
- 3 test files with 27 new tests

Closes jayparikh#81

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion, gist cache

- useDiscoveredSessions: fetchSessionContent now branches on source for
  shared-file (GET /api/session/shared?path=) and shared-gist
  (GET /api/session/shared?gist=), extracting .raw from JSON response
- App.jsx: pass full entry (not bare path) to fetchSessionContent for
  discovered sessions; widen condition to include entry.gistId so
  shared-gist entries (no path) are openable; copy gistId into discoveredOnly
- routes/sessions.js: add isAllowedSharedPath() export and apply it to
  /api/session/shared?path= (rejects files outside process.cwd(), 403)
- routes/sharedSessions.js: cache findSharedSessionGists() results with
  5-minute TTL to avoid blocking the event loop on every /api/sessions poll
- tests: 4 new isAllowedSharedPath cases (inside cwd, outside cwd,
  sibling-prefix rejection, null cwd)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@spboyer spboyer force-pushed the feat/multi-source-sessions branch from 7b0c181 to cbbd1fa Compare May 26, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: load sessions from all Copilot CLI storage locations

2 participants