feat: add sync-likes, sync-timeline, and sync-feed commands#43
feat: add sync-likes, sync-timeline, and sync-feed commands#43mmenestret wants to merge 3 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5984733. Configure here.
|
|
||
| const src = options?.source; | ||
| const sourceFilter = src ? 'WHERE source = ?' : ''; | ||
| const sourceAnd = src ? 'AND source = ?' : ''; |
There was a problem hiding this comment.
Missing ensureMigrations call in getStats
Medium Severity
getStats now queries the source column (added in schema v5) when options.source is provided, but unlike searchBookmarks, listBookmarks, and other DB-reading functions, it never calls ensureMigrations(db). On a pre-v5 database, running ft stats --source likes will crash because the source column doesn't exist yet.
Reviewed by Cursor Bugbot for commit 5984733. Configure here.
There was a problem hiding this comment.
Fixed in 64163a2 — added ensureMigrations(db) call in getStats.
| console.log(`\n \u2713 ${result.added} new ${cfg.label} synced (${result.totalBookmarks} total)`); | ||
| console.log(` ${friendlyStopReason(result.stopReason)}`); | ||
| console.log(` \u2713 Data: ${dataDir()}\n`); | ||
| })); |
There was a problem hiding this comment.
New sync commands don't rebuild search index
Medium Severity
The sync-likes, sync-timeline, and sync-feed commands only write to JSONL cache files but never call buildIndex afterward. Unlike ft sync (which calls rebuildIndex after syncing), the new commands leave data unsearchable. The README Quick Start shows ft sync-likes then ft search as a workflow, but this won't return results without a manual ft index step.
Reviewed by Cursor Bugbot for commit 5984733. Configure here.
There was a problem hiding this comment.
Fixed in 64163a2 — added await rebuildIndex(result.added) after sync in registerUserSyncCommand, matching existing ft sync behavior.
| for (const record of newRecords) { | ||
| insertRecord(db, record); | ||
| for (const { record, source } of newEntries) { | ||
| insertRecord(db, record, source); |
There was a problem hiding this comment.
Cross-source duplicates get wrong source tag in index
Low Severity
buildIndex collects records from all four JSONL files without deduplicating by record.id. When the same tweet exists in multiple sources (e.g., bookmarked AND liked), both entries pass the newEntries filter and INSERT OR REPLACE causes the last source in the array to win. A bookmarked tweet that was also liked would be tagged as likes, making it invisible to --source bookmarks queries.
Reviewed by Cursor Bugbot for commit 5984733. Configure here.
There was a problem hiding this comment.
Not a bug. buildIndex uses an existingIds set (line 346-354) to skip already-indexed records — there is no INSERT OR REPLACE on duplicates. The first source encountered wins, and subsequent sources are filtered out by !existingIds.has(record.id). The source order in the sources array is deterministic (bookmarks first), so bookmarks always take priority.
|
You have used all of your free Bugbot PR reviews. To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial. |
|
Was going to vibe up the same thing; thanks for making this @mmenestret !! |
Add three new sync commands that extend ft beyond bookmarks: - `ft sync-likes <user>`: sync liked tweets via GraphQL (Likes endpoint) - `ft sync-timeline <user>`: sync user's own tweets (UserTweets endpoint) - `ft sync-feed`: sync Following/chronological feed (HomeLatestTimeline endpoint) All synced data is stored in separate JSONL files (likes.jsonl, timeline.jsonl, feed.jsonl) and merged into the unified SQLite FTS5 index via `ft index`. A new `source` column (schema v5) enables filtering across all query commands: ft search "AI" --source likes ft list --source feed ft stats --source likes Implementation details: - New `src/graphql-user-sync.ts` module handles all three endpoints, reusing shared helpers (convertTweetToRecord, mergeRecords, buildHeaders, parseSnowflake, snowflakeToIso) from graphql-bookmarks.ts - Feed sync is session-scoped (no userId needed), likes/timeline resolve userId via UserByScreenName GraphQL query - CLI commands factored via registerUserSyncCommand to avoid boilerplate - --source validated against allowed values (bookmarks, likes, timeline, feed) - 19 new tests covering response parsing for both user timeline and feed response shapes, cursor extraction, conversation modules, ingestedVia assignment, and likedAt conversion
…c commands getStats now calls ensureMigrations(db) to avoid crashing on pre-v5 databases when --source is used. sync-likes, sync-timeline, and sync-feed now rebuild the search index after syncing, matching the existing sync command behavior.
mmenestret
force-pushed
the
feat/sync-likes-timeline-feed
branch
from
64163a2 to
39a2d2a
Compare
|
You have used all of your free Bugbot PR reviews. To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial. |
2 participants
Summary
ft sync-likes <user>,ft sync-timeline <user>, andft sync-feedcommands to sync liked tweets, user timeline, and Following feed via GraphQLsourcecolumn (schema v5), making it queryable viasearch,list, andstats--sourcefilter (bookmarks, likes, timeline, feed) tosearch,list, andstatscommandsDetails
New commands
ft sync-likes <user>Likesft sync-timeline <user>UserTweetsft sync-feedHomeLatestTimelineAll three commands support the same browser/cookie options as
ft sync(--browser,--cookies,--chrome-profile-directory, etc.).Architecture
src/graphql-user-sync.tsmodule reuses shared helpers fromgraphql-bookmarks.ts(convertTweetToRecord,mergeRecords,buildHeaders,parseSnowflake,snowflakeToIso)UserByScreenNameGraphQL queryregisterUserSyncCommandhelper to avoid boilerplate duplicationlikes.jsonl,timeline.jsonl,feed.jsonl)Index integration
buildIndex()reads all 4 JSONL files and tags each record with itssourcesource TEXT DEFAULT 'bookmarks'column (backward-compatible)--sourcefilter validated against allowed values with clear error messageKnown limitations
ft classifyonly classifies bookmarks (not likes/timeline/feed) — intentional scopelikedAtis stored in JSONL but not in the SQLite schema — could be added in a future iterationft categoriesandft domainsdon't support--sourceyetTest plan
parseUserTimelineResponse(user timeline + feed response shapes, cursors, conversation modules,ingestedVia,likedAt)--noEmit--sourcefiltering onsearch/list/stats--source typoproduces clear validation errorNote
Medium Risk
Adds new GraphQL sync surfaces and a DB schema migration (
schema_version5 with newsourcecolumn), which could impact indexing/query correctness and requires careful handling of incremental merges and migrations.Overview
Adds new CLI commands
ft sync-likes <user>,ft sync-timeline <user>, andft sync-feedthat sync additional X timelines via a shared GraphQL engine (graphql-user-sync.ts) and store them in separate JSONL caches.Updates indexing and query paths to merge all caches into the SQLite FTS index by introducing a
sourcefield (schema v5 + migration + index), and adds--sourcefiltering tosearch,list, andstats. Documentation is updated and new tests cover parsing/ingestion behavior for the new GraphQL response shapes (including cursor handling andlikedAt).Reviewed by Cursor Bugbot for commit 5984733. Bugbot is set up for automated code reviews on this repo. Configure here.