Skip to content

ft fetch-media progress output omits pending scope (bookmark + file counts already in the progress callback) #154

@RealADemin

Description

@RealADemin

ft fetch-media's current progress output tells the user what's happened (processed, downloaded) but not what's in scope — neither how many bookmarks are pending nor how many files will be fetched. The user has no anchor for "am I 1% done or 99% done?"

The current spinner format (src/cli.ts:218):

return `Fetching media...  ${lastMedia.processed} processed  │  ${lastMedia.downloaded} downloaded  │  ${elapsed}s`;

ft already computes the bookmark count internally — candidates.length at src/bookmark-media.ts:271 is exposed via MediaFetchProgress as candidateBookmarks. The spinner format string just drops it.

The bookmark-vs-file gotcha amplifies the gap

--limit applies to bookmarks, not files (src/cli.ts:1510: "Max pending bookmarks to process (default: all)"). One bookmark can have 1–4+ media items (carousel photos, plus a profile image). So:

  • --limit 100 is "up to 100 bookmarks" → could be 100–400+ actual file fetches
  • A user runs ft fetch-media --limit 100, watches 350 processed go by, is reasonably confused
  • Two units (bookmarks and files) matter — both should be visible

Proposed output

Unlimited (default — fetch all pending):

$ ft fetch-media
Found 2,999 bookmarks with pending media (7,250 files). Fetching all...
Fetching media...  124/7,250 files  │  118 downloaded  │  12s
✓ Done. 2,999/2,999 bookmarks, 7,247 downloaded, 3 skipped (>200MB).

With --limit:

$ ft fetch-media --limit 100
Found 2,999 bookmarks with pending media (7,250 files). Processing first 100 (250 files)...
Fetching media...  47/100 bookmarks  │  124/250 files  │  118 downloaded  │  12s
✓ Done. 100/100 bookmarks, 247 downloaded, 3 skipped. 2,899 bookmarks remaining.

User immediately understands: full scope, current-run scope, and what's left after this run.

Implementation

The data is already computed or trivially computable:

  1. Bookmark candidates: candidates.length at src/bookmark-media.ts:260 (after .slice(0, limit)). Total pending is the same array's length before the slice.

  2. File count: pure in-memory pass over the bookmarks calling resolveMediaTargets(b, coveredProfileImageUrls, skipProfileImages) and counting unique URLs (profile images dedupe across bookmarks; tweet media is per-bookmark). No I/O. ~8 lines.

    // exact file count for the candidate set (post-slice)
    const willFetch = new Set<string>();
    let fileTotal = 0;
    for (const b of candidates) {
      for (const target of resolveMediaTargets(b, coveredProfileImageUrls, skipProfileImages)) {
        if (target.isProfileImage) {
          if (!willFetch.has(target.sourceUrl)) { willFetch.add(target.sourceUrl); fileTotal++; }
        } else {
          fileTotal++;
        }
      }
    }
  3. Pre-flight line + extended spinner format: ~3 lines.

Total: ~10-15 lines. No new flags, no schema changes, no migration.

The exact count is "files to attempt." Some attempts may skip (per --max-bytes) or fail (404/5xx) during fetch — those show up in the post-run summary, not the pre-flight count.

Same logic for ft sync

ft sync calls fetchBookmarkMediaBatch internally at cli.ts:798-802 for its post-sync media phase. Same treatment surfaces the same context there too — same code site uses the same progress callback.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions