feature(internal, cmd): protocol-migrate history

[aristidesstaffieri]

Closes #515

What

Implements the protocol-migrate history CLI subcommand, backfills protocol history for historical ledgers within the retention window, synchronizing with live ingestion through a shared per-protocol history CAS cursor.

• ProtocolMigrateHistoryService — walks forward from the oldest ingest cursor to the latest cursor, calling PersistHistory at each ledger with batch processing and CAS-based progress tracking. On CAS failure, detects handoff from live ingestion and exits cleanly. Tracks history_migration_status lifecycle (in_progress → success/failed).
• protocol-data-migrate CLI command (cmd/protocol_data_migrate.go) — accepts --protocol-id flags to select which protocols to backfill.
• Shared helpers extracted to ingest_helpers.go — getLedgerWithRetry, buildProtocolProcessorMap, and cursor name formatters deduplicate logic between history migration and live ingestion.
• UpdateHistoryMigrationStatus added to ProtocolsModel for migration lifecycle tracking.
• Configurable cursor names — history migration respects --latest-ledger-cursor-name / --oldest-ledger-cursor-name overrides, matching live ingestion behavior.
• Convergence poll — distinguishes poll-deadline-exceeded (converged) from transient RPC errors (retry) to avoid premature success marking during network blips.
• Bulk StatusFailed — excludes protocols already handed off to live ingestion via CAS failure, preventing re-processing conflicts.
• Integration tests covering history-to-live handoff, asymmetric cursor CAS paths (history ready but current_state lags), and BoundedRange→UnboundedRange transitions on shared ledger backends.

Why

Protocol state production needs historical data within the retention window before live ingestion can provide complete coverage. Without this command, protocols only have history state from the point live ingestion started using their processors — all prior ledgers are missing. This subcommand lets operators backfill per-protocol, with safe handoff to live
ingestion via CAS cursors ensuring no gaps or double-processing at the migration boundary.

Known limitations

N/A

Issue that this PR addresses

#515

Checklist

PR Structure

• It is not possible to break this PR down into smaller PRs.
• This PR does not mix refactoring changes with feature changes.
• This PR's title starts with name of package that is most changed in the PR, or all if the changes are broad or impact many packages.

Thoroughness

• This PR adds tests for the new functionality or fixes.
• All updated queries have been tested (refer to this check if the data set returned by the updated query is expected to be same as the original one).

Release

• This is not a breaking change.
• This is ready to be tested in development.
• The new functionality is gated with a feature flag if this is not ready for production.

[Copilot]

Pull request overview

Adds a new protocol history backfill workflow to support protocol state production over the retention window, with a CLI entrypoint and shared ingest helpers to coordinate handoff with live ingestion via CAS cursors.

Changes:

• Introduces ProtocolMigrateHistoryService (+ extensive unit tests) to backfill per-protocol history using CAS progress tracking and clean handoff semantics.
• Adds protocol-migrate history CLI command and integration tests covering migration→live handoff and cursor edge-cases.
• Extracts shared ingest helpers (ledger fetch retry, processor map building, cursor name helpers) and adds UpdateHistoryMigrationStatus to the protocols model.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
internal/services/protocol_migrate_history.go	Implements history migration service with CAS-based cursor advancement and convergence polling.
internal/services/protocol_migrate_history_test.go	Adds comprehensive unit tests for migration success/failure, resume, handoff, and convergence polling.
internal/services/ingest_helpers.go	Adds shared helpers: getLedgerWithRetry, buildProtocolProcessorMap, and protocol cursor name formatters.
internal/services/ingest.go	Reuses buildProtocolProcessorMap; removes duplicated getLedgerWithRetry method.
internal/services/ingest_live.go	Switches to shared cursor name helpers and shared getLedgerWithRetry.
internal/services/ingest_backfill.go	Switches to shared getLedgerWithRetry.
internal/services/ingest_test.go	Updates tests to call shared getLedgerWithRetry directly.
internal/integrationtests/data_migration_test.go	Adds integration coverage for history migration→live ingestion handoff and cursor-asymmetry scenarios.
internal/data/protocols.go	Adds UpdateHistoryMigrationStatus to support migration lifecycle tracking.
internal/data/protocols_test.go	Adds test for UpdateHistoryMigrationStatus.
internal/data/mocks.go	Updates ProtocolsModelMock to include UpdateHistoryMigrationStatus.
internal/data/ingest_store.go	Adds exported constants for ingest cursor key names.
cmd/root.go	Registers the new protocol-migrate command.
cmd/protocol_data_migrate.go	Implements protocol-migrate history CLI subcommand and wires it to the service.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aditya1702]

Found some issues with the migration logic

[aditya1702]

Why did we remove this function (didnt we introduce in earlier PR?)

[aditya1702]

I think we should not have methods on receivers just for tests to use. We should find some way to do this in the test file. I think this might be against Go best practices. Can you double check this one?

[aditya1702]

Same check for this function too - this function is only used in tests so is this fine adding a method for that on ingestService reciever?

[aditya1702]

Is this going to be shared by goroutines?

[aditya1702]

Since this is a historical ledger processing (migration), why not use the datastore backend? Because that is better suited for backfilling, with RPC we will have to ensure the historical range is supported by its history retention

[aditya1702]

Why are we tracking the minCursor here?

[aditya1702]

Can we break this function in smaller sub functions for readability. It is very long and hard to understand on all the moving parts

[aditya1702]

Awesome test suite 👏

I think these 2 are missing:

1. Context cancellation during processing — There's a select { case <-ctx.Done() } check at line 304 but no test cancels a context mid-batch. A test with
   context.WithCancel that cancels after N ledgers would verify the cleanup path returns correct handedOffProtocolIDs.
2. oldest_ingest_ledger is 0 — The guard at line 226 (return nil, fmt.Errorf("ingestion has not started yet")) has no test.

[aditya1702]

I think I found a valid bug here based on my knowledge of ledger backend. I used Claude to craft this reply:

Bug: PrepareRange called multiple times on RPCLedgerBackend — fails in production

processAllProtocols calls PrepareRange on every iteration of the outer for {} loop (line 299) and again during the convergence poll (line 411).
However, RPCLedgerBackend.PrepareRange returns an error if the backend is already
prepared:

if b.preparedRange != nil {
    return fmt.Errorf("RPCLedgerBackend is already prepared with range [%d, %d]",
        b.preparedRange.from, b.preparedRange.to)
}

There is no reset mechanism — preparedRange is only nil at construction time.

Affected paths

Path	Where	What happens
Tip advances after first batch	Line 299 (second outer loop iteration)	PrepareRange returns error → function returns with failure
Convergence poll	Line 411	PrepareRange(UnboundedRange(...)) returns error → misinterpreted as transient error → infinite retry loop
Transient poll error retry	Line 421 → continue → line 299	Same as tip-advance case

Why tests don't catch this

All test backends (multiLedgerBackend, transientErrorBackend, rangeTrackingBackend) implement PrepareRange as a no-op that always returns nil.
None of them enforce the single-prepare constraint that RPCLedgerBackend has.

Impact

The function works correctly only when the entire backfill completes in a single batch — i.e., all ledgers from startLedger to latestLedger are
processed and the tip doesn't advance during that time. If the tip advances or the convergence poll is reached, the service errors out.

[aditya1702]

Following from the PrepareRange bug above — rather than patching the re-call sites, the entire outer loop and convergence poll can be replaced with a
single UnboundedRange.

Current design

PrepareRange(BoundedRange(start, latest))    ← first call
  process ledgers start..latest
  tip advanced?
    PrepareRange(BoundedRange(next, newLatest))  ← BUG: errors on RPCLedgerBackend
  at tip?
    PrepareRange(UnboundedRange(latest+1))       ← BUG: same
    GetLedger with 5s timeout (convergence poll)
    new ledger? loop again → PrepareRange(Bounded...) ← BUG: same

Proposed design

PrepareRange(UnboundedRange(startLedger))    ← once, at the top

for seq := startLedger; ; seq++ {
    // for each tracker, check if converged (via timeout on GetLedger)
    pollCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
    meta, err := s.ledgerBackend.GetLedger(pollCtx, seq)
    cancel()
    if errors.Is(err, context.DeadlineExceeded) {
        // no new ledger in 5s → converged at seq-1
        break
    }
    // process meta for eligible trackers...
}

Why this works with RPCLedgerBackend

• PrepareRange is called exactly once — no conflict with the single-prepare constraint
• UnboundedRange means GetLedger accepts any sequence number without range validation
• RPCLedgerBackend.GetLedger already has built-in 2s polling when a ledger is beyond the RPC tip — so the timeout context is the only convergence
  mechanism needed
• Bounded vs Unbounded has no behavioral difference in RPCLedgerBackend other than the range check at line
  131

What gets removed

• The outer for {} loop (lines 268–441)
• minCursor recomputation on each iteration
• The 40-line convergence poll section (lines 402–440)
• All Bounded/Unbounded mode switching
• The PrepareRange bug entirely

[aditya1702]

Btw this same refactor should also work with the datastore backend since that also supports live ingestion of new ledgers