Len fort tracker fixes

[lenisko]

Fort tracker: POI removal correctness, hardening, cleanup

Branch: len-fort-tracker-fixes — 4 commits.

Symptom

POIs that should be marked deleted were not removed until Golbat was restarted, and often not even then. A restart only "fixed" it if the first GMO post-restart contained full coverage
of the cell — partial first GMOs left the wrong fort set in tracking.

Root cause

Two interacting bugs in ProcessCellUpdate's firstScan branch and in preload bookkeeping:

1. cell.lastSeen was only ever written inside ProcessCellUpdate. Preload (Preload / LoadFortsFromDB / RegisterFort) never set it. After every restart, every cell had
   cell.lastSeen == 0, so the next GMO took the firstScan path.
2. The firstScan branch overwrote cell.pokestops / cell.gyms with the contents of that single GMO. Any preloaded fort not in that one GMO was dropped from per-cell tracking.
   Because stale detection iterates cell.pokestops / cell.gyms, those forts could never be flagged stale again — they were stuck as "live" entries in ft.forts pointing at a cell that
   no longer listed them.

A restart only helped when the next first-GMO happened to enumerate every fort in the cell.

Other bugs found and fixed

• RestoreFort units bug. Callers passed time.Now().Unix() (seconds) into a milliseconds field. lastSeen came out ~1000× too small; the very next GMO saw a ~1e12 ms gap, blew
  past the 1h stale threshold, and re-staled the fort immediately.
• RestoreFort wall-clock vs GMO timestamp drift. After fixing the units bug, callers in updatePokestopFromFort / updateGymFromFort still used time.Now().UnixMilli() instead
  of the GMO's own fort.Timestamp. On replay or ingest lag the restored fort's lastSeen ran ahead of (or behind) the cell's lastSeen, which is sourced from the GMO timestamp —
  leaving staleness math inconsistent across the cell and its restored member.
• Future-dated AsOfTimeMs wedge. A clock-skewed or malicious scanner could permanently silence a cell — every later real GMO had timestamp <= cell.lastSeen and was dropped at
  the early-return.
• clear*WithLock ghost entries. When the gym/pokestop could not be loaded from cache or DB, the tracker entry was left in place. Same fort was re-selected as stale on every
  subsequent GMO.
• Dangling UpdateFort. Only caller was RestoreFort, and the units bug rode through it.
• GMO type-change race. If the same id appeared in pokestopIds and gymIds of one GMO, it was inserted into both sets and the conversion logic flipped back and forth.
• Preload seeded lastSeen from pokestop.Updated. Once commit 1 made cell.lastSeen non-zero post-preload (so stale detection actually ran), this became live: any fort whose DB
  Updated predated the stale threshold was flagged stale on the first GMO that didn't include it. Co-primary with the root cause; fixed together in commit 3.

Fix summary

Four commits, in order:

1. fix(fort_tracker) — d20d5ca

• RegisterFort seeds cell.lastSeen, so the first post-restart GMO is no longer treated as a firstScan.
• firstScan branch merges (does not replace) cell fort sets, refreshing lastSeen for retained forts.
• Reject GMOs with timestamps more than 60s in the future.
• Dedup ids that appear in both pokestopIds and gymIds (gym wins, matches proto pack order).
• Defensive insertion into the new cell set during cell-move handling.
• clear*WithLock removes the tracker entry when the entity cannot be loaded.
• Drop unused UpdateFort. RestoreFort does the work directly with explicit ms units; callers now pass time.Now().UnixMilli().

2. refactor(fort_tracker) — 11f8a04

• Collapse loadPokestopsFromDB / loadGymsFromDB into one loadFortKindFromDB helper.
• Replace clearGymWithLock / clearPokestopWithLock with a generic clearFortWithLock plus per-kind ops tables (gymClearOps, pokestopClearOps).
• Hoist debug "pending removal" logs out of the global tracker lock by splitting ProcessCellUpdate into a thin wrapper plus locked core that returns the pending lists.
• Downgrade per-conversion log to Debug (already logged at Info by clearFortWithLock).
• Iterate cellForts directly in CheckRemovedForts; drop the parallel cellsToBeCleaned slice in decodeGMO.
• strconv.FormatUint for cellId formatting in API responses (S2 cell ids exceed JS 2^53 safe-int — observed DB max 8935267727656878080 — so they're stringified for JSON consumers).
• Fix doc-comment / type-name mismatches on FortTrackerLastSeen and FortTrackerGMOContents.

3. feat(fort_tracker) — 26140b2

• New config cleanup.forts_min_miss_count (default 1, preserves prior behaviour). When set higher, a fort must be absent from N consecutive cell scans past the time threshold before
  being flagged stale — defends against transient single-frame coverage gaps and level-30 gating.
• Track missCount per fort, reset on every sighting (including the firstScan merge path).
• Treat preload as a confirmation event: lastSeen is time.Now().UnixMilli() instead of row.Updated * 1000. Required for commit 1's cell.lastSeen seeding to be safe — otherwise
  preloaded forts with old Updated values get instantly staled.
• Tests for the §6.6 design-review scenarios:
  • partial first GMO post-preload preserves preloaded forts;
  • future-dated GMO does not wedge the cell;
  • RestoreFort followed by partial GMO does not re-stale;
  • id present in both id-lists dedups (gym wins);
  • missCount guard with N=2 needs two consecutive misses;
  • missCount resets on intervening sighting.

4. fix(fort_tracker) — RestoreFort uses GMO timestamp

• updatePokestopFromFort passes now*1000 (param already received as fort.Timestamp/1000 seconds) into RestoreFort instead of time.Now().UnixMilli().
• updateGymFromFort gains a timestampMs int64 parameter; gmo_decode.go call site passes fort.Timestamp.
• Eliminates drift between a restored fort's lastSeen and its cell's lastSeen under ingest lag or replay.

Out of scope

• Unbounded growth of ft.cells / ft.forts. GMOs sometimes return empty; we don't want to drop cells whose forts vanished transiently.
• Race between LoadFortsFromDB and live ingest. Loading already blocks before listeners accept traffic.
• Soft-delete intermediate state in DB (§6.3). We don't want to change the DB-side stale-to-deleted transition here.

Testing

• go test ./... — all green.
• Tracker tests cover all five §6.6 scenarios from the design review, plus regression tests for missCount semantics.

Config

[cleanup]
forts_stale_threshold = 3600   # seconds, default 1 h (unchanged)
forts_min_miss_count  = 1      # consecutive cell-scan misses, default 1

# Operators wanting fewer false positives from transient coverage gaps can set forts_min_miss_count = 2 or higher.

[lenisko]

Might need some long-term testing, cc @Fabio1988

[jfberry]

Some parts of the description don’t make any sense

[lenisko]

I will review and slap my tool later on :)

[lenisko]

Updated. Haven't tested myself yet.

[lenisko]

Started testing

[lenisko]

So far results are far better. @Fabio1988 mind joining in testing?

[jfberry]

Reviewed on my phone but I saw: cellid as string in data structure? Why?
Restoring forts from gmo takes now milliseconds, not time from gmo?

[lenisko]

Reviewed on my phone but I saw: cellid as string in data structure? Why? Restoring forts from gmo takes now milliseconds, not time from gmo?

It's uint64 internally, string for JSON debug output. I pulled max value of 8935267727656878080 on my end, which is past what JS int can get.

Fixed passed ts.

[Fabio1988]

tested, works well so far