Skip to content

fix(shadow): evaluator self-writes results/shadow_live.json (FIX-2)#540

Merged
neuron7xLab merged 1 commit intomainfrom
fix/shadow-live-json-self-update
May 6, 2026
Merged

fix(shadow): evaluator self-writes results/shadow_live.json (FIX-2)#540
neuron7xLab merged 1 commit intomainfrom
fix/shadow-live-json-self-update

Conversation

@neuron7xLab
Copy link
Copy Markdown
Owner

Creator

Make scripts/evaluate_cross_asset_kuramoto_shadow.py self-write the full {"eval": ...} payload to results/shadow_live.json on every invocation, so downstream consumers (verdict scripts, trajectory loggers, external pipelines) read live state instead of silently parsing stale JSON.

Critic

  • Edge-case 1: write happens AFTER all metric computation; if eval errors out earlier the JSON is not produced. Acceptable: this matches stdout behaviour — no partial-state JSON. Downstream code must check file presence + freshness.
  • Edge-case 2: filesystem mtime resolution on coarse-grained FS (some NFS / FAT32) can collapse two sub-second writes to identical mtime. Mitigation: test sleeps 1.1s between runs.
  • Edge-case 3: file is created with default permissions; if a strict umask is in force, downstream readers under different uid may fail. Acceptable: same risk profile as LIVE_STATE.json already on disk.
  • Risk not covered: if a future refactor adds an early-return that bypasses the new write, the test still requires paper_state/equity.csv (skips on environments without it). CI runners without the spike skip the contract test — that's the documented limitation.

Auditor

Falsification gate (FIX-2): consecutive evaluator runs MUST produce results/shadow_live.json with strictly advancing mtime.

False-positive avoidance:

  • The test asserts mtime_2 > mtime_1, not >=. Two writes with identical mtime → FAIL.
  • The schema check ("eval" key present + 6 required sub-keys) prevents a degenerate fix where the file is touched but content not written.
  • A hypothetical "atomic-write-via-rename" implementation that creates a fresh inode each time is also caught: Path.stat().st_mtime resolves the new inode.

Verifier

```
$ rm -f results/shadow_live.json
$ python scripts/evaluate_cross_asset_kuramoto_shadow.py >/dev/null
$ T1=$(stat -c %Y results/shadow_live.json)
$ sleep 1.2
$ python scripts/evaluate_cross_asset_kuramoto_shadow.py >/dev/null
$ T2=$(stat -c %Y results/shadow_live.json)
$ echo "T1=$T1 T2=$T2 advanced=$([ $T2 -gt $T1 ] && echo YES || echo NO)"
T1=1778065314 T2=1778065316 advanced=YES

$ python -m pytest tests/scripts/test_shadow_eval_self_writes_json.py -x -q
. [100%]
1 passed
```

@neuron7xLab neuron7xLab enabled auto-merge (squash) May 6, 2026 11:02
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa2daf5c37

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +36 to +40
@pytest.mark.skipif(
not (
Path.home() / "spikes" / "cross_asset_sync_regime" / "paper_state" / "equity.csv"
).is_file(),
reason="Live spike paper-state not available in this environment.",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove skip gate that disables contract test coverage

This skipif makes the new contract test no-op on environments without ~/spikes/.../paper_state/equity.csv, which is common in CI, so regressions in writing results/shadow_live.json can pass undetected. The evaluator already tolerates a missing paper-state ledger and exits successfully with empty live metrics, so gating the test on that file is stricter than the runtime behavior and defeats the purpose of this regression test.

Useful? React with 👍 / 👎.

@neuron7xLab neuron7xLab force-pushed the fix/shadow-live-json-self-update branch 4 times, most recently from 2533716 to d1b7e62 Compare May 6, 2026 11:22
The cross-asset Kuramoto shadow evaluator previously only printed its
result JSON to stdout; consumers (verdict scripts, trajectory loggers,
external pipelines) had to capture stdout to persist eval state. That
made `results/shadow_live.json` a one-shot file that any downstream
read of the most recent state would silently see stale.

This change makes the evaluator write the full {"eval": eval_row}
payload directly to `results/shadow_live.json` on every invocation,
in addition to printing to stdout (preserves any stdout-capturing
caller). The path is `REPO / "results" / "shadow_live.json"`.

Falsification gate (FIX-2):
  consecutive evaluator runs MUST advance results/shadow_live.json mtime.

Verified locally:
  $ rm -f results/shadow_live.json
  $ python scripts/evaluate_cross_asset_kuramoto_shadow.py
  $ T1=$(stat -c %Y results/shadow_live.json)  # 1778065314
  $ sleep 1.2
  $ python scripts/evaluate_cross_asset_kuramoto_shadow.py
  $ T2=$(stat -c %Y results/shadow_live.json)  # 1778065316
  $ [ $T2 -gt $T1 ]  # ⇒ YES

Files:
  - scripts/evaluate_cross_asset_kuramoto_shadow.py
      - constant SHADOW_LIVE_JSON
      - write payload before print at end of main()
  - Makefile
      - new target `eval-tick` (gate command for FIX-2)
  - tests/scripts/test_shadow_eval_self_writes_json.py
      - mtime-monotonicity contract pytest, skip-gated on local
        spike paper-state availability (CI runners w/o spike data
        skip; local + integration env runs the contract)
@neuron7xLab neuron7xLab force-pushed the fix/shadow-live-json-self-update branch from d1b7e62 to 738f6ea Compare May 6, 2026 11:25
@neuron7xLab neuron7xLab merged commit 7a98ea1 into main May 6, 2026
18 of 19 checks passed
@neuron7xLab neuron7xLab deleted the fix/shadow-live-json-self-update branch May 6, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant