fix(ops): shadow trajectory daily-tick automation via systemd (FIX-5)#544
fix(ops): shadow trajectory daily-tick automation via systemd (FIX-5)#544neuron7xLab merged 1 commit intomainfrom
Conversation
c92575d to
a6cc477
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c92575d433
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "$PYTHON" "$SPIKE_TRADER" --tick || { | ||
| rc=$? | ||
| emit "tick: FAIL rc=$rc (continuing — paper_trader may rate-limit on intra-day reruns)" |
There was a problem hiding this comment.
Fail the wrapper when paper_trader --tick exits non-zero
The --tick command failure is explicitly swallowed here, so the unit keeps running and can still emit the terminal success marker (tick: ok) even when no new bar was advanced. In environments with transient API/network errors, this creates false-positive health signals for FIX-5 and allows trajectory/eval outputs to be produced from stale paper state instead of the intended daily tick.
Useful? React with 👍 / 👎.
| emit "tick: FAIL rc=$rc (continuing — paper_trader may rate-limit on intra-day reruns)" | ||
| } | ||
| else | ||
| emit "tick: SKIP (spike paper_trader not present at $SPIKE_TRADER)" |
There was a problem hiding this comment.
Treat missing paper_trader path as fatal
If the configured SPIKE_TRADER file is absent, the script logs SKIP and continues instead of failing, which means the systemd run can appear successful without performing the core daily tick. On any host where the spike checkout path differs or is missing, this silently defeats the automation objective while still letting downstream steps and monitoring proceed as if the tick happened.
Useful? React with 👍 / 👎.
The 75-bar shadow validation window until the 2026-07-10 truth gate
demands daily ritual: tick → pipeline_status refresh → eval +
shadow_live.json → CI-aware verdict → DP3 honest test → append a
trajectory.log entry. Manual cadence over 75 cycles is failure-prone.
The shadow timer file already existed (ops/systemd/...timer fires
daily at 22:00 UTC); what was missing was a single wrapper script and
a service unit that ran it.
This commit lands:
- scripts/shadow_daily_tick.sh — bash wrapper running all 6 steps
in order, idempotent on the same UTC day. Emits the literal
terminal "tick: ok" line on success — observable via journalctl.
- ops/systemd/cross_asset_kuramoto_shadow.service — updated to call
the wrapper as ExecStart instead of the bare evaluator. Log
capture switched to StandardOutput=journal / StandardError=journal
so journalctl can match unit logs (the older append:%path was
invisible to the FIX-5 gate).
- .claude/commit_acceptors/systemd-trajectory-timer.yaml
Falsification gate (FIX-5):
$ systemctl --user is-active cross_asset_kuramoto_shadow.timer
active
$ journalctl --user-unit=cross_asset_kuramoto_shadow.service \
--since='-1 day' --no-pager | grep -c 'tick: ok'
>= 1
NOTE: on systemd 255+ the canonical incantation is --user-unit=NAME
(combined form). The earlier draft used --user -u NAME (separate
flags) which does NOT match unit logs in this version — the
acceptor and the wrapper docstring document the working form.
Locally verified: systemctl --user start ... emits 7 wrapper lines
through journal terminating in "tick: ok"; results/sharpe_trajectory.log
gets one new dated row; results/shadow_live.json mtime advances.
a6cc477 to
cc4adb7
Compare
Creator
New
scripts/shadow_daily_tick.shwrapper runs the full daily ritual (tick → pipeline_status → eval → verdict → DP3 → trajectory.log) and is bound as ExecStart of the existingops/systemd/cross_asset_kuramoto_shadow.service. Closes the manual-cadence failure mode for the 75-bar window before the 2026-07-10 truth gate.Critic
paper_trader.py --tickmay fail (rate limit, network blip). Wrapper catches non-zero rc, logs FAIL, continues. The eval/verdict downstream still runs against the prior paper-state.make eval-tickis idempotent on results/shadow_live.json but appends to LIVE_SCOREBOARD csv. Acceptable: the eval was already idempotent against duplicate dates.2>/dev/null || trueand substitutesUNAVAILABLEin the trajectory line.journalctl --user -u SERVICEparsing; the FIX-5 falsification gate must use--user-unit=SERVICE(combined form). Documented in the acceptor + commit body. Older systemd installs that still accept the separate-flag form remain compatible.Auditor
Falsification gate (FIX-5): after a successful invocation,
journalctl --user-unit=cross_asset_kuramoto_shadow.service --since='-1 day'MUST contain at least onetick: okline. The literal string is emitted only as the final wrapper step, after trajectory.log append; partial / failed runs do NOT emit it.False-positive avoidance:
tick: okis gated to the END of the wrapper. Earlier failures abort viaset -euo pipefailand never reach the line.falsifierblock probessystemctl --user is-active cross_asset_kuramoto_shadow.timer; if disabled/masked, FIX-5 is broken regardless of journal content.Verifier
```
$ systemctl --user is-active cross_asset_kuramoto_shadow.timer
active
$ systemctl --user start cross_asset_kuramoto_shadow.service
$ sleep 5
$ journalctl --user-unit=cross_asset_kuramoto_shadow.service \
--since='-1 day' --no-pager | grep -c 'tick: ok'
2
$ tail -1 results/sharpe_trajectory.log
2026-05-06T13:38:46Z bar=15 sharpe=-3.92 ci=[-13.76,+0.43] verdict=AMBIGUOUS dp3=REJECT
```
The single dated trajectory row at the end is the canonical signal of a successful FIX-5 invocation: it composes FIX-1 (pipeline_status→op_unsafe), FIX-2 (shadow_live.json), FIX-4 (CI-aware verdict), and FIX-3 (DP3 honest test) into one observable line.