Post apr3 test fixes by haoruizhou · Pull Request #60 · Western-Formula-Racing/daq-radio

haoruizhou · 2026-04-03T15:25:47Z

Add local CAN logging
Fix time injection logic to the car
port 8080 dashboard UI update

Implement parseTSMasterBlf to parse TSMaster/BLF CAN and CANFD objects (including LOBJ records and zlib-compressed BLF via pako). Update parseReplayFile to handle .blf files, normalize/sort timestamps, and emit warnings/errors. Add unit tests for BLF parsing and update .gitignore to exclude testdata. Add pako to package.json (and adjust devDependencies ordering).

Allow users to import .blf replay files and add parsing support for TSMaster/Vector BLF formats. Updated file inputs in TimelineBar and ReplayViewer to accept .blf files. Enhanced parseTSMasterBlf to detect LOGG/LOBJ structures, handle LOG_COMPRESSED zlib blocks (decompressing with pako), parse 48-byte LOBJ CAN_MESSAGE records, and fall back to previous decompression/LOBJ parsing for non-LOGG files. Added @types/pako to dev dependencies. Includes improved error/warning reporting for decompression and block parsing.

Coldstore

Replace occurrences of the 192.168.1.x radio subnet with 10.71.1.x in SETUP_CARD.tex and update deploy environment templates (.env.base, .env.car, .env.macbook) so REMOTE_IP defaults reflect the new addressing. Add deploy/MACBOOK_DEPLOY.md with instructions for running the telemetry stack on a MacBook/WSL2 (quick start, defaults, credentials, profiles, and troubleshooting).

Allow mounting a host DBC and persist raw CAN data, and modernize cloud sync for InfluxDB3. Adds DBC_HOST_PATH to env files and docker-compose mounts /app/active.dbc from the host (with a raw_can_data volume) and fixes LOCAL/CLOUD bucket defaults to WFR26. Introduces a parquet-based raw CAN logger (pyarrow) that batches writes from an async queue in data.py and exposes the active DBC path in published system stats. Updates cloud_sync.py to use influxdb_client_3 and pandas, query with SQL into a DataFrame, apply vectorized time corrections, and write the DataFrame to cloud with proper tag columns and client cleanup. Also adds a UI warning banner when the example DBC is in use.

Introduce a car time injector that POSTs the base station clock to the car Pi's /set-time endpoint every 30s (with a 5s startup delay). Add a safety gate that blocks injection if the base clock is before 2026-04-01 and expose two new status flags (car_time_synced, base_clock_bad) in system stats. Update the web UI to show a banner when the base clock is bad, surface auto-sync status and adjust sync copy/buttons. Also add SET_TIME_ENABLED to the car .env and fix the macOS deploy compose command path.

Introduce persistent logging of raw CAN messages: CAN readers now push raw records (time_ms, can_id, data bytes) into a car_raw_queue. Added a car_parquet_logger that writes rotating Parquet parts (5-minute rotation) for zero-loss direct can0 capture (uses BOOT_SESSION_ID in filenames). Replaced the previous raw Parquet uplink logger with raw_csv_logger that appends durable CSV rows (time_ms, can_id, data_hex) to raw_can_<session>.csv, writing a header when creating a new file and flushing after each batch. Also import csv and wire the new loggers into the TelemetryNode tasks list; batch sizes and basic error handling are preserved.

Enable remote car shutdown/time injection and UI/status updates. - deploy/.env.car: enable SHUTDOWN_ENABLED by default for the car profile. - src/status_server.py: import urllib, add SHUTDOWN_ENABLED/REMOTE_IP/REMOTE_STATUS_PORT config and new endpoints: /inject-car-time, /shutdown and /shutdown-car. Implement handlers to proxy time injection to the remote car, send remote shutdown, and perform local shutdown (subprocess shutdown). Add robust HTTPError/URLError handling and appropriate JSON responses. - src/data.py: improve car_time_injector exception handling to catch urllib.error.HTTPError and URLError, set _car_time_synced accordingly, and log more specific messages. - status/index.html: update UI to inject time to the Car Pi (instead of only base station), add status indicators for inject/shutdown actions, and add a Shutdown Car Pi button with JS that calls /shutdown-car and displays result. These changes allow manual/remote clock injection and shutdown commands with clearer status feedback and better error handling.

Copilot

Pull request overview

This PR updates the DAQ base station stack to (a) persist raw CAN logs locally, (b) improve/automate base→car clock injection and expose related status in the UI, and (c) adjust the port-8080 status dashboard plus deployment defaults (DBC mounting, volumes, and Influx bucket config).

Changes:

Add raw CAN logging: car-side Parquet logging and base-side CSV logging under /app/raw_can_logs with new docker volume wiring.
Add clock-sync UX + new status endpoints for injecting time to the car Pi and sending a shutdown command to the car Pi.
Refactor cloud sync to query local InfluxDB3 via SQL and write to cloud using a DataFrame.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
universal-telemetry-software/status/index.html	Adds new banners (base clock bad / DBC placeholder), inject-time UI, and a shutdown-car button.
universal-telemetry-software/src/status_server.py	Adds `/inject-car-time`, `/shutdown`, `/shutdown-car` POST handlers and remote forwarding logic.
universal-telemetry-software/src/data.py	Adds raw CAN loggers (car Parquet + base CSV), publishes new UI stats, and adds periodic car time injection.
universal-telemetry-software/src/cloud_sync.py	Switches local read path to InfluxDB3 SQL + DataFrame-based cloud write.
universal-telemetry-software/deploy/MACBOOK_DEPLOY.md	Updates MacBook deploy instructions (compose invocation path).
universal-telemetry-software/deploy/docker-compose.yml	Sets active DBC mount, adds raw CAN volume, and hard-codes Influx buckets.
universal-telemetry-software/deploy/docker-compose.staging.yml	Same as above for staging compose.
universal-telemetry-software/deploy/docker-compose.prod.yml	Same as above for prod compose.
universal-telemetry-software/deploy/docker-compose.macbook.yml	Same as above for MacBook compose.
universal-telemetry-software/deploy/.env.macbook	Adds `DBC_HOST_PATH` example.
universal-telemetry-software/deploy/.env.car	Enables `SET_TIME_ENABLED` and `SHUTDOWN_ENABLED` by default for the car template.
universal-telemetry-software/deploy/.env.base	Adds `DBC_HOST_PATH` example.
.gitignore	Ignores additional `.claude` folders.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-03T15:33:23Z

universal-telemetry-software/status/index.html

-                <small id="syncDetail">Data is held until a valid clock source is established (ECU RTC, NTP, or manual set)</small>
+                <strong id="syncTitle">⏳ Waiting for clock sync</strong>
+                <small id="syncDetail">Timestamps may be inaccurate until a clock source is established (ECU RTC, NTP, or manual set)</small>
+                <small id="carSyncStatus" style="display:none; color:#4ade80; margin-top:4px; display:block;">Car clock: auto-synced ✓</small>


carSyncStatus has conflicting inline styles (display:none and display:block); the last one wins, so the “Car clock: auto-synced” line will render even before stats arrive. Remove the duplicate and leave the initial state hidden (or set it from JS only).

Suggested change

<small id="carSyncStatus" style="display:none; color:#4ade80; margin-top:4px; display:block;">Car clock: auto-synced ✓</small>

<small id="carSyncStatus" style="display:none; color:#4ade80; margin-top:4px;">Car clock: auto-synced ✓</small>

Copilot · 2026-04-03T15:33:24Z

universal-telemetry-software/status/index.html

+            if (stats.dbc_file) {
+                const dbcWarning = document.getElementById('dbcWarningBanner');
+                if (stats.dbc_file.includes('example.dbc')) {
+                    dbcWarning.style.display = 'block';
+                } else {
+                    dbcWarning.style.display = 'none';
+                }


The DBC placeholder warning checks stats.dbc_file.includes('example.dbc'), but the compose files now set DBC_FILE_PATH=/app/active.dbc regardless of whether the mounted host DBC is the placeholder. This means the warning banner will likely never show even when using the example DBC—consider sending an explicit boolean like using_example_dbc from the backend, or checking the actual host path/env that indicates the placeholder is in use.

Suggested change

if (stats.dbc_file) {

const dbcWarning = document.getElementById('dbcWarningBanner');

if (stats.dbc_file.includes('example.dbc')) {

dbcWarning.style.display = 'block';

} else {

dbcWarning.style.display = 'none';

}

const dbcWarning = document.getElementById('dbcWarningBanner');

if (dbcWarning) {

const usingExampleDbc = stats.using_example_dbc !== undefined

? Boolean(stats.using_example_dbc)

: Boolean(stats.dbc_file && stats.dbc_file.includes('example.dbc'));

dbcWarning.style.display = usingExampleDbc ? 'block' : 'none';

Copilot · 2026-04-03T15:33:24Z

universal-telemetry-software/src/status_server.py

+SET_TIME_ENABLED  = os.getenv("SET_TIME_ENABLED",  "false").lower() == "true"
+SHUTDOWN_ENABLED  = os.getenv("SHUTDOWN_ENABLED",  "false").lower() == "true"
+REMOTE_IP         = os.getenv("REMOTE_IP", "")
+REMOTE_STATUS_PORT = int(os.getenv("STATUS_PORT", 8080))


REMOTE_STATUS_PORT is derived from STATUS_PORT, which is also used for the local status server port. If the base station status UI is served on a non-8080 port, the server will incorrectly target the car on that same port. Use a dedicated env var (e.g., CAR_STATUS_PORT) for the remote car status server port.

Suggested change

REMOTE_STATUS_PORT = int(os.getenv("STATUS_PORT", 8080))

REMOTE_STATUS_PORT = int(os.getenv("CAR_STATUS_PORT", 8080))

Copilot · 2026-04-03T15:33:24Z

universal-telemetry-software/src/status_server.py

+        elif self.path == '/shutdown':
+            if not SHUTDOWN_ENABLED:
+                self._json_response(403, {"error": "shutdown is disabled (set SHUTDOWN_ENABLED=true)"})
+                return
+            self._handle_shutdown()
+        elif self.path == '/shutdown-car':
+            self._handle_shutdown_car()
+        elif self.path == '/inject-car-time':
+            self._handle_inject_car_time()


/shutdown-car and /inject-car-time are exposed without any feature-flag gating or authentication, while this server also sets Access-Control-Allow-Origin: * and allows POSTs. On a shared network, any webpage could trigger these endpoints and shut down the car Pi or change its clock. Gate these endpoints behind explicit env flags (separate from the car’s own flags) and/or require a shared secret/token (and strongly consider restricting CORS for these endpoints).

Copilot · 2026-04-03T15:33:25Z

universal-telemetry-software/src/data.py

+        car_raw_queue = asyncio.Queue()
+        _car_parquet_schema = pa.schema([
+            ("time_ms", pa.int64()),
+            ("can_id", pa.int64()),
+            ("data", pa.binary()),
+        ])


car_raw_queue is an unbounded asyncio.Queue() and frames are pushed with put_nowait(). If disk IO falls behind or CAN rate exceeds the logger’s drain rate, the queue will grow without bound and can OOM the process. Consider setting a reasonable maxsize plus a drop/backpressure policy (e.g., drop oldest/newest with a counter exposed in stats).

Copilot · 2026-04-03T15:33:26Z

universal-telemetry-software/deploy/docker-compose.yml

      - LOCAL_INFLUX_URL=http://influxdb3:8181
      - LOCAL_INFLUX_TOKEN=apiv3_local-telemetry-token
      - LOCAL_INFLUX_ORG=WFR
-      - LOCAL_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - LOCAL_INFLUX_BUCKET=WFR26
      - INFLUX_TABLE=${INFLUX_SEASON:-WFR26}_base
-      - DBC_FILE_PATH=/app/example.dbc
+      - DBC_FILE_PATH=/app/active.dbc
      # Cloud sync (disabled by default in dev)
      - CLOUD_INFLUX_URL=${CLOUD_INFLUX_URL:-https://influxdb3.westernformularacing.org}
      - CLOUD_INFLUX_TOKEN=${CLOUD_INFLUX_TOKEN:-}
      - CLOUD_INFLUX_ORG=WFR
-      - CLOUD_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - CLOUD_INFLUX_BUCKET=WFR26
      - SYNC_INTERVAL_SECONDS=60


LOCAL_INFLUX_BUCKET (and CLOUD_INFLUX_BUCKET) are now hard-coded to WFR26, but other parts of the stack (docs, .env, and INFLUX_TABLE=${INFLUX_SEASON...}) still imply season/bucket is configurable via INFLUX_SEASON. If bucket selection is intended to remain configurable, revert to ${INFLUX_SEASON:-WFR26} here; otherwise update the related config/docs to avoid a misleading knob.

Copilot · 2026-04-03T15:33:26Z

universal-telemetry-software/deploy/docker-compose.staging.yml

      - LOCAL_INFLUX_URL=http://influxdb3:8181
      - LOCAL_INFLUX_TOKEN=${LOCAL_INFLUX_TOKEN:-apiv3_local-telemetry-token}
      - LOCAL_INFLUX_ORG=WFR
-      - LOCAL_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - LOCAL_INFLUX_BUCKET=WFR26
      - INFLUX_TABLE=${INFLUX_SEASON:-WFR26}_base
-      - DBC_FILE_PATH=/app/example.dbc
+      - DBC_FILE_PATH=/app/active.dbc
      # Cloud sync
      - CLOUD_INFLUX_URL=${CLOUD_INFLUX_URL:-https://influxdb3.westernformularacing.org}
      - CLOUD_INFLUX_TOKEN=${CLOUD_INFLUX_TOKEN:-}
      - CLOUD_INFLUX_ORG=WFR
-      - CLOUD_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - CLOUD_INFLUX_BUCKET=WFR26
      - SYNC_INTERVAL_SECONDS=60


LOCAL_INFLUX_BUCKET/CLOUD_INFLUX_BUCKET are now hard-coded to WFR26, but INFLUX_TABLE still uses INFLUX_SEASON and the docs/environment suggest season/bucket is configurable. If this is not intentional, restore ${INFLUX_SEASON:-WFR26} for the bucket variables (or update the surrounding docs/config to match).

Copilot · 2026-04-03T15:33:27Z

universal-telemetry-software/deploy/docker-compose.prod.yml

      - LOCAL_INFLUX_URL=http://127.0.0.1:9000 # Host-mapped port (9000:8181) with network_mode: host
      - LOCAL_INFLUX_TOKEN=${LOCAL_INFLUX_TOKEN:-apiv3_local-telemetry-token}
      - LOCAL_INFLUX_ORG=WFR
-      - LOCAL_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - LOCAL_INFLUX_BUCKET=WFR26
      - INFLUX_TABLE=${INFLUX_SEASON:-WFR26}_base # e.g. WFR26_base, WFR27_base
-      - DBC_FILE_PATH=/app/example.dbc
+      - DBC_FILE_PATH=/app/active.dbc
      # Cloud sync
      - CLOUD_INFLUX_URL=${CLOUD_INFLUX_URL:-https://influxdb3.westernformularacing.org}
      - CLOUD_INFLUX_TOKEN=${CLOUD_INFLUX_TOKEN:-}
      - CLOUD_INFLUX_ORG=WFR
-      - CLOUD_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - CLOUD_INFLUX_BUCKET=WFR26
      - SYNC_INTERVAL_SECONDS=60


LOCAL_INFLUX_BUCKET/CLOUD_INFLUX_BUCKET are now hard-coded to WFR26, but INFLUX_TABLE still uses INFLUX_SEASON and existing deploy conventions imply season is configurable. If the intent is to support multiple seasons/buckets, keep the bucket variables parameterized instead of hard-coding.

Copilot · 2026-04-03T15:33:27Z

universal-telemetry-software/deploy/docker-compose.macbook.yml

      - LOCAL_INFLUX_URL=http://daq-influxdb3:8181
      - LOCAL_INFLUX_TOKEN=${LOCAL_INFLUX_TOKEN:-apiv3_local-telemetry-token}
      - LOCAL_INFLUX_ORG=WFR
-      - LOCAL_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - LOCAL_INFLUX_BUCKET=WFR26
      - INFLUX_TABLE=${INFLUX_SEASON:-WFR26}_base
-      - DBC_FILE_PATH=/app/example.dbc
+      - DBC_FILE_PATH=/app/active.dbc
      - CLOUD_INFLUX_URL=${CLOUD_INFLUX_URL:-https://influxdb3.westernformularacing.org}
      - CLOUD_INFLUX_TOKEN=${CLOUD_INFLUX_TOKEN:-}
      - CLOUD_INFLUX_ORG=WFR
-      - CLOUD_INFLUX_BUCKET=${INFLUX_SEASON:-WFR26}
+      - CLOUD_INFLUX_BUCKET=WFR26
      - SYNC_INTERVAL_SECONDS=60


LOCAL_INFLUX_BUCKET/CLOUD_INFLUX_BUCKET are now hard-coded to WFR26 even though the MacBook deploy doc and INFLUX_TABLE=${INFLUX_SEASON...} still imply the season/bucket is configurable. Consider restoring ${INFLUX_SEASON:-WFR26} here or updating the documentation/config to match the new behavior.

Copilot · 2026-04-03T15:33:27Z

universal-telemetry-software/deploy/.env.car

+SET_TIME_ENABLED=true
+SHUTDOWN_ENABLED=true


This template enables SET_TIME_ENABLED and SHUTDOWN_ENABLED by default on the car. Since the status server has no authentication and explicitly allows cross-origin POSTs, leaving these on makes it easy for anyone on the network (or a malicious webpage) to change the car clock or shut it down. Consider defaulting these to false and only enabling them for controlled sessions, or add an auth mechanism before keeping them true by default.

Suggested change

SET_TIME_ENABLED=true

SHUTDOWN_ENABLED=true

SET_TIME_ENABLED=false

SHUTDOWN_ENABLED=false

Track UDP/TCP packet sequence status in TelemetryNode and surface it to the web UI. In src/data.py: add status_map, latest_seq and last_udp_time; mark missing sequences on gaps, update entries on out-of-order UDP and TCP recovery, prune old entries, and include last_udp_time and a status_buffer in the system_stats payload. In status/index.html: add CSS, canvas UI and legend for a sequence tape, consume status_buffer and last_udp_time, draw the sequence tape (missing/udp/recovered), add faster CAR timeout and more frequent UI updates, and show an out-of-range visual indicator. In docker-compose.yml: mount source, status assets and main.py into the container for the UI and runtime access.

Introduce an optional WebSocket downlink relay and multi-URL client failover for the PECAN stack. Frontend (pecan): add UI to SettingsModal to view/edit a newline-separated failover list and apply it (stored under localStorage key pecan-ws-candidates). Improve modal accessibility/interaction and only sync WS fields when the modal opens. Adjust WebSocketService tests to avoid slow multi-URL defaults. WebSocket client (pecan/src/services/WebSocketService.ts): add PECAN_WS_CANDIDATES_KEY and DEFAULT_WS_FAILOVER_URLS, implement resolution of candidate URLs, normalization, ordering (favor last successful URL), timed failover attempts (2.5s per candidate), and binding of socket handlers. Reconnect logic uses async connect flow and preserves status notifications; reconnect() now awaits connect. Server/tools (universal-telemetry-software): add a new ws_relay module (src/ws_relay.py) implementing a downlink-only fan-out relay with optional token-based handshake gating (for Cloudflare Tunnel/loopback). Expose run script via pyproject (uts-ws-relay) and conditionally start relay from main.py when ENABLE_WS_RELAY=true. Update README and WEBSOCKET_PROTOCOL docs with relay usage and environment variables. Add unit tests for the relay handshake and basic fan-out behavior. Other: minor formatting and exports, plus a small test tweak to set a single demo candidate in tests so they remain fast.

cloudflare-workers-and-pages · 2026-04-05T18:51:28Z

Deploying daq-radio with Cloudflare Pages

Latest commit:	`d2f5c75`
Status:	🚫 Build failed.

View logs

haoruizhou added 9 commits March 31, 2026 19:18

Merge pull request #58 from Western-Formula-Racing/coldstore

9eb1074

Coldstore

Add WFR DBC as private git submodule

9a49144

haoruizhou requested a review from Copilot April 3, 2026 15:25

Copilot started reviewing on behalf of haoruizhou April 3, 2026 15:26 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

haoruizhou changed the base branch from main to dev April 5, 2026 04:15

haoruizhou force-pushed the dev branch 4 times, most recently from 92a7572 to d6f7665 Compare April 5, 2026 17:44

Update SETUP_CARD.tex

53e3779

haoruizhou force-pushed the dev branch from 860b7ec to ba6872f Compare April 5, 2026 18:10

Merge branch 'dev' into post-apr3-test-fixes

d2f5c75

haoruizhou added 4 commits April 5, 2026 15:28

Update test_integration.py

4681d49

Hard code IP, Car side Docker -> Python script

68192c7

Add CAR deploy docs and clarify service notes

06c2ea0

Tweak compose and ignores

b63dc99

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Post apr3 test fixes#60

Post apr3 test fixes#60
haoruizhou wants to merge 17 commits intodevfrom
post-apr3-test-fixes

haoruizhou commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

cloudflare-workers-and-pages bot commented Apr 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	<small id="carSyncStatus" style="display:none; color:#4ade80; margin-top:4px; display:block;">Car clock: auto-synced ✓</small>
	<small id="carSyncStatus" style="display:none; color:#4ade80; margin-top:4px;">Car clock: auto-synced ✓</small>

	REMOTE_STATUS_PORT = int(os.getenv("STATUS_PORT", 8080))
	REMOTE_STATUS_PORT = int(os.getenv("CAR_STATUS_PORT", 8080))

Conversation

haoruizhou commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages bot commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying daq-radio with Cloudflare Pages

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cloudflare-workers-and-pages bot commented Apr 5, 2026 •

edited

Loading