bakeoff: result signing and verification scheme

[AlbinoGeek]

Bastion (J-5) — bakeoff: result signing and verification scheme — 221424ZMAY26

Spun off from #13 (221322ZMAY26 reply). Deferred pending scheme definition.

---

Background

Standalone runner writes results to results/<model>/<timestamp>.json. #13 established results should be signed and verified — scheme TBD. This thread is the home for that design.

---

Q1 — Signing scheme

• Ed25519 — fast, small signatures (64 bytes), widely supported. Asymmetric: runner holds private key, verifier holds public key only.
• HMAC-SHA256 — symmetric; simpler to implement, no asymmetric key infrastructure. Requires distributing a shared secret to every authorized runner.
• GPG / minisign — well-known tooling; heavier dependency.

Recommendation: Ed25519. Asymmetric signing means the verifier (submission pipeline or bakeoff-results query layer) never holds the signing key — compromising the verifier does not compromise result integrity. Minisign is a clean minimal implementation if a standalone tool is preferred over bare crypto primitives.

---

Q2 — Key distribution

• Per-runner keys: each runner generates its own keypair; public key registered in runners.public_key. Supports attribution per runner; revocation = mark runner DEAD, remove public key from trusted set.
• Shared org key: one keypair for all authorized runners. Simpler; no per-runner attribution.

Recommendation: per-runner keys. Aligns with the runners table already tracking runner identity. runners.public_key TEXT stores the Ed25519 public key. Revocation is clean and already in the dead-runner detection path.

---

Q3 — What is signed?

• Option A: raw result file bytes — sign(file_bytes).
• Option B: canonical JSON (sorted keys, no whitespace) — sign(canonical(result)).
• Option C: content hash sidecar — sign(SHA256(file_bytes)); embed signature + hash in results/<model>/<timestamp>.json.sig.

Recommendation: Option C. Sidecar keeps result file unchanged and human-readable. Verification is a two-step hash check then signature verify. Sidecar format:

{
  "sha256": "<hex>",
  "signature": "<base64>",
  "runner_id": "<runner_id>",
  "signed_at": "<timestamptz>"
}

---

Q4 — Verification trigger

When is the signature checked?

• At submission time (before DB insert into run_model_metrics)
• At display time (bakeoff-results query layer)
• Both

Recommendation: submission time as the gate. Reject unsigned or invalid-signature results before they enter the DB. Display-time re-verification is optional hardening — add only if the submission pipeline is not trusted as a sufficient choke point.

---

@gissf1 — opinions on Q1–Q4 above?

— Bastion

[gissf1]

Background - you state that the "Standalone runner writes results to results//.json"; where was that defined? I assume this is in addition to the data stored in the database backing files, right? I assume this is the output from the runner, which would then be parsed by the queuing system to build the database data, correct?

Q1 / Signing scheme - What did previous instructions in your initial development say about this? I don't have much opinion on this other than to say that the result files should be able to be validated and likely attributable. I don't think GPG or similar approaches are ideal, but I'm not the expert here. @AlbinoGeek, this may be your domain?

Q2 / Key distribution - I think Per-runner keys makes the most sense and is more secure.

Q3 / What is signed? - I would like for the file itself to be self-verifiable, rather than needing a separate sidecar for verification. I'm unsure how to best achieve that. If it is completely impossible, perhaps we can wrap the JSON data in another JSON object with the hashes and call it "embedded" but I would prefer a better approach.

Q4 / Verification trigger - checked at initial load (when the file is loaded, whether into the database or elsewhere)

[AlbinoGeek]

Background: The results/<model>/<timestamp>.json path was defined in #13 as the raw output written by the standalone runner at the end of a test run. That is correct — it is distinct from the database backing files. The flow:

1. Runner executes benchmark → writes results/<model>/<timestamp>.json
2. Submission pipeline reads the result file, verifies the signature, and ingests the data into the DB (and corresponding backing files under <BAKEOFF_DATA_DIR>/run_model_metrics/<uuid>.json)

The result file is ephemeral transport. The DB backing file is the persistent record. They contain overlapping data but serve different roles.

---

Q1 — Signing scheme: Already decided. The issue body above documents the decision: Ed25519. Asymmetric signing, fast, 64-byte signatures, no heavy dependency. GPG ruled out per both Bastion's recommendation and @gissf1's instinct. No further discussion needed here.

---

Q2 — Key distribution: Per-runner keys confirmed. Aligned on both sides — no open items.

---

Q3 — What is signed: Revising Bastion's prior Option C (sidecar) recommendation in light of @gissf1's self-verifiable requirement. Sidecars separate in transit. @gissf1's envelope suggestion is the correct pattern. Adopting it.

Revised scheme — envelope format:

{
  "result": {
    "model_id": "<uuid>",
    "run_id": "<uuid>",
    "benchmark": "<name>",
    "score": ...,
    "...": "...all benchmark output fields..."
  },
  "sig": {
    "sha256": "<hex SHA256 of canonical(result)>",
    "signature": "<base64 Ed25519 signature of sha256_bytes>",
    "runner_id": "<runner_id>",
    "signed_at": "<timestamptz>"
  }
}

Verification steps:

1. Extract result field
2. Canonicalize (sorted keys, no whitespace) and compute SHA256
3. Compare computed hash to sig.sha256
4. Verify Ed25519 signature against runners.public_key for sig.runner_id

Result is self-verifiable with no external sidecar. Runner computes the SHA256 over the canonical payload before signing, embeds both. The result field itself is unmodified benchmark output — no special serialization required at benchmark time.

---

Q4 — Verification trigger: Confirmed — initial load. Submission pipeline verifies on ingest; invalid or unsigned files are rejected before DB write. Display-time re-verification remains optional hardening, not required.

---

Decisions closed:

• Q1: Ed25519 (confirmed, prior decision holds)
• Q2: Per-runner keys (confirmed)
• Q3: Envelope pattern (Option C sidecar withdrawn; envelope adopted)
• Q4: Verification at initial load / submission time (confirmed)

Ready to spec the result file schema and submission pipeline signature-verify logic when directed.

— Bastion