bakeoff: model descriptor file format and capability ingestion

[AlbinoGeek]

Bastion (J-5) — bakeoff: model descriptor file format and capability ingestion — 221424ZMAY26

Spun off from #13 (221322ZMAY26 reply). Two coupled open questions that belong in their own thread.

---

Background

#13 established:

• Standalone runner reads a JSON model descriptor at startup — no DB required
• DB models table is populated from the same descriptor at submission time
• Descriptor is canonical input; DB is derived

Confirmed fields (#13 §5):

{
  "model_id": 42,
  "name": "llama3-8b",
  "architecture": "Dense",
  "parameter_count_b": 8.0,
  "active_parameter_count_b": 8.0,
  "quantization": "q4_k_m",
  "sources": [
    { "type": "huggingface", "repo": "meta-llama/Meta-Llama-3-8B-Instruct" }
  ]
}

---

Q1 — Versioned schema?

Should the descriptor include a schema_version field?

{ "schema_version": 1, ... }

With version field: runner can reject or adapt descriptors from older/newer tooling. Migration path without re-scanning all descriptor files.

Without: simpler now; add when first breaking change arrives. Risk: unversioned files in the wild become ambiguous once schema evolves.

Recommendation: include schema_version: 1 from day one. Cost is one field; benefit is a clean migration path before any files accumulate in the wild.

---

Q2 — Disk location

Option A — beside model weights:

/models/llama3-8b-q4km/
  llama3-8b-q4km.gguf
  model.json             ← descriptor co-located with weights

Option B — separate submission directory:

/models/llama3-8b-q4km/llama3-8b-q4km.gguf
queue/pending/llama3-8b-q4km.json         ← descriptor as submission artifact

Option A: self-contained model bundle; runner infers descriptor path from model path.

Option B: descriptor is a submission artifact, not part of the model bundle. Supports submitting the same model weights under multiple configurations (different quantization labels, priority, prompt sets) without duplicating the weights directory.

Recommendation: Option B. Models are large and often read-only or shared. Submission configuration belongs in the queue layer, not the model bundle.

---

Q3 — Capability ingestion pipeline

Where does model metadata (parameter counts, quantization, architecture) originate?

• A. Manual — submitter writes model.json by hand or from model card. Simple; error-prone for numeric fields.
• B. CI artifact — CI pipeline generates descriptor from HuggingFace metadata or model card at ingest time. Reliable; requires CI integration.
• C. Runner autodiscovery — runner parses GGUF metadata at startup, cross-checks against descriptor values. Catches descriptor drift; adds startup latency.

Recommendation: A for now (manual / tooling-assisted). Add C (GGUF metadata cross-check at startup) when runner is implemented — cheap to add, catches the common mistake of mismatched quantization label or wrong parameter count in the descriptor.

---

@gissf1 — opinions on Q1–Q3 above?

— Bastion

[gissf1]

As initially mentioned in #13:

Model descriptor file format - I agree JSON makes sense, I think versioning makes sense too, so we can import older models and other DB data as needed.

Per your suggestion, schema_version is a reasonable field name for versioning. I imagine this could be just a "schema_version: 1.0" in the file, or similar. It might make sense to use a date-based versioning scheme, but that is just one option, not a hard opinion. An example of a date-based versioning scheme would be using formatted dates like YYYYMMDDNN where YYYY is the 4 digit year, MM is the 2 digit month (with zero prefix when less than 10), and DD is the 2 digit day of the month (again, with zero prefix when less than 10), and NN is a serial index where "00" for most cases, but if there happened to be multiple releases in a day, one could increment this to keep them unique and reset it to zero with the next update on another day. The likelihood of more than 100 updates on a single day seems quite unlikely in normal development scenarios, but in the case we did encounter it, we can address it then. This format also provides some level of validation that a "known version" is used. That said using simple incrementing numbers (like 1.0 or 1) are simpler to remember. I'm open to discussing this more.

Exact location on disk will obviously depend on where we are being run from. We should not assume the model is in the same directory as the JSON model file data. in fact, I would think they should be in different directories since the model weights (the model source file) may be cached in some system-wide location whereas the model file data should be managed by us in our file-data locations. The model data will need to persist beyond the queue usage, so it should be in different directory than the run_queue files. I think it might make sense to have a "persistent data store" where we can structure the filesystem arrangement based on the intended database structure - using directories for tables, files for records, etc. Since we need to assume that the runner may not have access to the database, all data should be accessible through file exchange. I would say this is closer to option B, but not exactly. the run_queue files should not be in the same directory as the disk-backed table data files (like the model disk file). In fact, I would say that a run_queue disk file should be relatively self contained since it is a job submission request, and may need to populate other disk files (such as model disk files) as the queuing system prepares to do testing. The files on disk likely should use their UUID as the filename for the record they contain. Similarly, there may be a directory for each table (named the same as the table), under some base path. The run_queue itself is a bit of a special case here, since the data is much less static, so maybe it can be an exception. Let's discuss the disk structure in more detail for various types of files we will be using.

Q3: model metadata originates from the model_source. it is stored in the models database table, and the models disk files that back them. new models are likely going to be submitted as a URL for a known source (huggingface URL, ollama source, etc; but known and relatively trusted sites, not just random untrusted URLs). We would then have a handler than queries the source for details and determines model data/metadata and how to queue it for testing. It would also take that acquired data, validate it, and store it in the database and the model disk file backing the database entry. Specifically regarding your options:

• I don't think manual submission of model.json files is going to be acceptable, other than maybe by administrators or for our initial seeding.
• I don't think CI is the right place to do this since the data can be large and require internet access.
• I don't think we want full auto-discovery of new models, but auto-discovery and validation of the data is the goal here. We should be able to get the data from huggingface (or other sources) without pulling the full model initially. Once we do pull the model weights, we should validate the embedded metadata matches and generate the model_hash at the same time. That said, the verification of the embedded metadata is likely a more complicated task (even compared to generating the model_hash), so I'm okay with putting that embedded metadata verification off for a bit.

[AlbinoGeek]

schema_version format: Recommend simple integer increment (schema_version: 1, 2, etc.) over YYYYMMDDNN. Date-based versioning couples the schema to a calendar, adds parsing complexity, and provides no validation advantage a schema registry can't give more cleanly. YYYYMMDDNN is a good release artifact naming convention — not a format version identifier. If we need human-readable provenance in the file, add a separate schema_updated timestamp field alongside the integer version.

Disk structure: The directory-per-table, UUID-filename pattern is confirmed. Proposed base layout:

<BAKEOFF_DATA_DIR>/
  models/
    <uuid>.json          ← persistent model record
  run_queue/
    <uuid>.json          ← ephemeral job submission
  (future tables as subdirs)

BAKEOFF_DATA_DIR env-configurable, no hardcoded path. run_queue as ephemeral exception is correct — job files are self-contained submissions, lifecycle (created → consumed → archived/deleted) differs fundamentally from persistent model records. Agree they should not share a directory.

Model metadata sourcing: Agreed on all three negatives — no manual JSON submissions (except admin/seed), no CI, no full auto-discovery. The URL-submit → upstream metadata pull → validate → store (DB + disk file) path is the right architecture. Pulling metadata without weights first is essential for cost and latency; generating model_hash at weight-pull time is the correct coupling point.

Deferred embedded metadata verification is accepted. Flag it as a P2 task so it doesn't fall off the backlog.

Next gate — data contract definition: Before implementing handlers, I'd like to align on the exact field sets for the two core disk file types:

1. models/<uuid>.json — what fields does a persisted model record carry? At minimum: schema_version, uuid, source_url, source_type (huggingface/ollama/etc), model_hash (nullable until weights pulled), metadata blob from upstream, created_at, status. What else?

2. run_queue/<uuid>.json — what constitutes a "self-contained job submission"? Candidate fields: schema_version, uuid, model_uuid (or source_url if model not yet registered), requested_at, requester, status, result_ref. What's the minimum viable contract?

Proposing we define both schemas explicitly before writing handlers — avoids field drift between disk and DB representations.

— Bastion