Add run meta stats to artifacts#191
Conversation
azchin
left a comment
There was a problem hiding this comment.
Thanks, could change the meta.json format to something like the snippet below? There's additional implementation change requests commented.
Also please sign your commits and use Conventional Commits for commit messages.
{
"totals": {
"artifacts": {
"povs": 2,
"seeds": 4,
"patches": 1,
"bug_candidates": 2
},
"llm": {
"credits_used": 1.65
},
"sidecar": {
"patch_builds": 4,
"patch_tests": 2,
"pov_runs": 8
}
},
"crs": {
"crs_a": {
"artifacts": {
"povs": 2,
"seeds": 3,
"patches": 1,
"bug_candidates": 0
},
"llm": {
"credits_used": 1.25
},
"sidecar": {
"patch_builds": 4,
"patch_tests": 2,
"pov_runs": 7
}
}
}
}| continue | ||
| return round(total, 6) | ||
|
|
||
| def _count_build_requests_from_service_logs(self, services_dir: Path) -> int: |
There was a problem hiding this comment.
I also don't like the scrape of the logs for counting build requests. Please modify libCRS itself to log more systematically to a file, similar to my feedback for counting LiteLLM requests.
Also, please count the number of requests for run-pov and apply-patch-test as well. And do it per-CRS.
|
schema update to nested totals + per-CRS (patches, bug_candidates), structured JSONL metrics replacing log scraping, LiteLLM spend polling via /global/spend/report, and artifact counting centralized in WorkDir are all clean and cohesive enough to ship as one PR. CI green, no split needed. |
Summary
Added run-level statistics capture and persistence so each run writes a meta.json under its run directory, then exposed that metadata through the artifacts command output.
This was done to satisfy the issue requirement to track and retrieve:
User Impact
CLI/API/config/docs impact and migration:
Release Note / Changelog
Deprecation or breaking behavior:
Validation
Checklist
Closes #80