Skip to content

Add run meta stats to artifacts#191

Open
tusharshah21 wants to merge 6 commits into
ossf:mainfrom
tusharshah21:feat/run-meta-stats
Open

Add run meta stats to artifacts#191
tusharshah21 wants to merge 6 commits into
ossf:mainfrom
tusharshah21:feat/run-meta-stats

Conversation

@tusharshah21
Copy link
Copy Markdown
Contributor

Summary

Added run-level statistics capture and persistence so each run writes a meta.json under its run directory, then exposed that metadata through the artifacts command output.

This was done to satisfy the issue requirement to track and retrieve:

  • LLM credits used
  • POVs found
  • Seeds shared
  • Number of builds requested through libCRS

User Impact

  • User-facing change
  • Internal-only change

CLI/API/config/docs impact and migration:

  • CLI output change: artifacts JSON now includes a top-level meta object when resolving a run.
  • No new required flags and no command rename/removal.
  • Existing consumers that ignore unknown JSON fields continue to work.
  • Consumers with strict JSON schema validation should allow the new optional meta field.

Release Note / Changelog

  • I updated CHANGELOG.md ([Unreleased]) for user-facing changes
  • No changelog entry needed (internal-only refactor/test/chore)

Deprecation or breaking behavior:

  • None.
  • No replacement path needed.
  • No planned removal window.

Validation

  • Added/updated unit test coverage for artifacts meta output behavior.
  • Ran static diagnostics on changed files with no reported errors.
  • Attempted targeted pytest run for the updated artifacts test module, but local execution in this Windows environment is blocked during collection by a Linux-only fcntl import dependency.

Checklist

  • I followed Conventional Commits
  • I updated docs for behavior/config/CLI changes
  • I added/updated tests for behavior changes
  • I considered backward compatibility and migration impact

Closes #80

Copy link
Copy Markdown
Collaborator

@azchin azchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, could change the meta.json format to something like the snippet below? There's additional implementation change requests commented.

Also please sign your commits and use Conventional Commits for commit messages.

{
  "totals": {
    "artifacts": {
      "povs": 2,
      "seeds": 4,
      "patches": 1,
      "bug_candidates": 2
    },
    "llm": {
      "credits_used": 1.65
    },
    "sidecar": {
      "patch_builds": 4,
      "patch_tests": 2,
      "pov_runs": 8
    }
  },
  "crs": {
    "crs_a": {
      "artifacts": {
        "povs": 2,
        "seeds": 3,
        "patches": 1,
        "bug_candidates": 0
      },
      "llm": {
        "credits_used": 1.25
      },
      "sidecar": {
        "patch_builds": 4,
        "patch_tests": 2,
        "pov_runs": 7
      }
    }
  }
}

Comment thread oss_crs/src/crs_compose.py Outdated
continue
return round(total, 6)

def _count_build_requests_from_service_logs(self, services_dir: Path) -> int:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also don't like the scrape of the logs for counting build requests. Please modify libCRS itself to log more systematically to a file, similar to my feedback for counting LiteLLM requests.

Also, please count the number of requests for run-pov and apply-patch-test as well. And do it per-CRS.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted

Comment thread oss_crs/src/crs_compose.py
Comment thread oss_crs/src/crs_compose.py Outdated
Comment thread oss_crs/src/crs_compose.py Outdated
@tusharshah21
Copy link
Copy Markdown
Contributor Author

schema update to nested totals + per-CRS (patches, bug_candidates), structured JSONL metrics replacing log scraping, LiteLLM spend polling via /global/spend/report, and artifact counting centralized in WorkDir are all clean and cohesive enough to ship as one PR. CI green, no split needed.
Let me know if there are any other suggestion!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feat: Run Session Statistics

2 participants