Skip to content

fix: auto-log Tinker inference costs from ART managed openai_client#620

Open
vivekkalyan wants to merge 1 commit intomainfrom
fix/auto-inference-costs
Open

fix: auto-log Tinker inference costs from ART managed openai_client#620
vivekkalyan wants to merge 1 commit intomainfrom
fix/auto-inference-costs

Conversation

@vivekkalyan
Copy link
Collaborator

@vivekkalyan vivekkalyan commented Mar 18, 2026

Summary

  • auto-log costs/{train,eval}/tinker_prefill and costs/{train,eval}/tinker_sample for calls made through ART's managed model.openai_client()
  • enable the same pricing-backed cost calculator setup in LocalBackend.register() when model pricing exists
  • add focused unit coverage for train, eval, no-context, and multi-choice inference cost logging

Why

Internal repo's PipelineTrainer path was using ART-managed inference but was not emitting Tinker rollout costs unless the codebase manually called model.cost_calculator(...). This moves that responsibility into ART so repos do not need rollout-side plumbing.

Notes

  • this keeps the cleaner contract: ART-managed openai_client() calls auto-log costs when an ART metrics context is active
  • downstream repos that already add the same Tinker costs manually should remove that duplicate logging

Other option

  • Introduce a first-class ART inference API like model.generate(...) or model.chat_completions(...).
  • That API owns cost/timing/token accounting.
  • openai_client() stays raw.
  • Cleanest, but requires codebase migration.

Validation

  • .venv/bin/python -m pytest tests/unit/test_model_openai_client_costs.py tests/unit/test_track_api_cost.py -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant