fix[evals] remove braintrust api key requirement#2145
Conversation
Verify that: - hasBraintrustApiKey() reflects env correctly - tracedSpan() is a passthrough when no API key is set - AISdkClientWrapped is importable without BRAINTRUST_API_KEY - runner passes noSendLogs to Eval and skips flush when key is absent - runner sends logs and calls flush when key is present
When BRAINTRUST_API_KEY is not set: - braintrust.ts: tracedSpan() becomes a passthrough (no Braintrust import) - runner.ts: passes noSendLogs: true to Eval() and skips flush() - AISdkClientWrapped.ts: uses plain ai SDK instead of wrapAISDK() This eliminates the 'Please specify an api key' error and the 'Encountered error when constructing records to flush' warnings that occurred when running evals without a Braintrust API key. When BRAINTRUST_API_KEY IS set, behavior is unchanged — logs are sent and flushed as before.
|
|
This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run. |
There was a problem hiding this comment.
No issues found across 6 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant Runner as Evals Runner
participant Braintrust as Braintrust Module
participant AIWrapped as AISdkClientWrapped
participant AISDK as AI SDK
participant BraintrustAPI as Braintrust API
Note over Runner,BraintrustAPI: Runtime flow based on BRAINTRUST_API_KEY env var
alt BRAINTRUST_API_KEY is set
Runner->>Braintrust: hasBraintrustApiKey() → true
Braintrust-->>Runner: true
Runner->>Braintrust: loadBraintrust()
Braintrust-->>Runner: Eval, flush
Runner->>Runner: sendLogs = true
Runner->>Runner: noSendLogs not set
Runner->>AIWrapped: loadWrappedAISDK()
AIWrapped->>Braintrust: import("braintrust")
AIWrapped->>Braintrust: wrapAISDK(ai)
Braintrust-->>AIWrapped: wrapped AI instance
AIWrapped-->>Runner: wrapped AI
Runner->>Braintrust: Eval(..., { noSendLogs: false })
Braintrust->>BraintrustAPI: Send logs
BraintrustAPI-->>Braintrust: Ack
Runner->>Braintrust: flush()
Braintrust->>BraintrustAPI: Flush pending logs
BraintrustAPI-->>Braintrust: Complete
else BRAINTRUST_API_KEY is not set
Runner->>Braintrust: hasBraintrustApiKey() → false
Braintrust-->>Runner: false
Runner->>Runner: sendLogs = false
Runner->>Runner: noSendLogs = true
Runner->>AIWrapped: loadWrappedAISDK()
alt Key absent
AIWrapped->>AIWrapped: Return ai SDK directly (passthrough)
AIWrapped-->>Runner: plain AI SDK
end
Runner->>Braintrust: Eval(..., { noSendLogs: true })
Note over Runner,Braintrust: No logs sent to Braintrust
Braintrust-->>Runner: Result
Note over Runner: flush() is NOT called
alt tracedSpan called
Runner->>Braintrust: tracedSpan(fn, options)
alt No API key
Braintrust->>Braintrust: Call fn() directly (passthrough)
Braintrust-->>Runner: Result from fn
end
end
end
why
what changed
test plan
Summary by cubic
Make Braintrust optional for evals. Evals run without a BRAINTRUST_API_KEY and skip sending logs, removing prior errors/warnings; behavior is unchanged when a key is set.
AISdkClientWrappeduses the plainaiSDK unlessbraintrustcan wrap it (key present), avoiding “Please specify an api key” and flush warnings.Written for commit 0a850a0. Summary will update on new commits. Review in cubic