Skip to content

fix[evals] remove braintrust api key requirement#2145

Open
chromiebot wants to merge 2 commits into
browserbase:mainfrom
chromiebot:chromie/fix-a-braintrust-api-key-should-no
Open

fix[evals] remove braintrust api key requirement#2145
chromiebot wants to merge 2 commits into
browserbase:mainfrom
chromiebot:chromie/fix-a-braintrust-api-key-should-no

Conversation

@chromiebot
Copy link
Copy Markdown
Contributor

@chromiebot chromiebot commented May 19, 2026

why

what changed

test plan


Summary by cubic

Make Braintrust optional for evals. Evals run without a BRAINTRUST_API_KEY and skip sending logs, removing prior errors/warnings; behavior is unchanged when a key is set.

  • Bug Fixes
    • Added hasBraintrustApiKey() and made tracedSpan() a passthrough when no key.
    • Runner now passes noSendLogs: true to Eval and skips flush() without a key; sends logs and flushes when present.
    • AISdkClientWrapped uses the plain ai SDK unless braintrust can wrap it (key present), avoiding “Please specify an api key” and flush warnings.
    • Added tests covering both key-present and key-absent paths.

Written for commit 0a850a0. Summary will update on new commits. Review in cubic

claude added 2 commits May 19, 2026 00:53
Verify that:
- hasBraintrustApiKey() reflects env correctly
- tracedSpan() is a passthrough when no API key is set
- AISdkClientWrapped is importable without BRAINTRUST_API_KEY
- runner passes noSendLogs to Eval and skips flush when key is absent
- runner sends logs and calls flush when key is present
When BRAINTRUST_API_KEY is not set:
- braintrust.ts: tracedSpan() becomes a passthrough (no Braintrust import)
- runner.ts: passes noSendLogs: true to Eval() and skips flush()
- AISdkClientWrapped.ts: uses plain ai SDK instead of wrapAISDK()

This eliminates the 'Please specify an api key' error and the
'Encountered error when constructing records to flush' warnings
that occurred when running evals without a Braintrust API key.

When BRAINTRUST_API_KEY IS set, behavior is unchanged — logs are
sent and flushed as before.
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 19, 2026

⚠️ No Changeset found

Latest commit: 0a850a0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions
Copy link
Copy Markdown
Contributor

This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run.
Approving the latest commit mirrors it into an internal PR owned by the approver.
If new commits are pushed later, the internal PR stays open but is marked stale until someone approves the latest external commit and refreshes it.

@github-actions github-actions Bot added external-contributor Tracks PRs mirrored from external contributor forks. external-contributor:awaiting-approval Waiting for a stagehand team member to approve the latest external commit. labels May 19, 2026
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Runner as Evals Runner
    participant Braintrust as Braintrust Module
    participant AIWrapped as AISdkClientWrapped
    participant AISDK as AI SDK
    participant BraintrustAPI as Braintrust API

    Note over Runner,BraintrustAPI: Runtime flow based on BRAINTRUST_API_KEY env var

    alt BRAINTRUST_API_KEY is set
        Runner->>Braintrust: hasBraintrustApiKey() → true
        Braintrust-->>Runner: true
        
        Runner->>Braintrust: loadBraintrust()
        Braintrust-->>Runner: Eval, flush
        
        Runner->>Runner: sendLogs = true
        Runner->>Runner: noSendLogs not set
        
        Runner->>AIWrapped: loadWrappedAISDK()
        AIWrapped->>Braintrust: import("braintrust")
        AIWrapped->>Braintrust: wrapAISDK(ai)
        Braintrust-->>AIWrapped: wrapped AI instance
        AIWrapped-->>Runner: wrapped AI
        
        Runner->>Braintrust: Eval(..., { noSendLogs: false })
        Braintrust->>BraintrustAPI: Send logs
        BraintrustAPI-->>Braintrust: Ack
        
        Runner->>Braintrust: flush()
        Braintrust->>BraintrustAPI: Flush pending logs
        BraintrustAPI-->>Braintrust: Complete
    else BRAINTRUST_API_KEY is not set
        Runner->>Braintrust: hasBraintrustApiKey() → false
        Braintrust-->>Runner: false
        
        Runner->>Runner: sendLogs = false
        Runner->>Runner: noSendLogs = true
        
        Runner->>AIWrapped: loadWrappedAISDK()
        alt Key absent
            AIWrapped->>AIWrapped: Return ai SDK directly (passthrough)
            AIWrapped-->>Runner: plain AI SDK
        end
        
        Runner->>Braintrust: Eval(..., { noSendLogs: true })
        Note over Runner,Braintrust: No logs sent to Braintrust
        Braintrust-->>Runner: Result
        
        Note over Runner: flush() is NOT called
        
        alt tracedSpan called
            Runner->>Braintrust: tracedSpan(fn, options)
            alt No API key
                Braintrust->>Braintrust: Call fn() directly (passthrough)
                Braintrust-->>Runner: Result from fn
            end
        end
    end
Loading

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contributor:awaiting-approval Waiting for a stagehand team member to approve the latest external commit. external-contributor Tracks PRs mirrored from external contributor forks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants