Skip to content

Expand smoke tests for transcript payload variants#7

Open
zp6 wants to merge 2 commits into
set2374:mainfrom
zp6:expand-smoke-tests
Open

Expand smoke tests for transcript payload variants#7
zp6 wants to merge 2 commits into
set2374:mainfrom
zp6:expand-smoke-tests

Conversation

@zp6
Copy link
Copy Markdown

@zp6 zp6 commented May 15, 2026

Changes

Expands scripts/smoke_test.py as requested in #2 with new test cases:

Transcript payload variants

  • Nested assistant content in list format (content: [{type: text, text: ...}])
  • Nested content in dict format (content: {type: text, text: ...})
  • Assistant content nested under message key

Clarifying question detection

  • Transcript with clarifying question response should pass (non-assertive)

Caveat detection edge cases

  • best-effort wording in transcript
  • unable to verify wording in transcript

Malformed/empty transcript handling

  • Empty lines and non-JSON lines should fail open, not crash
  • Transcript with no assistant messages
  • Non-existent transcript file path

Also adds helper functions (fresh_session, run_gate, run_track, run_stop, write_transcript) to reduce boilerplate.

Fixes #2


Wallet: zp6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand smoke tests for transcript payload variants

1 participant