Skip to content

FEAT Add supports_multi_turn property to targets and adapt attacks accordingly#1433

Open
romanlutz wants to merge 30 commits intoAzure:mainfrom
romanlutz:romanlutz/supports-multi-turn
Open

FEAT Add supports_multi_turn property to targets and adapt attacks accordingly#1433
romanlutz wants to merge 30 commits intoAzure:mainfrom
romanlutz:romanlutz/supports-multi-turn

Conversation

@romanlutz
Copy link
Contributor

Problem

Some targets (e.g., OpenAIImageTarget, OpenAIVideoTarget) are fundamentally single-turn — they process one prompt at a time and don't use conversation
history. However, multi-turn attacks like RedTeamingAttack reuse the same conversation_id across turns, which causes failures when targets validate that
no prior messages exist in a conversation.

There was no formal mechanism for targets to declare single vs. multi-turn support, and no way for attacks to adapt their behavior accordingly.

Solution

  1. Target property — Add a supports_multi_turn property to the target hierarchy:
  • PromptTarget → False (default for stateless targets)
  • PromptChatTarget → True (chat targets maintain conversation state)
  • Explicit False overrides on single-turn OpenAI targets: OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, OpenAICompletionTarget
  • Explicit True overrides on stateful non-chat targets: RealtimeTarget, PlaywrightTarget, PlaywrightCopilotTarget, WebSocketCopilotTarget
  • Conversation-length validation in OpenAIImageTarget and OpenAIVideoTarget _validate_request as a safety net
  1. Attack adaptations:
  • RedTeamingAttack: Rotates conversation_id before each turn for single-turn targets, tracking prior conversations as ConversationType.PRUNED in
    related_conversations
  • TAP: Skips conversation history duplication for single-turn targets, creating fresh conversations per tree node
  • CrescendoAttack, ChunkedRequestAttack, MultiPromptSendingAttack: Raise ValueError in _setup_async when used with single-turn targets (fundamentally
    incompatible — these attacks rely on building up conversation context)
  1. Rotation helper — _rotate_conversation_for_single_turn_target on MultiTurnAttackStrategy base class:
  • No-op for multi-turn targets and on the first turn (executed_turns == 0)
  • Generates a new conversation_id and records the old one as ConversationType.PRUNED

Testing

  • 9 unit tests for supports_multi_turn property values across the target hierarchy
  • 7 unit tests for attack behaviors (rotation helper no-ops, rotation activation, ValueError guards)
  • All 4649 existing unit tests continue to pass
  • End-to-end verified: RedTeamingAttack with OpenAIImageTarget ran 2 turns successfully with conversation rotation

Related

romanlutz and others added 2 commits March 2, 2026 12:59
…rgets

- Add supports_multi_turn property to PromptTarget (False) and PromptChatTarget (True)
- Override to True for stateful non-chat targets (Realtime, Playwright, WebSocket)
- Override to False for single-turn OpenAI targets (Image, Video) with _validate_request checks
- Add _rotate_conversation_for_single_turn_target helper in MultiTurnAttackStrategy
- Integrate rotation in RedTeamingAttack before sending to objective target
- Adapt TAP duplicate() to skip history duplication for single-turn targets
- Add ValueError guards in Crescendo, ChunkedRequest, MultiPromptSending for single-turn targets
- Add unit tests for property values and attack behaviors

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…I targets

OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, and OpenAICompletionTarget
now explicitly return False from supports_multi_turn, overriding the True inherited
from PromptChatTarget via OpenAITarget. This ensures the rotation helper activates
immediately, without waiting for PR 1419 to change the base class.

Also fixes test assertions to match the corrected property values.

Verified end-to-end: RedTeamingAttack with OpenAIImageTarget runs successfully
with conversation rotation across 2 turns.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 2, 2026 21:45
romanlutz and others added 2 commits March 2, 2026 13:46
…rgets

- Add supports_multi_turn property to PromptTarget (False) and PromptChatTarget (True)
- Override to True for stateful non-chat targets (Realtime, Playwright, WebSocket)
- Override to False for single-turn OpenAI targets (Image, Video) with _validate_request checks
- Add _rotate_conversation_for_single_turn_target helper in MultiTurnAttackStrategy
- Integrate rotation in RedTeamingAttack before sending to objective target
- Adapt TAP duplicate() to skip history duplication for single-turn targets
- Add ValueError guards in Crescendo, ChunkedRequest, MultiPromptSending for single-turn targets
- Add unit tests for property values and attack behaviors

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…I targets

OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, and OpenAICompletionTarget
now explicitly return False from supports_multi_turn, overriding the True inherited
from PromptChatTarget via OpenAITarget. This ensures the rotation helper activates
immediately, without waiting for PR 1419 to change the base class.

Also fixes test assertions to match the corrected property values.

Verified end-to-end: RedTeamingAttack with OpenAIImageTarget runs successfully
with conversation rotation across 2 turns.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a first-class capability flag (supports_multi_turn) on prompt targets so multi-turn attacks can adapt their conversation handling when interacting with fundamentally single-turn targets (e.g., image/video/TTS/completions).

Changes:

  • Add supports_multi_turn to the prompt target hierarchy (default False, True for chat targets, with explicit overrides for specific targets).
  • Adapt multi-turn attacks to rotate or avoid conversation history for single-turn targets, and add guards for attacks that require multi-turn state.
  • Add unit tests covering target capability values and attack behavior/guards.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/unit/target/test_supports_multi_turn.py Verifies supports_multi_turn values across target classes.
tests/unit/executor/attack/multi_turn/test_supports_multi_turn_attacks.py Tests conversation rotation helper and single-turn incompatibility guards.
pyrit/prompt_target/common/prompt_target.py Adds supports_multi_turn default property on base target.
pyrit/prompt_target/common/prompt_chat_target.py Overrides supports_multi_turn=True for chat targets.
pyrit/prompt_target/openai/openai_image_target.py Marks image target single-turn; adds conversation-length safety check.
pyrit/prompt_target/openai/openai_video_target.py Marks video target single-turn; adds conversation-length safety check.
pyrit/prompt_target/openai/openai_tts_target.py Marks TTS target single-turn.
pyrit/prompt_target/openai/openai_completion_target.py Marks completions target single-turn.
pyrit/prompt_target/openai/openai_realtime_target.py Marks realtime target as multi-turn capable.
pyrit/prompt_target/playwright_target.py Marks Playwright target as multi-turn capable.
pyrit/prompt_target/playwright_copilot_target.py Marks Playwright Copilot target as multi-turn capable.
pyrit/prompt_target/websocket_copilot_target.py Marks WebSocket Copilot target as multi-turn capable.
pyrit/executor/attack/multi_turn/multi_turn_attack_strategy.py Adds _rotate_conversation_for_single_turn_target helper.
pyrit/executor/attack/multi_turn/red_teaming.py Rotates conversation_id per turn for single-turn targets.
pyrit/executor/attack/multi_turn/tree_of_attacks.py Avoids history duplication for single-turn targets (fresh conversation_id).
pyrit/executor/attack/multi_turn/multi_prompt_sending.py Raises on single-turn targets in _setup_async.
pyrit/executor/attack/multi_turn/crescendo.py Raises on single-turn targets in _setup_async.
pyrit/executor/attack/multi_turn/chunked_request.py Raises on single-turn targets in _setup_async.

Copilot AI review requested due to automatic review settings March 2, 2026 22:04
@romanlutz romanlutz force-pushed the romanlutz/supports-multi-turn branch from 9afa84a to 079751e Compare March 2, 2026 22:04
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

romanlutz and others added 2 commits March 2, 2026 14:54
…otation in ChunkedRequest

- Replace property overrides with _DEFAULT_SUPPORTS_MULTI_TURN class constants
  on all target subclasses (image, video, tts, completion, realtime, playwright,
  playwright_copilot, websocket_copilot)
- Make supports_multi_turn settable per-instance via constructor parameter,
  propagated through PromptChatTarget and OpenAITarget init chains
- Add supports_multi_turn to _create_identifier() params
- Use self._logger instead of module logger in rotation helper
- Fix video target _validate_request to use text_piece.conversation_id
- ChunkedRequest: replace ValueError guard with rotation (Crucible CTF use case)
- Update tests: add constructor override tests, remove ChunkedRequest ValueError
  test, fix PromptTarget default test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lti-turn

# Conflicts:
#	pyrit/prompt_target/openai/openai_image_target.py
#	pyrit/prompt_target/openai/openai_video_target.py
Copilot AI review requested due to automatic review settings March 2, 2026 22:56
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

romanlutz and others added 2 commits March 2, 2026 15:08
Implement rlundeen2's design feedback (comment 7) to use a TargetCapabilities
dataclass instead of individual class constants/properties:

- Add TargetCapabilities frozen dataclass in prompt_target/common/target_capabilities.py
  with supports_multi_turn field (extensible for future capabilities like
  editable_history, json_schema_support, system_message_support, etc.)
- PromptTarget: replace _DEFAULT_SUPPORTS_MULTI_TURN with _DEFAULT_CAPABILITIES,
  build per-instance capabilities from class defaults + constructor overrides
  using dataclasses.replace()
- Add capabilities property for full TargetCapabilities access
- Keep supports_multi_turn as convenience property delegating to capabilities
- Update all subclasses to use _DEFAULT_CAPABILITIES pattern
- Export TargetCapabilities from pyrit.prompt_target
- Add tests for capabilities property and constructor overrides

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…structor args

Replace individual supports_multi_turn kwargs on subclass constructors with
a TargetCapabilities object approach:

- Remove supports_multi_turn param from PromptChatTarget and OpenAITarget __init__
- PromptTarget.__init__ accepts capabilities: Optional[TargetCapabilities] for
  custom subclasses that call super().__init__() directly
- Add capabilities property setter for per-instance overrides on any target
- Update tests to use capabilities setter pattern

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 2, 2026 23:25
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

romanlutz and others added 2 commits March 2, 2026 15:42
…cendo error

- TAP duplicate(): duplicate system messages into new conversation for single-turn
  targets so prepended conversation system prompts are preserved
- Rotation helper: same fix - duplicate system messages when rotating conversation_id
  for single-turn targets instead of using bare uuid4()
- Crescendo: update error message to reflect permanent incompatibility with
  single-turn targets (not 'does not yet support')
- Update test to match new Crescendo error message

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 2, 2026 23:49
Copilot AI review requested due to automatic review settings March 3, 2026 00:32
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 25 changed files in this pull request and generated 5 comments.

romanlutz and others added 2 commits March 2, 2026 20:02
…getCapabilities to api.rst

When no API key is provided (via parameter or environment variable),
AzureContentFilterScorer now automatically falls back to Entra ID
authentication using get_azure_token_provider with the cognitive
services scope. This matches the pattern used in OpenAITarget.

Also adds TargetCapabilities to doc/api.rst and simplifies the video
notebook to use bare AzureContentFilterScorer() since auth is now
auto-detected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 3, 2026 04:47
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 27 changed files in this pull request and generated 4 comments.

romanlutz and others added 2 commits March 2, 2026 21:02
Treat 'error' data type messages as text in _is_text_message_format,
_build_chat_messages_for_text, and _build_chat_messages_for_multi_modal_async.
This prevents ValueError when conversation history contains error responses
(e.g., from content filter blocks) and a subsequent turn is sent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines +790 to +793
if system_messages:
new_id, pieces = self._memory.duplicate_messages(messages=system_messages)
self._memory.add_message_pieces_to_memory(message_pieces=pieces)
duplicate_node.objective_target_conversation_id = new_id
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO @romanlutz to check correctness

…manlutz/PyRIT into romanlutz/supports-multi-turn

# Conflicts:
#	pyrit/executor/attack/multi_turn/crescendo.py
#	pyrit/executor/attack/multi_turn/multi_turn_attack_strategy.py
#	pyrit/executor/attack/multi_turn/tree_of_attacks.py
#	pyrit/prompt_target/common/prompt_target.py
#	pyrit/prompt_target/openai/openai_completion_target.py
#	pyrit/prompt_target/openai/openai_image_target.py
#	pyrit/prompt_target/openai/openai_tts_target.py
#	pyrit/prompt_target/openai/openai_video_target.py
#	tests/unit/executor/attack/multi_turn/test_supports_multi_turn_attacks.py
#	tests/unit/target/test_supports_multi_turn.py
Copilot AI review requested due to automatic review settings March 3, 2026 05:29
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 8 comments.

romanlutz and others added 4 commits March 3, 2026 04:43
…ts_conversation_history

- Add conversation rotation for single-turn targets in ChunkedRequestAttack
  so each chunk gets a fresh conversation_id (instead of rejecting them).
- Remove supports_conversation_history from target identifier params since
  it is never consumed and misrepresents stateful non-chat targets.
- Rerun chunked_request_attack notebook.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add capabilities parameter to PromptChatTarget, OpenAITarget,
PlaywrightTarget, WebSocketCopilotTarget, and PlaywrightCopilotTarget
constructors so users can override TargetCapabilities at construction
time instead of only via the setter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lti-turn

# Conflicts:
#	doc/code/executor/attack/2_red_teaming_attack.ipynb
#	doc/code/executor/attack/chunked_request_attack.ipynb
#	doc/code/targets/3_openai_image_target.ipynb
…' into romanlutz/supports-multi-turn

# Conflicts:
#	doc/code/targets/4_openai_video_target.ipynb
romanlutz and others added 2 commits March 3, 2026 15:13
All 5 subclass overrides (PromptChatTarget, PlaywrightTarget,
PlaywrightCopilotTarget, WebSocketCopilotTarget, RealtimeTarget)
hard-coded True, bypassing the TargetCapabilities system. Now all
targets delegate to the base PromptTarget.supports_multi_turn which
reads from self._capabilities set via _DEFAULT_CAPABILITIES or
constructor override.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rlundeen2 rlundeen2 self-assigned this Mar 3, 2026
romanlutz and others added 2 commits March 3, 2026 15:59
- Move get_azure_openai_auth import to top-level in openai_target.py
- Make capabilities immutable (remove setter from PromptTarget)
- Update error message in openai_chat_target.py to include error data type
- Reject async token providers in AzureContentFilterScorer
- Remove unused patch_central_database fixture params from tests
- Regenerate crescendo notebook

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 4, 2026 00:12
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 32 changed files in this pull request and generated 4 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +114 to +118
endpoint="https://mock.azure.com/",
api_key="mock-api-key",
capabilities=TargetCapabilities(supports_multi_turn=True),
)
assert target.supports_multi_turn is True
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenAIImageTarget enforces single-turn behavior in _validate_request (it errors if prior messages exist), so overriding its capabilities to supports_multi_turn=True creates a self-contradictory configuration: attacks will treat it as multi-turn but the target will still reject turn > 1. Consider using a target where multi-turn support is actually configurable (e.g., Playwright/WebSocket) or asserting that overriding doesn’t bypass single-turn validation.

Copilot uses AI. Check for mistakes.
Comment on lines +270 to +272
# For single-turn targets, rotate conversation_id so each chunk gets a fresh conversation
self._rotate_conversation_for_single_turn_target(context=context)

Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_setup_async now raises for single-turn targets, so the per-chunk call to _rotate_conversation_for_single_turn_target() is either unreachable (for truly single-turn targets) or a no-op (for multi-turn targets). This comment/call site is misleading; consider removing the rotation call here or updating the comment to reflect that single-turn targets are rejected in setup.

Suggested change
# For single-turn targets, rotate conversation_id so each chunk gets a fresh conversation
self._rotate_conversation_for_single_turn_target(context=context)

Copilot uses AI. Check for mistakes.
Comment on lines +232 to +236
if not self._objective_target.supports_multi_turn:
raise ValueError(
"ChunkedRequestAttack requires a multi-turn target. "
"The objective target does not support multi-turn conversations."
)
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR introduces a new behavior gate for single-turn targets (ChunkedRequestAttack now raises in _setup_async when supports_multi_turn is false), but there doesn’t appear to be a unit test asserting this guard. Adding a targeted test would prevent regressions and also clarify whether the intended behavior is “hard error” vs “rotate conversation per chunk.”

Copilot uses AI. Check for mistakes.
Comment on lines +165 to +166
Returns:
bool: False by default. Subclasses that support multi-turn should override.
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The supports_multi_turn property docstring says “Subclasses that support multi-turn should override”, but this PR implements multi-turn support via the _DEFAULT_CAPABILITIES / capabilities mechanism (without overriding the property). Updating the docstring to reflect the new pattern will reduce confusion for target authors.

Suggested change
Returns:
bool: False by default. Subclasses that support multi-turn should override.
Subclasses should configure multi-turn support via the ``_DEFAULT_CAPABILITIES``
class attribute or by passing a ``TargetCapabilities`` instance to the
constructor, rather than overriding this property.
Returns:
bool: True if this target supports multi-turn conversations.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants