FEAT Add supports_multi_turn property to targets and adapt attacks accordingly#1433
FEAT Add supports_multi_turn property to targets and adapt attacks accordingly#1433romanlutz wants to merge 30 commits intoAzure:mainfrom
Conversation
…rgets - Add supports_multi_turn property to PromptTarget (False) and PromptChatTarget (True) - Override to True for stateful non-chat targets (Realtime, Playwright, WebSocket) - Override to False for single-turn OpenAI targets (Image, Video) with _validate_request checks - Add _rotate_conversation_for_single_turn_target helper in MultiTurnAttackStrategy - Integrate rotation in RedTeamingAttack before sending to objective target - Adapt TAP duplicate() to skip history duplication for single-turn targets - Add ValueError guards in Crescendo, ChunkedRequest, MultiPromptSending for single-turn targets - Add unit tests for property values and attack behaviors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…I targets OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, and OpenAICompletionTarget now explicitly return False from supports_multi_turn, overriding the True inherited from PromptChatTarget via OpenAITarget. This ensures the rotation helper activates immediately, without waiting for PR 1419 to change the base class. Also fixes test assertions to match the corrected property values. Verified end-to-end: RedTeamingAttack with OpenAIImageTarget runs successfully with conversation rotation across 2 turns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rgets - Add supports_multi_turn property to PromptTarget (False) and PromptChatTarget (True) - Override to True for stateful non-chat targets (Realtime, Playwright, WebSocket) - Override to False for single-turn OpenAI targets (Image, Video) with _validate_request checks - Add _rotate_conversation_for_single_turn_target helper in MultiTurnAttackStrategy - Integrate rotation in RedTeamingAttack before sending to objective target - Adapt TAP duplicate() to skip history duplication for single-turn targets - Add ValueError guards in Crescendo, ChunkedRequest, MultiPromptSending for single-turn targets - Add unit tests for property values and attack behaviors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…I targets OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, and OpenAICompletionTarget now explicitly return False from supports_multi_turn, overriding the True inherited from PromptChatTarget via OpenAITarget. This ensures the rotation helper activates immediately, without waiting for PR 1419 to change the base class. Also fixes test assertions to match the corrected property values. Verified end-to-end: RedTeamingAttack with OpenAIImageTarget runs successfully with conversation rotation across 2 turns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces a first-class capability flag (supports_multi_turn) on prompt targets so multi-turn attacks can adapt their conversation handling when interacting with fundamentally single-turn targets (e.g., image/video/TTS/completions).
Changes:
- Add
supports_multi_turnto the prompt target hierarchy (defaultFalse,Truefor chat targets, with explicit overrides for specific targets). - Adapt multi-turn attacks to rotate or avoid conversation history for single-turn targets, and add guards for attacks that require multi-turn state.
- Add unit tests covering target capability values and attack behavior/guards.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/target/test_supports_multi_turn.py | Verifies supports_multi_turn values across target classes. |
| tests/unit/executor/attack/multi_turn/test_supports_multi_turn_attacks.py | Tests conversation rotation helper and single-turn incompatibility guards. |
| pyrit/prompt_target/common/prompt_target.py | Adds supports_multi_turn default property on base target. |
| pyrit/prompt_target/common/prompt_chat_target.py | Overrides supports_multi_turn=True for chat targets. |
| pyrit/prompt_target/openai/openai_image_target.py | Marks image target single-turn; adds conversation-length safety check. |
| pyrit/prompt_target/openai/openai_video_target.py | Marks video target single-turn; adds conversation-length safety check. |
| pyrit/prompt_target/openai/openai_tts_target.py | Marks TTS target single-turn. |
| pyrit/prompt_target/openai/openai_completion_target.py | Marks completions target single-turn. |
| pyrit/prompt_target/openai/openai_realtime_target.py | Marks realtime target as multi-turn capable. |
| pyrit/prompt_target/playwright_target.py | Marks Playwright target as multi-turn capable. |
| pyrit/prompt_target/playwright_copilot_target.py | Marks Playwright Copilot target as multi-turn capable. |
| pyrit/prompt_target/websocket_copilot_target.py | Marks WebSocket Copilot target as multi-turn capable. |
| pyrit/executor/attack/multi_turn/multi_turn_attack_strategy.py | Adds _rotate_conversation_for_single_turn_target helper. |
| pyrit/executor/attack/multi_turn/red_teaming.py | Rotates conversation_id per turn for single-turn targets. |
| pyrit/executor/attack/multi_turn/tree_of_attacks.py | Avoids history duplication for single-turn targets (fresh conversation_id). |
| pyrit/executor/attack/multi_turn/multi_prompt_sending.py | Raises on single-turn targets in _setup_async. |
| pyrit/executor/attack/multi_turn/crescendo.py | Raises on single-turn targets in _setup_async. |
| pyrit/executor/attack/multi_turn/chunked_request.py | Raises on single-turn targets in _setup_async. |
9afa84a to
079751e
Compare
…otation in ChunkedRequest - Replace property overrides with _DEFAULT_SUPPORTS_MULTI_TURN class constants on all target subclasses (image, video, tts, completion, realtime, playwright, playwright_copilot, websocket_copilot) - Make supports_multi_turn settable per-instance via constructor parameter, propagated through PromptChatTarget and OpenAITarget init chains - Add supports_multi_turn to _create_identifier() params - Use self._logger instead of module logger in rotation helper - Fix video target _validate_request to use text_piece.conversation_id - ChunkedRequest: replace ValueError guard with rotation (Crucible CTF use case) - Update tests: add constructor override tests, remove ChunkedRequest ValueError test, fix PromptTarget default test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lti-turn # Conflicts: # pyrit/prompt_target/openai/openai_image_target.py # pyrit/prompt_target/openai/openai_video_target.py
Implement rlundeen2's design feedback (comment 7) to use a TargetCapabilities dataclass instead of individual class constants/properties: - Add TargetCapabilities frozen dataclass in prompt_target/common/target_capabilities.py with supports_multi_turn field (extensible for future capabilities like editable_history, json_schema_support, system_message_support, etc.) - PromptTarget: replace _DEFAULT_SUPPORTS_MULTI_TURN with _DEFAULT_CAPABILITIES, build per-instance capabilities from class defaults + constructor overrides using dataclasses.replace() - Add capabilities property for full TargetCapabilities access - Keep supports_multi_turn as convenience property delegating to capabilities - Update all subclasses to use _DEFAULT_CAPABILITIES pattern - Export TargetCapabilities from pyrit.prompt_target - Add tests for capabilities property and constructor overrides Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…structor args Replace individual supports_multi_turn kwargs on subclass constructors with a TargetCapabilities object approach: - Remove supports_multi_turn param from PromptChatTarget and OpenAITarget __init__ - PromptTarget.__init__ accepts capabilities: Optional[TargetCapabilities] for custom subclasses that call super().__init__() directly - Add capabilities property setter for per-instance overrides on any target - Update tests to use capabilities setter pattern Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cendo error - TAP duplicate(): duplicate system messages into new conversation for single-turn targets so prepended conversation system prompts are preserved - Rotation helper: same fix - duplicate system messages when rotating conversation_id for single-turn targets instead of using bare uuid4() - Crescendo: update error message to reflect permanent incompatibility with single-turn targets (not 'does not yet support') - Update test to match new Crescendo error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…getCapabilities to api.rst When no API key is provided (via parameter or environment variable), AzureContentFilterScorer now automatically falls back to Entra ID authentication using get_azure_token_provider with the cognitive services scope. This matches the pattern used in OpenAITarget. Also adds TargetCapabilities to doc/api.rst and simplifies the video notebook to use bare AzureContentFilterScorer() since auth is now auto-detected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Treat 'error' data type messages as text in _is_text_message_format, _build_chat_messages_for_text, and _build_chat_messages_for_multi_modal_async. This prevents ValueError when conversation history contains error responses (e.g., from content filter blocks) and a subsequent turn is sent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| if system_messages: | ||
| new_id, pieces = self._memory.duplicate_messages(messages=system_messages) | ||
| self._memory.add_message_pieces_to_memory(message_pieces=pieces) | ||
| duplicate_node.objective_target_conversation_id = new_id |
…manlutz/PyRIT into romanlutz/supports-multi-turn # Conflicts: # pyrit/executor/attack/multi_turn/crescendo.py # pyrit/executor/attack/multi_turn/multi_turn_attack_strategy.py # pyrit/executor/attack/multi_turn/tree_of_attacks.py # pyrit/prompt_target/common/prompt_target.py # pyrit/prompt_target/openai/openai_completion_target.py # pyrit/prompt_target/openai/openai_image_target.py # pyrit/prompt_target/openai/openai_tts_target.py # pyrit/prompt_target/openai/openai_video_target.py # tests/unit/executor/attack/multi_turn/test_supports_multi_turn_attacks.py # tests/unit/target/test_supports_multi_turn.py
…ts_conversation_history - Add conversation rotation for single-turn targets in ChunkedRequestAttack so each chunk gets a fresh conversation_id (instead of rejecting them). - Remove supports_conversation_history from target identifier params since it is never consumed and misrepresents stateful non-chat targets. - Rerun chunked_request_attack notebook. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add capabilities parameter to PromptChatTarget, OpenAITarget, PlaywrightTarget, WebSocketCopilotTarget, and PlaywrightCopilotTarget constructors so users can override TargetCapabilities at construction time instead of only via the setter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lti-turn # Conflicts: # doc/code/executor/attack/2_red_teaming_attack.ipynb # doc/code/executor/attack/chunked_request_attack.ipynb # doc/code/targets/3_openai_image_target.ipynb
…' into romanlutz/supports-multi-turn # Conflicts: # doc/code/targets/4_openai_video_target.ipynb
All 5 subclass overrides (PromptChatTarget, PlaywrightTarget, PlaywrightCopilotTarget, WebSocketCopilotTarget, RealtimeTarget) hard-coded True, bypassing the TargetCapabilities system. Now all targets delegate to the base PromptTarget.supports_multi_turn which reads from self._capabilities set via _DEFAULT_CAPABILITIES or constructor override. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Move get_azure_openai_auth import to top-level in openai_target.py - Make capabilities immutable (remove setter from PromptTarget) - Update error message in openai_chat_target.py to include error data type - Reject async token providers in AzureContentFilterScorer - Remove unused patch_central_database fixture params from tests - Regenerate crescendo notebook Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 32 changed files in this pull request and generated 4 comments.
You can also share your feedback on Copilot code review. Take the survey.
| endpoint="https://mock.azure.com/", | ||
| api_key="mock-api-key", | ||
| capabilities=TargetCapabilities(supports_multi_turn=True), | ||
| ) | ||
| assert target.supports_multi_turn is True |
There was a problem hiding this comment.
OpenAIImageTarget enforces single-turn behavior in _validate_request (it errors if prior messages exist), so overriding its capabilities to supports_multi_turn=True creates a self-contradictory configuration: attacks will treat it as multi-turn but the target will still reject turn > 1. Consider using a target where multi-turn support is actually configurable (e.g., Playwright/WebSocket) or asserting that overriding doesn’t bypass single-turn validation.
| # For single-turn targets, rotate conversation_id so each chunk gets a fresh conversation | ||
| self._rotate_conversation_for_single_turn_target(context=context) | ||
|
|
There was a problem hiding this comment.
_setup_async now raises for single-turn targets, so the per-chunk call to _rotate_conversation_for_single_turn_target() is either unreachable (for truly single-turn targets) or a no-op (for multi-turn targets). This comment/call site is misleading; consider removing the rotation call here or updating the comment to reflect that single-turn targets are rejected in setup.
| # For single-turn targets, rotate conversation_id so each chunk gets a fresh conversation | |
| self._rotate_conversation_for_single_turn_target(context=context) |
| if not self._objective_target.supports_multi_turn: | ||
| raise ValueError( | ||
| "ChunkedRequestAttack requires a multi-turn target. " | ||
| "The objective target does not support multi-turn conversations." | ||
| ) |
There was a problem hiding this comment.
This PR introduces a new behavior gate for single-turn targets (ChunkedRequestAttack now raises in _setup_async when supports_multi_turn is false), but there doesn’t appear to be a unit test asserting this guard. Adding a targeted test would prevent regressions and also clarify whether the intended behavior is “hard error” vs “rotate conversation per chunk.”
| Returns: | ||
| bool: False by default. Subclasses that support multi-turn should override. |
There was a problem hiding this comment.
The supports_multi_turn property docstring says “Subclasses that support multi-turn should override”, but this PR implements multi-turn support via the _DEFAULT_CAPABILITIES / capabilities mechanism (without overriding the property). Updating the docstring to reflect the new pattern will reduce confusion for target authors.
| Returns: | |
| bool: False by default. Subclasses that support multi-turn should override. | |
| Subclasses should configure multi-turn support via the ``_DEFAULT_CAPABILITIES`` | |
| class attribute or by passing a ``TargetCapabilities`` instance to the | |
| constructor, rather than overriding this property. | |
| Returns: | |
| bool: True if this target supports multi-turn conversations. |
Problem
Some targets (e.g., OpenAIImageTarget, OpenAIVideoTarget) are fundamentally single-turn — they process one prompt at a time and don't use conversation
history. However, multi-turn attacks like RedTeamingAttack reuse the same conversation_id across turns, which causes failures when targets validate that
no prior messages exist in a conversation.
There was no formal mechanism for targets to declare single vs. multi-turn support, and no way for attacks to adapt their behavior accordingly.
Solution
related_conversations
incompatible — these attacks rely on building up conversation context)
Testing
Related