Skip to content

[BREAKING] FIX: CLI Updates#1545

Merged
jsong468 merged 7 commits intomicrosoft:mainfrom
jsong468:cli_improvements
Mar 28, 2026
Merged

[BREAKING] FIX: CLI Updates#1545
jsong468 merged 7 commits intomicrosoft:mainfrom
jsong468:cli_improvements

Conversation

@jsong468
Copy link
Copy Markdown
Contributor

@jsong468 jsong468 commented Mar 26, 2026

Description

This PR improves the PyRIT CLI experience by making registry naming consistent, fixing bugs, removing dead code, and updating all user-facing text (help, examples, docs) to match the new naming conventions.


SQLite in-memory fix

  • SQLite in-memory option was not working with pyrit_shell because we were initializing pyrit (and in turn, memory) in the background thread when we run pyrit_shell, and the SQLAlchemy default pooling method is SingletonThreadPool for :memory: databases, which meant the background and main threads maintain their own connections to the DB. So, when the background thread created the DB tables, the main thread had no tables in its connection. Fix: Use StaticPool with check_same_thread=False when db_path == ":memory:" to prevent SQLAlchemy ProgrammingError when multiple threads access the same in-memory database.

Interrupt feature for pyrit_shell

  • KeyboardInterrupt handling in shell (pyrit_shell.py): Added except KeyboardInterrupt in do_run so Ctrl+C during a scenario run returns to the shell prompt instead of crashing the process.

Registry Naming Consistency (BREAKING name changes for running commands in CLI)
The initializer and scenario names that were shown to users when running list-initializers or list-scenarios did not match the registry names that were actually needed for running initializers or scenarios.

  • Scenario registry keys derived from class names (scenario_registry.py): Registry keys are now computed consistently using dotted module paths (e.g. garak.encoding, foundry.red_team_agent). Added collision detection that keeps the first registration and logs a warning for duplicates. Also skips registration of deprecated aliases like LeakageScenario.
  • Initializer registry keys derived from class names (initializer_registry.py): InitializerRegistry keys are now computed via class_name_to_snake_case(cls.__name__, suffix="Initializer") instead of using filenames. This changes targetstarget, scorersscorer, objective_listscenario_objective_list, etc.
  • Added registry_name field to ClassRegistryEntry (base.py): Both registries now pass the suffix-stripped name as registry_name to metadata, which is used for display headers in format_scenario_metadata and format_initializer_metadata. This provides a centralized place to access registry name.
  • Removed snake_class_name property (base.py): No longer needed since registry_name handles display naming.

Dead Code Removal

  • Removed get_default_initializer_discovery_path() (frontend_core.py): This function was hardcoded to return initializers/scenarios/, which only contained 3 of 7 initializers. All callers now use the context's initializer registry which discovers all initializers automatically.
  • Removed discovery_path parameter from list_initializers_async and print_initializers_list_async (frontend_core.py), and simplified callers in pyrit_scan.py, pyrit_shell.py, and pyrit_backend.py.
  • Removed dead code from initializer registry (initializer_registry.py): Deleted _initializer_paths dict, resolve_initializer_paths() method, and short_name/file_path parameters from _register_initializer. Removed stale file-dedup check in _process_file.

Initializer and Scenario Name Renames

  • Updated all references across CLI help text, examples, error messages, and argument definitions in pyrit_scan.py, pyrit_shell.py, _banner.py, _cli_args.py, and frontend_core.py.
  • Updated example config file .pyrit_conf_example and config env file build_scripts/env_local_integration_test.

Shell Banner Updates (_banner.py)

  • Added list-targets and clear commands to the banner command list.

Tests and Documentation

New Tests

  • StaticPool test (test_sqlite_memory.py): Verifies that in-memory SQLiteMemory engines use StaticPool
  • KeyboardInterrupt test (test_pyrit_shell.py): test_do_run_keyboard_interrupt_returns_to_shell — verifies Ctrl+C prints an interrupted message and does not add the run to history.

Updated Tests

  • All ClassRegistryEntry constructions in test_frontend_core.py now include the registry_name field to match the updated dataclass.
  • Removed TestGetDefaultInitializerDiscoveryPath test class and test_list_initializers_with_discovery_path test — the function and parameter they tested no longer exist.
  • Removed test_do_list_initializers_with_path from test_pyrit_shell.py and simplified test_do_list_initializersdiscovery_path parameter was removed.
  • Removed get_default_initializer_discovery_path mocking from test_pyrit_scan.py (test_main_list_initializers, test_main_list_initializers_integration).
  • Updated initializer name references (targetstarget, scorersscorer) in test_frontend_core.py parse tests and test_scenarios.py.

Documentation Updates

  • doc/code/front_end/1_pyrit_scan.ipynb / 1_pyrit_scan.py: All examples updated to use proper scenario names and initializer names.
  • doc/code/front_end/2_pyrit_shell.md: All shell examples updated; added list-targets command to command table; removed --database override from examples.
  • doc/code/registry/1_class_registry.ipynb / 1_class_registry.py: Registry usage examples updated from dotted names (e.g. garak.encoding) to short names (e.g. encoding); outputs refreshed.
  • doc/getting_started/pyrit_conf.md: Initializer reference table and config examples updated (targetstarget, scorersscorer).

@jsong468 jsong468 changed the title Cli improvements FIX: CLI Updates Mar 27, 2026
@jsong468 jsong468 changed the title FIX: CLI Updates [BREAKING] FIX: CLI Updates Mar 27, 2026
@jsong468 jsong468 marked this pull request as ready for review March 27, 2026 18:26
@jsong468 jsong468 merged commit 8741624 into microsoft:main Mar 28, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants