Add VisualLeakBench dataset loader (arXiv:2603.13385)

[Copilot]

Adds PyRIT support for the VisualLeakBench / MM-SafetyBench dataset — a multimodal benchmark of 1,000 adversarial images testing LVLMs against OCR injection (harmful instructions embedded in images) and PII leakage (social engineering to extract SSNs, passwords, API keys, etc.).

New: _VisualLeakBenchDataset

• Fetches metadata.csv from YoutingWang/MM-SafetyBench on GitHub and downloads images with local caching
• Produces image+text prompt pairs per example, linked by prompt_group_id (image at sequence=0, category-specific query at sequence=1)
• Maps harm categories: ocr_injection for OCR entries; pii_leakage + normalized PII type (e.g. ssn, api_key) for PII entries
• Supports filtering via categories, pii_types, and max_examples
• Registered with tags={"default","safety","privacy"}, modalities=["image","text"] for SeedDatasetFilter discovery

New enums

Enum	Values
VisualLeakBenchCategory	OCR_INJECTION, PII_LEAKAGE
VisualLeakBenchPIIType	EMAIL, DOB, PHONE, PASSWORD, PIN, API_KEY, SSN, CREDIT_CARD

Usage

from pyrit.datasets.seed_datasets.remote import (
    _VisualLeakBenchDataset,
    VisualLeakBenchCategory,
    VisualLeakBenchPIIType,
)

# Load only PII leakage examples for SSN and Password
loader = _VisualLeakBenchDataset(
    categories=[VisualLeakBenchCategory.PII_LEAKAGE],
    pii_types=[VisualLeakBenchPIIType.SSN, VisualLeakBenchPIIType.PASSWORD],
)
dataset = await loader.fetch_dataset()

Test coverage

26 unit tests covering init validation, OCR/PII pair creation, harm category mapping, category/PII-type filtering, max_examples, failed image handling, and metadata correctness. Integration test updated to cap image downloads at max_examples=6 (same pattern as _VLSUMultimodalDataset).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• arxiv.org
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
• huggingface.co
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
  • Triggering command: /home/REDACTED/work/PyRIT/PyRIT/.venv/bin/python3 /home/REDACTED/work/PyRIT/PyRIT/.venv/bin/python3 -u -c import sys;exec(eval(sys.stdin.readline())) (dns block)
• researchtrend.ai
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>FEAT VisualLeakBench</issue_title>
<issue_description>#### Is your feature request related to a problem? Please describe.

We don't have https://arxiv.org/pdf/2603.13385 dataset in PyRIT yet.

Describe the solution you'd like

Add dataset loader, check integration test for dataset, write unit tests, ensure harm categories are captured.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes FEAT VisualLeakBench #1530

---

📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.