Add exploit development workflow and vector memory system design#77
Merged
Add exploit development workflow and vector memory system design#77
Conversation
…dmap
Detailed analysis of PentAGI's 13-agent delegation system ("simulated security
firm") compared to BLHACKBOX's MCP-native architecture. Identifies 8 specific
optimization opportunities prioritized by ROI, with P0 items being: exploit
development skill (Coder agent equivalent) and vector memory system (pgvector
for cross-session learning). Also documents areas where BLHACKBOX is already
superior (70+ tools, WireMCP, Screenshot MCP, MCP standard, PoC rigor).
https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
Added findings from Go source code analysis (commit e97bbe5): - Barrier function pattern per agent (hack_result, code_result, etc.) - Tool isolation: each agent sees ONLY its allowed tools (enforced in Go) - Full orchestration flow: Generator → Primary Agent → sub-agent loops - Reflector catches text-only responses AND tool-limit approach - Memory system dual thresholds: 0.45 for broad recall, 0.75 for search - Retry constants: maxRetries=3, delay=5s - Key source file reference table for implementation guidance https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
…inding - PentAGI uses 4-tier similarity thresholds: 0.45 general, 0.7 code, 0.75 search, 0.8 guides - Clarified PentAGI is custom Go framework, NOT LangChain/LangGraph - Added "Key Architectural Insights" section with 5 design principles - Noted tool isolation as security model (Primary Agent can't run commands) - Documented Coder→Pentester delegation flow for exploit development https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
PentAGI recommends isolating worker execution on a separate server with Docker-in-Docker over TLS to prevent untrusted exploit code from running on the primary system. Worth considering for BLHACKBOX production deployments. https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
Five features inspired by PentAGI's multi-agent architecture: 1. /exploit-dev skill (P0) — New Claude Code skill for custom exploit development with write-test-iterate loops. Mirrors PentAGI's Coder agent: research vuln, design exploit, write code, test in Kali container, iterate on failures, capture evidence. 2. Loop detection hook (P1) — PreToolUse hook that tracks MCP tool calls and detects repeated identical calls (threshold: 4). Blocks looping calls and warns at session milestones (50/80 calls). Mirrors PentAGI's Reflector agent. 3. Auto knowledge graph population (P1) — aggregate_results now auto-populates Neo4j with hosts, services, vulnerabilities, subdomains, and endpoints from AggregatedPayload. Previously only stored a flat session node. 4. search_exploits + get_exploit_code tools (P2) — Dedicated searchsploit MCP tools for Kali MCP server with structured JSON output. Mirrors PentAGI's Sploitus integration. 5. Vector memory system design doc (P0) — Architecture for pgvector- backed cross-session memory with 6 MCP tools and 4-tier similarity thresholds. Ready for implementation. All 187 existing tests pass. No breaking changes. https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
Documentation (README.md, DOCKER.md): - Add /exploit-dev to skills table (10→11 skills) - Add "Advanced Features" section: exploit-dev, searchsploit tools, loop detection, auto knowledge graph - Update architecture diagram to include searchsploit - Update project structure with loop-detector.sh and exploit-dev - Update components table with searchsploit mention Neo4j auto-graph fixes: - Validate port range (1-65535) before creating port nodes - Add index to vuln identifier fallback to prevent node collisions - Strip and validate subdomains before creating domain nodes - Use proper type annotations (string forward refs) instead of Any - Upgrade Neo4j exception logging from DEBUG to INFO Template system: - Create blhackbox/prompts/templates/exploit-dev.md (MCP template) - Register exploit-dev in TEMPLATES dict (prompts/__init__.py) - Now 11 templates accessible via get_template/list_templates API Environment: - Add vector memory system vars to .env.example (commented out) - Update OpenAI key description to mention embedding use All 187 tests pass. https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
- Revert type annotations to Any (ruff F821: types not imported at module scope) - Rewrite exploit-dev SKILL.md with specific tool calls, decision trees, iteration tracking, and PentAGI Coder patterns (search first, 5-iter limit) - All ruff checks pass, all 187 tests pass https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
Applied three key improvements across all skills:
1. Tool discovery mandate — All skills now include "Query each MCP
server's tool listing" in their pre-flight checklist
2. MCP Tool Quick Reference — Every skill now has a reference section
with specific tool calls:
- Exploitation skills: searchsploit, msfconsole search, capture_packets,
extract_credentials, follow_stream, take_screenshot,
take_element_screenshot, annotate_screenshot, /exploit-dev reference
- Recon skills: passive DNS tools, passive OSINT tools
- OSINT skill: passive-only tools with explicit "no active testing" note
3. Decision gates and specifics — vuln-assessment and quick-scan now
have explicit exploitation decision flows (validate → exploit → if
WAF blocks → try bypass → if needs custom code → /exploit-dev)
Skills updated: full-pentest, full-attack-chain, quick-scan,
vuln-assessment, web-app-assessment, network-infrastructure,
api-security, bug-bounty, recon-deep, osint-gathering
All ruff checks pass, all 187 tests pass.
https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
CVE-2026-4539 is a low-severity local ReDoS in pygments' AdlLexer (CVSS 3.3, local-only attack vector). No upstream patch exists yet. - Pin pygments>=2.19.2 in requirements.txt to auto-upgrade when fix lands - Add --ignore-vuln CVE-2026-4539 to CI pip-audit step with comment to remove when pygments patches - Reference: https://nvd.nist.gov/vuln/detail/CVE-2026-4539 https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces two major capabilities to BLHACKBOX inspired by PentAGI's architecture:
/exploit-dev) — A structured workflow for writing, testing, and iterating on custom exploit codeKey Changes
Exploit Development Workflow
/exploit-devcommand for autonomous exploit development with write-test-iterate loopsblhackbox/prompts/templates/exploit-dev.md) — 6-phase methodology: intelligence gathering, design, development, validation, payload variants, and results aggregationsearchsploittool integration with two new MCP tools:search_exploits()— Query ExploitDB with structured JSON resultsget_exploit_code()— Retrieve full source code of discovered exploitsVector Memory System (Design Document)
docs/vector-memory-design.md) — Complete architecture for semantic search across engagement historyrecall_similar,store_finding,store_exploit,store_guide,get_memory_stats,auto_store_sessionmemory-mcpcontainer behind--profile memoryflagaggregate_results, memory recall in skill templates, exploit storage in/exploit-devPentAGI Comparison & Roadmap
docs/pentagi-comparison.md) — Detailed comparison of PentAGI's 13-agent architecture vs BLHACKBOX's single-LLM approachSupporting Infrastructure
.claude/hooks/loop-detector.sh) — Detects repeated identical tool calls and injects warnings to prevent infinite loops (inspired by PentAGI's Reflector agent)aggregate_resultsto auto-populate knowledge graph with hosts, services, vulnerabilities, and relationships_populate_knowledge_graph()) — Structured extraction from AggregatedPayload into Neo4j nodesDocumentation Updates
/exploit-devskill and mentionsearchsploitin Kali MCP tools.env.exampleplaceholders for future vector memory configurationImplementation Notes
run_kali_tool,capture_packets,follow_stream)aggregate_resultsexecutionTesting Recommendations
/exploit-devskill against a known vulnerable service (e.g., DVWA, WebGoat)search_exploitsand `https://claude.ai/code/session_01E2e15egjZn2tbVSRPCHWpm