Skip to content

DOCUMENTATION_INDEX.md

Latest commit

 

History

History
348 lines (279 loc) · 12.9 KB

DOCUMENTATION_INDEX.md

File metadata and controls

348 lines (279 loc) · 12.9 KB

NetBear Documentation Index

Welcome to the complete NetBear documentation! This index will help you navigate all available resources.

Quick Navigation

🚀 Getting Started (Start Here!)

  • QUICKSTART.md - Get scanning in 5 minutes
    • Installation steps
    • Your first scan workflow
    • Common workflows (API testing, authenticated, browser capture)
    • Understanding results
    • Troubleshooting common issues

📖 Complete Reference

  • COMPLETE_GUIDE.md - Comprehensive user guide (25+ pages)
    • Full architecture overview
    • Detailed getting started
    • Output & reports reference
    • Customization fundamentals
    • Advanced features
    • Performance tips
    • Security notes

🔧 Customization & Advanced Usage

  • CUSTOMIZATION.md - Deep customization guide

    • Crawl parameter tuning
    • Nuclei template creation (with examples)
    • Endpoint extraction tuning
    • Performance optimization
    • Authentication setup
    • Real-world scenarios & examples

  • TEMPLATES_GUIDE.md - Complete Nuclei template documentation

    • What "-direct" templates mean
    • Understanding matcher logic (DSL, word, regex, status)
    • Deep dive into each template (API Exposure, Auth Bypass, IDOR)
    • Creating custom templates (step-by-step)
    • Advanced techniques & patterns
    • Debugging templates
    • Integration with Burp Suite

💻 Code Documentation & Configuration

  • config.py - Central configuration file with inline comments

    • General settings (timeout, retry, proxy)
    • Netbear crawling parameters (depth, rate limit)
    • Nuclei integration settings (timeout, templates, severity)
    • NextCloud testing configuration
    • Reporting formats

  • nuclei_handler.py - Nuclei scanning orchestration

    • Key method: run_nuclei() - Execute vulnerability scans with detailed comments
    • Method: load_targets() - Load and filter targets
    • Method: parse_nuclei_output() - Parse results

  • endpoint_extractor.py - API endpoint discovery

    • Class-level documentation explaining all extraction sources
    • Method: normalize_endpoints() - With filtering customization guide
    • Method: extract_from_js_files() - JS pattern extraction
    • CUSTOMIZATION comments for adding/removing noise patterns

  • netbear_crawler.py - Main crawler engine

    • Function: load_scopes() - Scope validation setup
    • Complete module documentation for customization

📋 Project Documentation

  • Previous Implementation Docs (context for fixes applied)
    • NUCLEI_FIX_SUMMARY.md - All bugs fixed and solutions applied
    • NUCLEI_TEMPLATE_IMPROVEMENTS.md - Template evolution details
    • NUCLEI_IMPROVEMENTS_QUICKSTART.md - Quick reference for improvements

By Use Case

"I want to scan a website right now"

  1. Read: QUICKSTART.md
  2. Run: python netbear_crawler.py
  3. Find results in: reports/run_*/nuclei_output_*/nuclei_results.txt

"I want to understand how everything works"

  1. Start: COMPLETE_GUIDE.md - Read "Architecture" section
  2. Review: TEMPLATES_GUIDE.md - Understand matcher logic
  3. Explore: endpoint_extractor.py - See extraction logic
  4. Test: Run a scan and review results

"I want to customize NetBear for my specific targets"

  1. Read: CUSTOMIZATION.md - Pick your scenario
  2. Edit: config.py - Update settings with clear comments
  3. Modify: endpoint_extractor.py - Adjust filtering
  4. Create: Custom YAML templates following TEMPLATES_GUIDE.md
  5. Test: Run on a test target

"I want to create custom Nuclei templates"

  1. Reference: TEMPLATES_GUIDE.md - Full template guide
  2. Examples: See all matcher types and patterns
  3. Copy: Start from existing netbear-*-direct.yaml template
  4. Test: Use nuclei -u URL -t template.yaml -v
  5. Register: Add to NUCLEI_TEMPLATES in config.py

"Something broke, I need to troubleshoot"

  1. Check: QUICKSTART.md - Troubleshooting section
  2. Review: COMPLETE_GUIDE.md - Troubleshooting section
  3. Debug: Run with -v verbose flag: python netbear_crawler.py
  4. Logs: Check reports/run_*/nuclei_output_*/nuclei_*.log
  5. Verify: Use nuclei -validate -t template.yaml for template errors

"I want to test authenticated endpoints"

  1. Reference: CUSTOMIZATION.md
  2. Option A: Use browser HAR capture
  3. Option B: Use authenticated_endpoint_crawler.py
  4. Option C: Set credentials in config.py

File Map

Essential Files

NETBEAR/
├── QUICKSTART.md                    ← START HERE for fast setup
├── COMPLETE_GUIDE.md                ← Full documentation
├── CUSTOMIZATION.md                 ← Advanced customization
├── TEMPLATES_GUIDE.md               ← Nuclei template reference
├── config.py                        ← Configuration (with comments)
├── nuclei_handler.py                ← Nuclei orchestration (with comments)
├── endpoint_extractor.py            ← Endpoint discovery (with comments)
└── netbear_crawler.py               ← Main crawler (with comments)

Template Files

NETBEAR/
├── netbear-api-exposure-direct.yaml ← API key/secret detection
├── netbear-auth-bypass-direct.yaml  ← Auth bypass detection
├── netbear-idor-direct.yaml         ← IDOR vulnerability detection
└── [your-custom].yaml               ← Your custom templates

Tools & Utilities

NETBEAR/
├── authenticated_endpoint_crawler.py ← Test authenticated areas
├── har_endpoint_extractor.py         ← Extract from browser HAR files
├── main.py                           ← Interactive CLI
├── run_nuclei_on_crawl.sh            ← Re-run Nuclei on existing crawls
└── nuclei_workflow.sh                ← Complete automation wrapper

Input Files (Edit these!)

NETBEAR/
├── targets.txt          ← List of URLs to test (one per line)
├── scopes.txt           ← Allowed domains (one per line)
└── proxies.txt          ← Proxy servers (one per line, optional)

Output Directory

NETBEAR/reports/
└── run_TIMESTAMP/
    ├── domain_com/
    │   ├── report.txt               ← Crawl summary
    │   ├── nuclei_targets.txt       ← Extracted endpoints
    │   ├── nuclei_targets_clean.txt ← Filtered endpoints
    │   ├── js_structures.json       ← API/auth patterns
    │   ├── res_*-CRITICAL.js        ← High-priority JS files
    │   └── res_*-HIGH_INTEREST.js   ← Interesting JS findings
    ├── nuclei_output_TIMESTAMP/
    │   ├── nuclei_results.txt       ← Vulnerability findings
    │   ├── nuclei_results.json      ← JSON format findings
    │   └── nuclei_*.log             ← Debug logs
    └── exports/
        ├── domain.har               ← Burp Suite import
        ├── domain_requests.curl     ← cURL commands
        └── domain.json              ← Raw API calls

Documentation By Component

1. Web Crawler (netbear_crawler.py)

2. JavaScript Analysis (js_analyser.py)

  • What it does: Extracts API endpoints, auth functions, XSS sinks from JS
  • Used by: endpoint_extractor.py, crawler outputs
  • Reference: See method comments in source file

3. Endpoint Extractor (endpoint_extractor.py)

  • What it does: Aggressive extraction from crawl data with noise filtering
  • Customization: CUSTOMIZATION.md
  • How to modify: Check normalize_endpoints() method for exclusion patterns
  • Performance: Processes first 20 largest JS files only

4. Nuclei Integration (nuclei_handler.py)

5. Reporting (reporting.py)

  • What it does: Export results as HAR, cURL, JSON
  • Formats: HAR (Burp), cURL (replay), JSON (automation)
  • Location: reports/run_*/exports/

6. Authentication (auth.py)

  • What it does: Handle login flows for authenticated testing
  • Setup: CUSTOMIZATION.md
  • Alternative: HAR extraction for browser-captured traffic

7. Fuzzing (fuzzer.py)

  • What it does: Parameter fuzzing and testing
  • Advanced feature: See command examples in COMPLETE_GUIDE.md

8. NextCloud Module (nextcloud/)

  • What it does: Specialized NextCloud security testing
  • Configuration: config.py - NEXTCLOUD_* settings
  • Quick start: See COMPLETE_GUIDE.md

Configuration Reference

Critical Settings (Most Important)

Setting File Purpose Default When to Change
NETBEAR_MAX_DEPTH config.py How deep to crawl (1-3) 2 Fast (1) vs Thorough (3)
NUCLEI_ENABLED config.py Enable Nuclei scanning True Crawl-only mode (False)
NUCLEI_TIMEOUT config.py Scan timeout (seconds) 1800 100+ targets (3600)
NUCLEI_TEMPLATES config.py Which templates to use 3 direct Add custom templates
SCOPES_FILE netbear_crawler.py Allowed domains file "scopes.txt" Change filename

Performance Settings

Setting Fast Balanced Thorough
NETBEAR_MAX_DEPTH 1 2 3
NETBEAR_MAX_PAGES 5 15 30
NETBEAR_RATE_LIMIT 0.5s 1.5s 3.0s
NUCLEI_TIMEOUT 300s 1800s 3600s
Time to complete 5-10m 15-30m 45-90m

See: CUSTOMIZATION.md

Common Commands

Run a Complete Scan

python netbear_crawler.py

Scan with Specific Configuration

# Edit config.py first, then:
NETBEAR_MAX_DEPTH=2 NUCLEI_TIMEOUT=3600 python netbear_crawler.py

Extract from HAR File (Browser Capture)

python har_endpoint_extractor.py --har-file traffic.har --output-txt targets.txt
nuclei -l targets.txt -t netbear-*-direct.yaml

Test Authenticated Endpoints

python authenticated_endpoint_crawler.py \
    --url https://app.example.com \
    --username user@example.com \
    --password password123

Re-run Nuclei on Existing Crawl Data

./run_nuclei_on_crawl.sh reports/run_20260208_050101/example_com example.com critical,high

Interactive CLI Mode

python main.py

Test a Custom Template

nuclei -u https://api.example.com -t my-custom.yaml -v

Important Notes

⚠️ Security

  • Always use scopes.txt to prevent out-of-scope scanning
  • Increase rate limit (2-5s) for stealth
  • Get written permission before testing
  • Store results securely (may contain sensitive data)

⚡ Performance

  • First scan: 15-30 minutes for complete coverage
  • Subsequent scans: Use authenticated/HAR extraction for speed
  • Large targets (100+ endpoints): Increase NUCLEI_TIMEOUT to 3600s
  • Parallel scans: Use background mode for multiple domains

🔄 Best Practices

  1. Start with QUICKSTART.md
  2. Test on your own app first
  3. Review results in nuclei_results.txt
  4. Gradually customize templates for your targets
  5. Re-run regularly as targets change

Getting Help

  1. Quick answers: Check relevant section in QUICKSTART.md
  2. Detailed info: See COMPLETE_GUIDE.md
  3. Customization: Read CUSTOMIZATION.md + inline code comments
  4. Templates: Review TEMPLATES_GUIDE.md
  5. Debugging: Run with -v flag and check logs in reports/run_*/nuclei_output_*/

Documentation Maintenance

This documentation index is current as of the latest version. All files have inline comments for customization guidance:

  • ✅ config.py - Detailed setting explanations
  • ✅ nuclei_handler.py - Method documentation with customization tips
  • ✅ endpoint_extractor.py - Class documentation + customization guide
  • ✅ netbear_crawler.py - Module-level documentation for customization

Each Python file includes docstrings explaining what to customize and why.

Last Updated: 2025 For the latest updates and examples, refer to the inline comments in each file.

Happy hunting! 🎯