feat: Cloudflare bypass for headless mode using Patchright#12
Open
simwai wants to merge 36 commits into
Open
Conversation
… bypass Replaced Playwright with Patchright (a stealthy drop-in replacement) to prevent Cloudflare from detecting headless mode. Implemented randomized delays (jitter) between API requests in LibraryDiscovery to mimic human-like pacing. Removed redundant manual automation-hiding flags as Patchright handles stealth internally. Updated all imports and test configurations to use patchright. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
Contributor
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
… bypass Replaced Playwright with Patchright (a stealthy drop-in replacement) to prevent Cloudflare from detecting headless mode. Implemented randomized delays (jitter) between API requests in LibraryDiscovery to mimic human-like pacing. Removed redundant manual automation-hiding flags as Patchright handles stealth internally. Updated all imports and test configurations to use patchright. Updated scripts/build-exe.js to ensure compatibility with Patchright for SEA builds. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Replaced Playwright with Patchright for deep stealth capabilities. - Implemented Strategy pattern for discovery and extraction (API, Scroll, Interaction, AI modes). - Added automatic Cloudflare detection and bypass logic with strategy fallback. - Added mock-based integration tests for scraping strategies. - Updated SEA build script and README.md with new stealth directives. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
…bypass - Integrated Patchright for deep browser stealth and Cloudflare bypass. - Implemented Strategy pattern for thread discovery (API, Scroll, Interaction, AI). - Implemented Strategy pattern for content extraction (API, DOM, Native Export, AI-Assisted). - Refined Native Export strategy to follow interactive thread settings flow. - Refined AI-Assisted strategy to use Ollama for dynamic selector extraction. - Added automatic Cloudflare detection with strategy fallback mechanism. - Updated .env.example with new DISCOVERY_MODE and EXTRACTION_MODE options. - Verified with integration tests and updated SEA build script. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Replaced Playwright with Patchright for deep browser stealth. - Implemented a dual-layer Strategy Pattern for discovery and extraction. - Added 4 distinct strategies per phase: API, Scroll, Interaction/Native, and AI-Assisted. - Native Export strategy now mimics human interaction (Thread -> Settings -> Export). - AI Scrape strategy uses local Ollama models for dynamic selector extraction. - Added a Cloudflare handler that automatically bypasses challenges and triggers strategy fallback. - Updated .env.example and README.md with comprehensive stealth documentation. - Verified with mock-based integration tests. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Replaced Playwright with Patchright for deep browser stealth. - Implemented a dual-layer Strategy Pattern for discovery and extraction. - Added 4 distinct strategies per phase: API, Scroll, Interaction/Native, and AI-Assisted. - Refined Native Export strategy to follow interactive thread settings flow. - Refined AI-Assisted strategy to use Ollama for dynamic selector extraction. - Enhanced Cloudflare handler with multi-attempt frame interaction and humanized mouse movements. - Updated .env.example and README.md with comprehensive stealth documentation. - Verified with mock-based integration tests. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Integrated HumanNavigator for organic mouse movements (Bezier curves) and sinusoidal scrolling. - Added session warming: visits Perplexity home page and simulates browsing before sensitive actions. - Enhanced navigator spoofing: masks hardware properties and cleans automation signatures. - Added human-like "reading" pauses and movement jitter to all scraping strategies. - Improved Cloudflare bypass with multi-frame detection and hovered interaction. - Updated README with Stealth & Behavioral Resilience documentation. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Integrated Vision-based bypass: takes 1920x1080 screenshots and uses Ollama to identify interaction coordinates, bypassing Turnstile honeypots. - Replaced Playwright with Patchright for deep browser stealth. - Implemented dual-layer Strategy Pattern for discovery and extraction (8 strategies total). - Added HumanNavigator for organic mouse movements (Bezier curves) and sinusoidal scrolling. - Implemented session warming (home page visit) and advanced navigator masking. - Refined Native Export and AI-Assisted scraping strategies. - Updated README.md with comprehensive "Stealth & Behavioral Resilience" documentation. - Verified with integration tests and SEA build compatibility. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Updated AI architecture to use `ministral-3` for vision reasoning and `cogito` for text reasoning. - Implemented automatic model pulling: system checks for required models on startup and pulls them if missing. - Added a 10GB minimum disk space requirement check at application startup. - Enhanced Vision-based Cloudflare bypass using 1920x1080 snapshots and targeted pixel-coordinate clicking. - Restored and expanded README.md with comprehensive guides for both new and advanced users. - Updated .env.example with the new dual-model configuration. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
…ardening - Replaced Playwright with Patchright for deep browser stealth. - Implemented a dual-layer Strategy Pattern for discovery and extraction (8 strategies total). - Added Vision-based Cloudflare bypass: captures 1920x1080 screenshots and uses `ministral-3` to identify precise interaction coordinates, circumventing iframe honeypots. - Added 10GB disk space requirement check at startup to prevent runtime model failures. - Implemented automatic Ollama model pulling for `cogito` (text) and `ministral-3` (vision). - Integrated `HumanNavigator` for organic mouse movements and sinusoidal scrolling. - Enhanced browser masking with advanced navigator spoofing scripts and session warming. - Revamped README.md and CONTRIBUTING.md with comprehensive guides for all users. - Verified with integration tests and updated SEA build scripts. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Fixed "Page not initialized" error in BrowserManager by ensuring page creation before session warming. - Verified dual-model architecture: ministral-3 (vision) and cogito (text). - Solidified automatic model pulling and 10GB disk space check. - Refined session warming and behavioral bypass logic. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Implemented a 3-retry loop in `handleCloudflare` with incremental temperature reduction and increased prompt pressure for valid JSON. - Enhanced JSON parsing in vision bypass to strip comments and non-JSON text. - Added fast-fail logic: the scraper now throws immediately if Cloudflare bypass fails, preventing hangs. - Updated `OllamaClient` to support custom temperatures per request. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Re-implemented model readiness check to verify presence of cogito, ministral-3, and embedding models. - Integrated `ollama pull` via CLI to ensure robust installation and provide user-friendly progress tracking in the terminal. - Enhanced model name matching to handle tags and 'latest' suffixes. - Verified that vision-based bypass and text reasoning are properly configured to use respective models. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Updated OLLAMA_URL default port in .env.example - Replaced fetch with got-scraping in OllamaClient to improve request reliability - Added more robust error logging and handling in OllamaClient - Adjusted default AI temperature settings for more deterministic responses in OllamaClient and cloudflare.ts - Refined the vision-based Cloudflare bypass prompt in cloudflare.ts for better coordinate accuracy - Cleaned up code formatting and linting across browser.ts, conversation-extractor.ts, and other scraper files - Updated Zod schemas and logic in extraction-strategy.ts to handle various API response shapes - Deleted the standard fetch-based request logic in OllamaClient
- Implemented `AiProvider` factory supporting `ollama` and `openrouter` (via `got-scraping`). - Renamed AI configuration for clarity: `LLM_RAG_MODEL` (cogito) and `LLM_VISION_MODEL` (ministral-3). - Integrated `LLM_VISION_MODEL` specifically for vision-based captcha bypass. - Integrated `LLM_RAG_MODEL` for RAG synthesis and research planning. - Added 10GB disk space check and automatic model pulling for Ollama. - Updated .env.example with new intuitive configuration options. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Refined Vision-based Cloudflare bypass prompt for higher coordinate accuracy on 1920x1080 snapshots. - Optimized AI temperature settings: 0.2 for text reasoning (cogito) and 0.1 for vision tasks (ministral-3) to ensure deterministic outputs. - Adjusted interaction timing and mouse movement precision for bypass clicks. - Ensured OllamaClient remains robust against response format variations. - Verified all 8 scraping strategies and auto-fallback logic. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Integrated `AiProvider` factory supporting Ollama (local) and OpenRouter (cloud). - Implemented Vision-based Cloudflare bypass: captures 1920x1080 snapshots and leverages AI to identify precise interaction coordinates, circumventing iframe honeypots. - Enforced Ollama for embeddings to ensure local data privacy and consistency. - Updated intuitive configuration: `LLM_RAG_MODEL` for text and `LLM_VISION_MODEL` for bypass tasks. - Defaults to `stepfun/step-3.5-flash:free` for OpenRouter and `cogito`/`ministral-3` for Ollama. - Hardened system startup with 10GB disk check and automatic Ollama model pulling. - Reworked README and CONTRIBUTING with professional formatting and detailed provider setup. - Replaced manual interactions with organic `HumanNavigator` movements and scrolling. - Verified with integration tests and SEA build compatibility. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Changed default Ollama port to 11435 in config and .env.example. - Disabled headless mode by default in config and .env.example to improve out-of-the-box reliability. - Verified configuration consistency across the project. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Added detailed, color-coded logging for the entire Cloudflare bypass sequence. - Logs include provider info, vision model used, behavioral signature status, and coordinate targeting progress. - Improved user feedback for success and failure states of the vision protocol. - Verified configuration and build stability. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Implemented direct structural interaction with Turnstile widgets as the primary bypass strategy (based on Python POC). - Demoted Vision-based analysis to a fallback strategy due to local model performance limits. - Enhanced structural bypass with response token monitoring and recursive attempts. - Reworked README.md with a professional, table-based configuration guide and cleaner layout. - Cleaned up unused imports and verified build stability. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Switched default RAG reasoning model to `deepseek-r1:7b`. - Switched default vision reasoning model to `qwen3.5:4b`. - Updated configuration, .env.example, and README documentation to reflect the new model defaults. - Verified automatic model pull integration for the new defaults. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
…ghost-cursor - Implemented Structural Turnstile Interaction as the primary bypass strategy (based on Python POC). - Integrated `ghost-cursor-patchright-core` for authentic, non-linear mouse movements and clicks. - Replaced manual behavioral simulation with strategy-based Turnstile solving. - Set default reasoning models: `deepseek-r1:7b` (RAG) and `qwen3.5:4b` (Vision). - Refined AI configuration with intuitive naming: `LLM_RAG_MODEL`, `LLM_VISION_MODEL`. - Ensured all embeddings are generated locally via Ollama. - Hardened system startup with 10GB disk check and CLI-based automatic model pulling. - Standardized custom error classes: `DiscoveryError`, `ExtractionError`, `CloudflareBypassError`, etc. - Revamped README.md with professional table-based configuration and clear setup guides. - Fixed browser initialization race conditions and type-check errors. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Implemented Structural Turnstile Interaction as the primary bypass strategy. - Integrated `ghost-cursor-patchright-core` for authentic human-like mouse paths and clicks. - Updated default reasoning models: `deepseek-r1:7b` (RAG) and `qwen3.5:4b` (Vision). - Refined AI configuration with intuitive naming: `LLM_RAG_MODEL`, `LLM_VISION_MODEL`. - Re-implemented and improved model verification and auto-pull logic via CLI. - Added 10GB disk space requirement check at startup. - Restored and enhanced README.md and CONTRIBUTING.md following original project style. - Verified all 8 scraping strategies and fallback logic with integration tests. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Enhanced `StructuralTurnstileStrategy` with multi-point targeting (left, center, top-left) to better handle complex widget layouts. - Hardened `VisionTurnstileStrategy` with a stricter prompt and placeholder cleanup to prevent JSON parsing errors. - Reduced vision LLM temperature to 0.1 for maximum coordinate precision. - Fixed minor build errors in configuration utility. - Added a 2-second stabilization delay before bypass sequence initiation. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Removed environment setup and Ollama instructions from CONTRIBUTING.md as they are already detailed in README.md. - Streamlined CONTRIBUTING.md to focus on development workflow, commit guidelines, and PR process. - Ensured README.md remains the single source of truth for project setup and usage. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Added fallback mechanism in `OpenRouterClient` for vision requests. - If native `image_url` data blocks fail or are unsupported, the system automatically retries by inlining the base64-encoded screenshot directly into the prompt. - This ensures maximum compatibility across various models hosted on OpenRouter. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Corrected OpenRouter vision request format to use `image_url` instead of `imageUrl` for OpenAI compatibility. - Added explicit error checking for OpenRouter API error objects. - Increased request timeouts to 120s for vision tasks to prevent ETIMEDOUT during image processing. - Refined inline base64 fallback to include the data URI prefix. - Verified build and type-safety. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Optimized `OpenRouterClient` by disabling HTTP/2 and header generation for cloud API calls, preventing protocol-level timeouts. - Switched vision screenshots to JPEG (quality 70) to significantly reduce payload size for cloud providers while maintaining coordinate accuracy. - Enhanced OpenRouter error reporting to surface specific API issues (credits, model availability). - Refined vision fallback to use structured JPEG data URIs. - Verified build and type-safety. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Implemented 50% image resizing (960x540) using `jimp` before sending screenshots to AI. - Added coordinate upscaling (2x) to maintain click accuracy on 1920x1080 viewports. - Streamlined `OpenRouterClient` by consolidating logic around the standard `image_url` protocol. - Reduced overall vision request latency and eliminated large-payload timeout issues. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Refined RAG orchestration prompts using XML-style tagging for better context isolation. - Implemented structured "System Roles" (Lead Researcher, Fact Extraction Engine, Narrator) for multi-stage reasoning. - Strengthened AI-assisted DOM selector extraction with domain-specific context. - Optimized JSON extraction logic to handle varied LLM response formats. - Verified all 8 scraping strategies and fallback integrity. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
…ration - Restored original "Goertzel-lite" project style and structure in README.md and CONTRIBUTING.md. - Surgically integrated advanced stealth and turnstile bypass documentation. - Implemented multi-tiered Turnstile bypass (Structural + Vision) with ghost-cursor. - Enforced local Ollama embeddings and dual AI reasoning model defaults. - Hardened system startup with disk check and CLI model pulling. - Verified all 8 scraping strategies and configuration consistency. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Created `VisualLogger` utility to capture diagnostic screenshots during bypass attempts. - Added coordinate marking (red crosshairs) on screenshots to visualize exact click locations. - Integrated visual logging into `StructuralTurnstileStrategy` and `VisionTurnstileStrategy`. - Automated sequential file naming (e.g., `001_challenge_detected`, `002_structural_attempt_1_pre_left`) for easy identification of failure points. - Screenshots are saved in the `debug_screenshots` directory for user evaluation. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Updated all `waitForTimeout` calls in `StructuralTurnstileStrategy` and `VisionTurnstileStrategy` to 6000ms. - Provides more time for Cloudflare/Turnstile token generation and page state resolution between interaction attempts. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Reduced base interaction timeout to 5 seconds. - Added randomized noise (jitter) of 0-2 seconds to every interaction wait. - This results in a 5-7 second variable wait window, improving stealth and resilience. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
- Increased base interaction timeout to 14 seconds with 0-2 seconds of random noise. - Provides a total of 14-16 seconds for Cloudflare/Turnstile token validation, which is often required for slower network environments or high-security challenges. Co-authored-by: simwai <16225108+simwai@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change addresses the issue where Cloudflare would block headless mode but not headful mode.
Key changes:
@playwright/testandplaywright-coreforpatchright, which provides better stealth and bypasses common bot detection.LibraryDiscovery.tsto include a randomized delay between API batch fetches. This prevents the rapid succession of requests that often triggers Cloudflare challenges.patchrightimports.BrowserManager.tsto letpatchrightmanage stealth settings automatically.PR created automatically by Jules for task 18377907482892794050 started by @simwai