cloudflare-crawl

A Claude Code skill for crawling websites and extracting content using the Cloudflare Browser Rendering /crawl REST API.

What it does

When installed, this skill teaches Claude Code how to:

• Crawl websites using Cloudflare's Browser Rendering API — initiate crawls, poll for results, and paginate through responses
• Extract content as HTML, Markdown, or structured JSON (with AI-powered extraction)
• Guide site owners on making their sites crawlable — robots.txt, sitemaps, WAF skip rules, semantic HTML, and Cloudflare AI Crawl Control

Install

claude skill add --from https://github.com/portdeveloper/cloudflare-crawl

Prerequisites

• A Cloudflare account with an API token
• Your Cloudflare account_id
• The Browser Rendering API is in open beta, available on Workers Free and Paid plans

Usage

Once installed, just ask Claude Code to crawl a site:

> Crawl https://docs.example.com and save the markdown to a local folder

> Extract product names and prices from https://shop.example.com/products as JSON

> Help me set up my site's robots.txt and sitemap so Cloudflare can crawl it

The skill handles the async workflow automatically — POST to start a crawl, GET to poll for results.

What's included

File	Description
cloudflare-crawl/SKILL.md	Core skill — API reference, parameters, examples, troubleshooting
cloudflare-crawl/references/making-sites-crawlable.md	Site owner guide — robots.txt, sitemaps, WAF rules, semantic HTML

Key features of the API

• Crawl up to 100,000 pages per job
• Output as HTML, Markdown, or structured JSON
• AI-powered JSON extraction with custom prompts and schemas
• Incremental crawls with modifiedSince and maxAge
• URL filtering with include/exclude wildcard patterns
• render: false mode for fast static HTML fetch (no browser time cost)
• Respects robots.txt and Crawl-delay directives

License

MIT