Integration idea: anybrowse as fallback for Cloudflare-protected sites

[kc23go]

Context

Crawlee is excellent for large-scale crawling but Cloudflare-protected sites are still a challenge -- even with Playwright, datacenter IPs get blocked quickly.

Idea

anybrowse uses residential Chrome (real IP, real browser) and could work as a fallback request handler for URLs that fail Cloudflare challenges.

Conceptually:

const router = createPlaywrightRouter();

router.addDefaultHandler(async ({ request, page }) => {
    // Try standard Playwright first
    const content = await page.content();
    if (content.includes('cf-browser-verification') || content.length < 500) {
        // Fallback to anybrowse for Cloudflare sites
        const r = await fetch('https://anybrowse.dev/scrape', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({url: request.url})
        });
        const data = await r.json();
        // Process data.markdown
    }
});

Not suggesting a native integration necessarily -- just sharing the pattern in case it's useful for Cloudflare use cases.

Docs: https://anybrowse.dev/docs

[CircuitSavage]

The pattern's solid, and it's worth splitting into two flavours, because they fail differently once you wire them into Crawlee.

1. Ship the whole render out (what anybrowse does here): hand the URL to a residential-browser service, get HTML/markdown back. Simple, but you leave Crawlee's session pool, request queue and proxy handling behind for that URL, and you pay per full page render. Fine for a handful of stubborn URLs, expensive if half your queue sits behind Cloudflare.

2. Solve just the token, inject it, stay in Crawlee. For Turnstile you don't actually need the page rendered somewhere else. The thing blocking you is one value: cf-turnstile-response (or the interstitial's challenge token). A datacenter IP plus headless Chrome nearly guarantees Turnstile won't hand you a pass client-side; that's the whole point of it, it scores the browser and IP and quietly withholds the token. So you fetch the token from a solver, drop it into the field, fire the callback, and your existing PlaywrightCrawler handler carries on with the cleared context. You keep sessions, retries and proxy rotation.

One check that saves time before you reach for either: log the real status. A live Turnstile challenge is a 403 with the widget present. A cf-browser-verification / Just a moment interstitial is the managed challenge. A plain 403 with no widget is usually IP reputation, and no solver touches that; you just need cleaner IPs.

For the token route I keep a small Crawlee wrapper that does the inject-and-continue step: https://github.com/CircuitSavage/crawlee-turnstile (it calls Peak's solve API under the hood; disclosure: I work on Peak). Peak is ~$0.90/1k successful solves, down to $0.35 at volume, and you pay per success, so the ~1k free solves are enough to check whether the token route even works on your target before you commit. If it turns out the block is IP-based rather than a real challenge, save the money and fix the proxies first.