bypass anything.return in 500ms.
An adaptive, self-hosted scraping engine — tries the cheapest path first and only escalates when needed. Solves Cloudflare challenges natively in 4–15 seconds. Caches browser sessions per domain and returns sub-500ms on every repeat request. No tokens. No quotas. No cloud. Just Docker.
capabilities
built for the modern web.
Every protection layer. Every captcha type. One API. Nothing hosted externally.
Adaptive Tier Engine unique
Every request takes the cheapest path first — plain HTTP, then cached session, then live browser solve, then residential proxy. You pay the full browser cost only when you have to.
Camoufox Firefox stealthy
Powered by Camoufox — a hardened Firefox build that patches fingerprints at the C++ level. Cloudflare's bot score sees a real user. No CDP leaks. No Chrome fingerprint flags.
Cloudflare Bypass
Fresh browser context triggers CF managed-mode evaluation — solves in 4–15s vs 11–18s with alternatives. Cookies are extracted and cached immediately after the first solve.
Captcha Suite auto
Turnstile via shadow DOM click, reCAPTCHA v2 via Google's free audio STT, hCaptcha auto-pass, and GeeTest v4 slide via canvas gap detection. No external solver APIs. No cost per solve.
Session Cache
Cloudflare cookies are stored in Redis per domain after every successful solve. The next request returns in under 500ms — zero re-challenge. 1-hour TTL, automatically refreshed.
Persistent Pool
N browser instances stay warm and ready at all times. A request picks up a browser in ~50ms instead of waiting 3–5 seconds for a cold launch. Domain-sticky routing maximises session reuse.
Proxy Escalation
Datacenter IP for Tier 3 solves, residential proxy for Tier 4. Escalates automatically when an IP is reputation-flagged. Residential bandwidth is only consumed when truly needed.
FlareSolverr Compatible
Implements the FlareSolverr v2 API exactly — including response shape, cookie format, and version string. Prowlarr, Jackett, Sonarr, and Radarr connect with a single URL change.
challenge coverage
every wall. handled.
TRAWL is the only self-hosted scraper with native solvers for all major protection layers — not just Cloudflare interstitials, but the captchas embedded inside the pages themselves.
| challenge type | TRAWL | FlareSolverr | Byparr |
|---|---|---|---|
| Cloudflare Interstitial"Just a moment" redirect page | ✓ 4–15s · fresh context | ~ 11–18s | ~ 13–18s |
| CF Turnstile WidgetEmbedded checkbox on target page | ✓ shadow DOM click · auto | ✗ not handled | ✗ not handled |
| reCAPTCHA v2Google checkbox + audio challenge | ✓ Google STT audio · free | ✗ not handled | ✗ not handled |
| hCaptchaCheckbox + image challenge | ✓ auto-pass path · click | ✗ not handled | ✗ not handled |
| GeeTest v4 SlideDrag-to-fit puzzle captcha | ✓ canvas gap detection · drag | ✗ not handled | ✗ not handled |
| No protectionPlain HTML, no bot check | ✓ < 100ms · plain HTTP | ~ 2–3s · full browser | ~ 2–3s · full browser |
| 6 / 6challenge types | 1 / 6challenge types | 1 / 6challenge types |
how it works
four tiers, one request.
TRAWL tries the cheapest path first and escalates only on failure. The majority of traffic never touches a browser. When it does, the result is cached immediately.
Standard fetch with realistic browser headers — Accept, Accept-Language, Accept-Encoding, and a real Chrome UA. Handles unprotected pages instantly. Zero browser cost.
Restores Cloudflare cookies from Redis into a warm pooled browser and navigates. No challenge re-solve. Returns the page in under a second. Stays cached for 1 hour.
A fresh browser context triggers CF managed-mode — the fastest possible challenge path. Turnstile, reCAPTCHA, hCaptcha, and GeeTest are solved automatically. Cookies are cached on success.
Same as Tier 3 but routes through a residential proxy. Only triggered when the datacenter IP is reputation-flagged by Cloudflare. Bandwidth cost is incurred only when truly necessary.
benchmarks
TRAWL vs the alternatives.
Measured on the same machine, same network, same target URLs. Every benchmark run fresh with no pre-warmed session cache.
| capability | TRAWL | FlareSolverr | Byparr |
|---|---|---|---|
| Persistent browser pool | ✓ N instances | ✗ 1 instance | ✗ 1 instance |
| Domain session cache | ✓ Redis, ~500ms repeat | ✗ always full solve | ✗ always full solve |
| Adaptive tier execution | ✓ 4 tiers | ✗ always browser | ~ |
| Browser engine | ✓ Camoufox Firefox | ✗ Chrome | ✓ Camoufox Firefox |
| Cloudflare challenge speed | ✓ 4–15s | ✗ 11–18s | ✗ 13–18s |
| CF Turnstile solving | ✓ shadow DOM click | ✗ | ✗ |
| reCAPTCHA v2 solving | ✓ audio STT (free) | ✗ | ✗ |
| hCaptcha solving | ✓ auto-pass | ✗ | ✗ |
| GeeTest v4 solving | ✓ canvas gap detection | ✗ | ✗ |
| Proxy escalation | ✓ DC + residential | ✗ | ~ |
| Self-healing pool | ✓ | ✗ | ✗ |
| FlareSolverr v2 compat | ✓ | ✓ native | ✓ |
| External solver APIs | ✓ none required | ✓ none required | ✓ none required |
| Self-hosted | ✓ | ✓ | ✓ |
up and running
in 60 seconds.
No clone. No build. Pull the image and start.
services:
trawl:
image: ghcr.io/germondai/trawl:latest
ports: ["8191:8191"]
shm_size: 1gb
environment:
BROWSER_POOL_SIZE: 1docker compose up -dAPI usage
works with everything.
# FlareSolverr-compatible endpoint (always open — *arr compatible)
curl -s -X POST http://localhost:8191/v1 \
-H "Content-Type: application/json" \
-d '{
"cmd": "request.get",
"url": "https://nowsecure.nl",
"maxTimeout": 60000
}' | jq '.solution.response'
# Native API — richer response with tier, timings, sessionCached
curl -s -X POST http://localhost:8191/scrape \
-H "Content-Type: application/json" \
-d '{"url":"https://nowsecure.nl"}' | jq '{tier,totalMs}'