ultimate-scraper

name: ultimate-scraper description: Scrapes web pages with intelligent tier escalation and AI extraction. Use when user provides a URL and needs to extract content, bypass anti-bot protection, or parse protected pages. Handles static data extraction (NEXT_DATA, JSON-LD), TLS fingerprint spoofing, stealth browsers (CloakBrowser + Patchright), Cloudflare bypass, CAPTCHA solving, proxy rotation, rate limiting, session persistence, fingerprint persistence, visual extraction, behavioral simulation, tracker blocking, shadow DOM piercing, WebMCP extraction, and LLM-powered data extraction. allowed-tools: Bash(python*) version: 2.0.0 compatibility: python>=3.8 triggers: - scrape - extract - crawl - fetch - web scraping - anti-bot - cloudflare bypass

Ultimate Web Scraper

Quick Start

python scripts/scrape.py "https://example.com"

With AI extraction:

python scripts/scrape.py "https://example.com" \
  -e "Extract all product names and prices" -o json

Workflow

1. Basic Scrape

python scripts/scrape.py "URL"

Output: Markdown content to stdout.

2. With AI Extraction

python scripts/scrape.py "URL" \
  -e "Natural language instruction" -o json

Output: JSON with data field containing extracted info.

3. Protected Sites

python scripts/scrape.py "URL" \
  -m stealth -g us

Output: Content fetched via Camoufox + US residential proxy.

4. Handle Failures

If exit code 2 (premium proxy required), use your preferred unlocker/proxy service.

Decision Tree

User wants to scrape?
├─ Simple page, no anti-bot → python scripts/scrape.py "URL"
├─ Need specific data → python scripts/scrape.py "URL" -e "Extract X" -o json
├─ Protected site → python scripts/scrape.py "URL" -m stealth -g us
├─ Multiple URLs → python scripts/scrape.py URL1 URL2 -p 10
├─ URLs from file → python scripts/scrape.py --batch urls.jsonl -p 10
├─ Save to separate files → python scripts/scrape.py --batch urls.txt --output-dir ./articles/
├─ Long-running job → python scripts/scrape.py --batch FILE --output-dir ./ --checkpoint job.json
├─ Exit code 1 + NotFound → Page doesn't exist (404), no action needed
├─ Exit code 2 → Requires premium proxy/unlocker service
└─ Detect protection → python scripts/scrape.py "URL" --probe-only

CLI Options

Architecture

Tier System

Auto-escalation: Tier N fails → rotate proxy → retry → escalate to N+1.

JA4T Detection

Sites using transport-layer fingerprinting (JA4T) are automatically detected. When JA4T is detected:

• Tier 1 (HTTP with TLS spoofing) is skipped
• Starts at Tier 2 (browser) minimum
• Verbose mode shows: [JA4T] Skipping Tier 1 - transport-layer fingerprinting detected

CAPTCHA Solving

When a CAPTCHA is encountered at Tier 2 or 3:

1. Detects type (reCAPTCHA v2/v3, hCaptcha, Cloudflare Turnstile)
2. Extracts sitekey from page
3. Sends to CapSolver (AI, ~5s) or 2Captcha (human, ~30s) for solving
4. Injects token into page and continues

Requires CAPSOLVER_API_KEY or TWOCAPTCHA_API_KEY env var. Without keys, CAPTCHAs trigger tier escalation as before.

CloakBrowser

Tier 2 prefers CloakBrowser (C++ patched Chromium with 26 source-level patches) over Patchright. Falls back to Patchright if CloakBrowser is not installed. Set CLOAKBROWSER_ENABLED=0 to force Patchright.

Per-Domain Rate Limiting

Browser tiers (2+) enforce per-domain rate limits via sliding window (60s). Default: 8 req/min. Sensitive sites have lower limits (LinkedIn: 4, Instagram: 4, Facebook: 5). Disable with --no-rate-limit.

Sensitive Site Mode

Sites like LinkedIn, X/Twitter, Facebook, Instagram, TikTok are auto-detected as sensitive:

• Minimum tier 2 (browser) in auto mode
• Fingerprint rotation locked (no rotation on block)
• Behavior intensity boosted (minimum 1.3x)
• Rate limits enforced at lower thresholds

Tier History

Successful tier usage is tracked per domain. On repeat visits, auto mode starts at the lowest known-good tier (requires 3+ successes with 80%+ success ratio). This skips unnecessary lower-tier attempts.

Fingerprint Persistence

Consistent browser fingerprints per domain:

• Same browser/version for each domain (not randomized)
• Fingerprints stored in SQLite, linked to sessions
• Auto-rotation after blocks or 30+ days
• Market-share weighted browser selection by geo

LLM Routing (--extract)

The default fallback chain reflects the developer's setup. Models and providers are user preference — modify scripts/extraction/ai_router.py to wire in your own. Any OpenAI-compatible API works for Tier 1.

Recommended local model: GLM-4.7-Flash-UD Q4 via vLLM/llama.cpp, or any instruction-following model with JSON output (Qwen 2.5, Llama 3.1, Mistral, etc.).

1. Local LLM (any OpenAI-compatible API) → configure via LOCAL_LLM_URL
   ↓ unavailable
2. z.ai GLM-4.5-Air → configure via ZAI_API_KEY
   ↓ rate limited
3. Claude Haiku → configure via ANTHROPIC_API_KEY

Error Handling

404 Detection

Pages that don't exist are detected via:

1. HTTP 404 status code (99% confidence)
2. Content patterns ("page not found", "404", etc.) + HEAD request verification

When detected:

• Marked as failed with error_type="NotFound"
• Does NOT trigger fallback (page genuinely doesn't exist)
• Skipped in batch processing (won't waste retries)

Exit Codes

Output Formats

markdown (default)

Clean markdown converted from HTML via html2text. Best for reading/summarizing.

json

{
  "success": true,
  "url": "...",
  "final_url": "...",
  "tier_used": 1,
  "data": { ... },
  "static_data": { ... },
  "markdown": "...",
  "metadata": { ... }
}

raw

Raw HTML. Use when processing HTML directly.

Examples

E-commerce Product

python scripts/scrape.py \
  "https://amazon.com/dp/B0ABC123" \
  -m stealth -g us \
  -e "Extract product name, price, rating, review count" -j

News Article (Potential Paywall)

python scripts/scrape.py \
  "https://nytimes.com/article" \
  -e "Extract title, author, date, full text" -j

If exit code 2: use premium proxy/unlocker service.

Recipe (Structured Data)

python scripts/scrape.py \
  "https://recipe-site.com/cake" \
  -m static -j

Returns JSON-LD recipe schema if available.

Visual Extraction (Screenshot + Vision LLM)

For heavily protected pages or canvas-rendered content:

python scripts/scrape.py \
  "https://protected-site.com/data" \
  --visual -e "Extract all prices and product names visible on the page" -j

Visual extraction:

• Takes full-page screenshot using Tier 3 (Camoufox)
• Extracts data using Vision LLM
• Bypasses DOM monitoring/mutation detection
• Works on canvas-rendered or heavily obfuscated pages

Multi-page Session

# Login and save session
python scripts/scrape.py \
  "https://site.com/login" --session acct -m stealth \
  --actions '[{"type":"fill","selector":"#email","text":"user@example.com"},{"type":"fill","selector":"#pass","text":"xxx"},{"type":"click","selector":"#submit"}]'

# Reuse session
python scripts/scrape.py \
  "https://site.com/dashboard" --session acct -e "Extract user data" -j

Batch Processing

# From command line args
python scripts/scrape.py \
  URL1 URL2 URL3 -p 10 --batch-output results.jsonl

# From file (JSONL or plain text)
python scripts/scrape.py \
  --batch urls.jsonl -p 10 --batch-output results.jsonl

# Write each result to separate markdown file (filename from URL slug)
python scripts/scrape.py \
  --batch urls.txt --output-dir ./articles/ -p 10

# Write as JSON files instead
python scripts/scrape.py \
  --batch urls.txt --output-dir ./data/ --output-ext json -o json

Streaming Output (Long-Running Jobs)

# Stream results as they complete + checkpoint for resume
python scripts/scrape.py \
  --batch urls.jsonl \
  --output-stream results.jsonl \
  --checkpoint job1.json \
  -p 10 -v

# Resume interrupted job (skips already-processed URLs)
python scripts/scrape.py \
  --batch urls.jsonl \
  --output-stream results.jsonl \
  --checkpoint job1.json \
  -p 10

Cross-tool Workflow (agent-browser state import)

# Login with agent-browser (or any tool that exports state)
# ... login flow ...

# Scrape with imported state
python scripts/scrape.py \
  "https://site.com/dashboard" \
  --session imported --import-state ~/auth.json \
  -e "Extract user data" -j

Cache

• Location: ~/.cache/ultimate-scraper/cache.db
• TTL: 24 hours
• Bypass: --no-cache
• Key includes: URL + mode + extract_prompt (different prompts = different entries)

Dependencies

Core (all tiers):

• Python 3.8+
• httpx (pip install httpx) — async HTTP client
• beautifulsoup4 (pip install beautifulsoup4) — HTML parsing
• lxml (pip install lxml) — fast XML/HTML parser
• html2text (pip install html2text) — HTML→Markdown conversion
• pyyaml (pip install pyyaml) — YAML config loading
• python-dotenv (pip install python-dotenv) — .env file loading

Tier 0 — Static extraction:

• chompjs (pip install chompjs) — JavaScript object→Python dict parsing
• extruct (pip install extruct) — JSON-LD, Microdata, OpenGraph extraction

Tier 1 — HTTP with TLS spoofing:

• curl_cffi (pip install curl_cffi) — HTTP client with browser TLS fingerprint impersonation

Tier 2 — Stealth browser:

• scrapling (pip install scrapling) — Patchright-based stealth browser automation
• cloakbrowser (pip install cloakbrowser) — 26 C++ source-level Chromium patches (preferred over Patchright). Binary auto-downloads ~200MB on first use. Set CLOAKBROWSER_ENABLED=0 to force Patchright fallback.

Tier 3 — Anti-detect browser:

• camoufox (pip install camoufox[geoip] && python -m camoufox fetch) — C++ anti-detect Firefox with hardware-backed fingerprinting. First run downloads ~780MB browser package.

Tier 4 — AI extraction:

• crawl4ai (pip install crawl4ai) — AI-powered web crawling with LLM integration

Test dependencies:

• pytest, pytest-asyncio, pytest-cov, scipy, httpx

Install order:

# 1. Core
pip install httpx beautifulsoup4 lxml html2text pyyaml python-dotenv

# 2. Static extraction (Tier 0)
pip install chompjs extruct

# 3. HTTP tier (Tier 1)
pip install curl_cffi

# 4. Browser tiers (Tier 2-3)
pip install scrapling cloakbrowser
pip install 'camoufox[geoip]' && python -m camoufox fetch

# 5. AI tier (Tier 4)
pip install crawl4ai

# 6. Test dependencies (optional)
pip install pytest pytest-asyncio pytest-cov scipy

WSL2 Known Issues

Tier 3 on WSL2 is unreliable for Turnstile-protected sites. WSL2's virtual GPU produces inconsistent canvas, WebGL, and audio fingerprints. Tiers 0-2 and 4-5 work normally on WSL2.

Environment Variables

See .env.example for all configurable options. Key variables:

# Proxy (bring your own)
PROXY_HOST=                    # Proxy hostname
PROXY_PORT=                    # Proxy port
PROXY_USERNAME=                # Auth username
PROXY_PASSWORD=                # Auth password

# AI extraction (optional)
LOCAL_LLM_URL=                 # OpenAI-compatible endpoint
LOCAL_LLM_ENABLED=false        # Enable local LLM
ZAI_API_KEY=                   # z.ai API key
ANTHROPIC_API_KEY=             # Anthropic API key

# CAPTCHA solving (optional)
CAPSOLVER_API_KEY=             # AI-based solver (faster)
TWOCAPTCHA_API_KEY=            # Human workers fallback

# CloakBrowser — auto|1|0
CLOAKBROWSER_ENABLED=auto

# WebMCP — auto|1|0
WEBMCP_ENABLED=auto
CHROME_CHANNEL=                # chrome-dev, chrome-beta, chrome-canary

Testing

# Run fast tests (unit + integration, ~5s)
cd scripts
python -m pytest tests/ -v

# Run all tests including E2E (~15s)
python -m pytest tests/ -m "" -v

# Run with coverage report
python -m pytest tests/ --cov=. --cov-report=html

Test Categories

Test Dependencies

See Dependencies section above for install commands.