# Cloudflare Bot Protection Detection — Implementation Plan Draft

## Problem

Sites behind Cloudflare bot protection (e.g. `www.xal.com`) return a security challenge page instead of real content. The crawler captures this as legit content, wasting crawl time and producing useless screenshots + HTML.

**Observed behavior on xal.com:**
- Screenshot shows: "Performing security verification... Verify you are human [Cloudflare]"
- HTML content: `"This website uses a security service to protect against malicious bots... Verify you are human... CLOUDFLARE Privacy"`
- Crawl result: "ok" status, but page content is useless

## Solution: Gemini Vision Detection

Same pattern as existing **cookie consent detection** — use Gemini Flash Lite to analyze the viewport screenshot and detect bot protection challenges.

### Flow

```
Page Load
  ↓
Take viewport screenshot (already happens for consent)
  ↓
Send to Gemini: "Is this a bot protection challenge page?"
  ↓
If YES → mark page as blocked_by_cloudflare, skip crawl
If NO  → continue with normal consent flow
```

### Gemini Prompt

```
Is this a Cloudflare, Akamai, or similar bot-protection security challenge page?
Does it contain any of: "Verify you are human", "Security verification",
"Checking your browser", "Cloudflare", "Akamai" logo/branding?

Return ONLY this JSON:
{
  "is_challenge": true/false,
  "provider": "cloudflare"|"akamai"|"other"|"none",
  "confidence": 0.0-1.0,
  "visible_text": "text visible on the challenge page"
}
```

### Code Changes

#### 1. `gemini_detect.py` — New function

```python
def detect_cloudflare_challenge(screenshot_path: str) -> dict:
    """Analyze screenshot for bot protection challenge pages.
    Returns {is_challenge, provider, confidence, visible_text}."""
    client = _get_gemini_client()
    response = client.models.generate_content(
        model=GEMINI_MODEL,
        contents=[_build_prompt(), _load_image(screenshot_path)]
    )
    return _parse_challenge_response(response.text)
```

#### 2. `crawler.py` consent flow (~line 1060)

Before cookie consent detection:

```python
# Check for Cloudflare/bot protection BEFORE cookie consent
screenshot_path = screenshots_dir / f"{page_stem}__challenge_check.png"
await save_viewport_screenshot(tab, screenshot_path, ...)
challenge = detect_cloudflare_challenge(screenshot_path)

if challenge.get("is_challenge"):
    log.info("Cloudflare/%s challenge detected for %s (confidence: %.2f)",
             challenge.get("provider"), url, challenge.get("confidence"))
    return {"status": "blocked_by_cloudflare", **challenge}
```

#### 3. `crawler.py` — `_fetch_and_save()`

Handle the blocked status in the crawl pipeline (~line 3060):

```python
consent_result = await verify_and_accept_cookie_consent(...)
if consent_result.get("status") == "blocked_by_cloudflare":
    page_url = url
    log.warning("Page blocked by %s: %s", consent_result.get("provider"), page_url)
    # Save blocked marker + minimal metadata
    write_json(out_dir / f"{page_stem}_blocked.json", {
        "blocked_by": "cloudflare",
        "provider": consent_result.get("provider", "unknown"),
        "confidence": consent_result.get("confidence", 0),
        "detected_text": consent_result.get("visible_text", ""),
        "url": url,
        "timestamp": utc_now_iso(),
    })
    return False, None
```

#### 4. Crawl metadata

Add blocked status to `crawl.json`:
```json
{
  "blocked_by_cloudflare": true,
  "cloudflare_provider": "cloudflare",
  "cloudflare_confidence": 0.95
}
```

### Performance

- One additional Gemini call per page (same as consent: ~1-2 seconds, ~$0.0001)
- Total page time increase: ~1.5s (screenshot + API call)
- Can be disabled via `--no-cloudflare-check` CLI flag for known-clean sites

### Edge cases

| Scenario | Handling |
|----------|----------|
| Challenge page loads but Gemini returns false negative | Retry with second prompt ("What specific text is visible?"), use keyword matching as fallback |
| Challenge resolves after delay (CAPTCHA solved) | Not handled — if the JS succeeds, page is normal |
| Site uses Akamai instead of Cloudflare | Generic detection prompt covers both |
| Cookie consent AND Cloudflare on same page | Check Cloudflare FIRST (challenge pages don't have consent) |

### Files Modified

| File | Lines | Change |
|------|-------|--------|
| `gemini_detect.py` | +30 | New `detect_cloudflare_challenge()` function |
| `crawler.py` consent flow | +15 | Call challenge detection before consent |
| `crawler.py` `_fetch_and_save` | +10 | Handle `blocked_by_cloudflare` status |
| `crawler.py` CLI args | +3 | `--no-cloudflare-check` flag |
| `server.py` models | +3 | `blocked_reason` in response |

### Estimated Effort: ~3 hours

- Gemini prompt tuning: 1h
- Detection function: 0.5h
- Integration into consent flow: 0.5h
- Testing on xal.com + edge cases: 1h
