# CLOUDFLARE-PROTECTION CRAWLER REPORT — STARTUP PROMPT

You are an expert Python and Docker crawler developer working on the `D:\Docker\crawler` project.
You know nodriver (Chrome DevTools Protocol wrapper v0.50.3), FastAPI, and Google Gemini Vision API.

## THE PROBLEM

When crawling certain sites (e.g. `www.xal.com`), the browser displays a **Cloudflare
bot protection challenge** page instead of the actual content:

```
www.xal.com
Performing security verification
This website uses a security service to protect against malicious bots.
This page is displayed while the website verifies you are not a bot.
Verify you are human
[CLOUDFLARE logo]
```

The crawler captures the screenshot but cannot detect that the page is blocked.
The screenshot is saved as if it were the real page, and the crawl fails silently.

## EXISTING COOKIE CONSENT DETECTION (reference implementation)

The crawler already handles **cookie consent popups** using Gemini Flash Lite:
- `crawler.py:1058-1190` — `verify_and_accept_cookie_consent()`
- `crawler.py:1195-1250` — `_quick_consent_check()`
- `gemini_detect.py` — Gemini Flash Lite screenshot → overlay detection

The consent flow works like this:
1. Take a viewport screenshot after page load
2. Send to Gemini Flash Lite → returns `{has_cookie_overlay: bool, consent_button_text: str, bounding_box: ...}`
3. Click the consent button at the detected coordinates
4. Take second screenshot → verify consent overlay is gone

## CLOUDFLARE PROTECTION DETECTION (what we need to build)

The same Gemini-based approach can detect Cloudflare challenges:

1. **During consent flow**: Before cookie detection, first check if the page is a
   Cloudflare challenge page. If so, mark the page as "blocked_by_cloudflare" and skip.
2. **Detection prompt**: Send the viewport screenshot to Gemini Flash Lite with a prompt:
   *"Is this a Cloudflare or similar bot-protection security challenge page?
   Does it contain text like 'Verify you are human', 'Security verification',
   'Checking your browser', or a Cloudflare logo? Return JSON: {is_challenge: bool,
   provider: 'cloudflare'|'akamai'|'none', confidence: float, visible_text: string}"*

3. **Image analysis**: Use `gemini_detect.py` function pattern —
   `analyze_page_screenshot(screenshot_path, prompt)` — reusing the existing
   Gemini SDK setup.

## FILES TO MODIFY

| File | Change |
|------|--------|
| `crawler.py` consent flow (~line 1060) | Add challenge detection BEFORE cookie consent: take screenshot, send to Gemini, check for challenge page |
| `gemini_detect.py` | Add new function `detect_cloudflare_challenge(screenshot_path)` — returns `{is_challenge, provider, confidence, visible_text}` |
| `crawler.py` `_fetch_and_save` | If page is Cloudflare-blocked, save marker file and skip |
| `crawler.py` page state | Track `blocked_by_cloudflare` in crawl metadata |

## RELEVANT CODE SNIPPETS

### Existing screenshot for consent (crawler.py ~line 1080):
```python
screenshot_path = screenshots_dir / f"{page_stem}__consent_check_{attempt:02d}_before.png"
await save_viewport_screenshot(tab, screenshot_path, ...)
check = analyze_cookie_consent(screenshot_path)
```

### Existing Gemini detection (gemini_detect.py):
```python
def analyze_cookie_consent(screenshot_path):
    client = genai.Client(api_key=api_key)
    # ... sends image + prompt, parses JSON response
    return {"has_cookie_overlay": bool, "consent_button_text": str, "bounding_box": ...}
```

### XAL page output (what we get now):
```
www.xal.com/screenshots/xal__com__...__viewport_initial.png  ← this is the Cloudflare challenge page
www.xal.com/xal__com__...html  ← contains "Performing security verification"
```

## IMPLEMENTATION STEPS

1. Add `detect_cloudflare_challenge(screenshot_path)` to `gemini_detect.py`
2. In the consent flow, BEFORE cookie detection:
   - Take viewport screenshot
   - Call `detect_cloudflare_challenge()` 
   - If challenge detected → return `{"status": "blocked_by_cloudflare", "provider": "cloudflare", ...}`
3. In `_fetch_and_save()`:
   - Check consent result for `blocked_by_cloudflare`
   - If blocked → save marker file, log warning, return error status
4. Add `blocked_by_cloudflare` to crawl metadata output
5. Test on www.xal.com, verify detection works
