# CLOUDFLARE-PROTECTION CRAWLER REPORT — Database / State Report

## Current State

### Cookie Consent Detection (WORKING)
- Uses Gemini Flash Lite for screenshot-based cookie overlay detection
- Supports text-based consent clicking as fallback
- Saves accepted cookies per domain in `cookies/` directory
- Handles multiple consent patterns (accept all, necessary only, settings)

### Bot Protection Detection (NOT IMPLEMENTED)
- No detection of Cloudflare, Akamai, or other bot protection challenges
- Blocked pages are crawled as normal → screenshots show challenge page
- HTML contains "Performing security verification" text but it's not parsed
- No marker in crawl metadata indicating blocked pages

### Affected Sites
| Site | Protection Type | Screenshot Shows | HTML Contains |
|------|----------------|------------------|---------------|
| www.xal.com | Cloudflare | "Verify you are human" | "Performing security verification" |
| wikipedia.org | robots.txt | N/A (blocked at URL level) | "Blocked by robots.txt" |
| (others TBD) | — | — | — |

## Required Changes

### Database Schema
No new tables needed. Use existing `crawl_metadata` (crawl.json) fields:

```json
{
  "blocked_by_cloudflare": true,
  "cloudflare_provider": "cloudflare",
  "cloudflare_confidence": 0.95,
  "cloudflare_detected_text": "Verify you are human - Cloudflare"
}
```

### Cookie State
No changes needed — Cloudflare challenges are not cookie-based.

### API Response
Add `blocked_reason` field to crawl responses:
```json
{
  "status": "blocked",
  "blocked_reason": "cloudflare",
  "blocked_provider": "cloudflare",
  "blocked_confidence": 0.95
}
```

### Files Created
| File | Purpose |
|------|---------|
| `sites_crawled/<domain>/<page>_blocked.json` | Marker file when page is blocked |
| `crawl.json` field `blocked_by_cloudflare` | Boolean flag in crawl metadata |

## Gemini API Usage

### New Function Signature
```python
def detect_cloudflare_challenge(screenshot_path: str) -> dict:
    """Returns {is_challenge: bool, provider: str, confidence: float, visible_text: str}"""
```

### Prompt Template
```
Is this a Cloudflare, Akamai, or similar bot-protection security challenge page?
Does it contain any of: "Verify you are human", "Security verification",
"Checking your browser", "Cloudflare", "Akamai", bot-protection logos?

Return ONLY this JSON:
{
  "is_challenge": true/false,
  "provider": "cloudflare" | "akamai" | "none",
  "confidence": 0.0-1.0,
  "visible_text": "exact security text visible"
}
```

### Cost Estimate
Same as cookie consent detection: ~$0.0001 per image ($0.10 per 1000 pages).
Only triggered on pages that pass the initial cookie consent check.
