# NODRIVER-CDP-DIRECT.md — Raw CDP Calls, Cookies, and Frame Management

> Direct CDP protocol calls bypassing nodriver's typed wrappers.
> Source: `crawler.py:630-650` (evaluate Path 3), `crawler.py:814-897` (cookies), `crawler.py:1616-1700` (cache fetch), `server.py:919-967` (selector/extractor tests)

## Why Direct CDP?

Nodriver's typed CDP wrappers (in `nodriver/cdp/`) are auto-generated and
sometimes lack the exact parameters or methods we need. Direct CDP via
`tab.send()` gives full control.

## CDP Patterns Used

### Pattern A: Typed CDP via `from nodriver.cdp.X import function`

```python
from nodriver.cdp.runtime import evaluate as cdp_evaluate
from nodriver.cdp.page import get_frame_tree
from nodriver.cdp.network import load_network_resource, LoadNetworkResourceOptions
from nodriver.cdp.io import read as io_read, close as io_close

# All these are generator functions that yield CDP commands
result = await tab.send(cdp_evaluate(script, ...))
frame_tree = await tab.send(get_frame_tree())
resource = await tab.send(load_network_resource(url, options, frame_id))
chunk = await tab.send(io_read(stream))
```

### Pattern B: Raw CDP via Custom Generator

```python
def _raw_cdp(method: str, params: dict = None):
    """Generator that yields a CDP command dict and receives the response."""
    response = yield {"method": method, "params": params or {}}
    return response

# Usage:
resp = await tab.send(_raw_cdp("Storage.getCookies"))
resp = await tab.send(_raw_cdp("Storage.setCookies", {"cookies": cookies}))
```

This is used when nodriver's typed CDP module doesn't export the needed function
(e.g., `nodriver/cdp/storage.py` lacks `getCookies`/`setCookies`).

## Cookie Persistence (crawler.py:814-897)

### Why This Matters

Chromium cookies need to survive browser restarts. Without persistent cookies,
every crawl session would have to re-accept cookie consent popups, which is slow
and unreliable (Gemini vision calls = API costs).

### Flow

```
On browser start (before any crawl):
  load_browser_cookies(browser, domain)
    └── Storage.setCookies(cookies_from_json) → restore from disk

On browser shutdown (after all crawls):
  save_browser_cookies(browser, domain)
    └── Storage.getCookies() → filter by domain → save to JSON
```

### load_browser_cookies (line 848)

```python
async def load_browser_cookies(browser, domain):
    path = COOKIES_DIR / f"{domain}.json"
    if not path.exists():
        return

    cookies = json.loads(path.read_text())
    # Filter cookies that match this domain
    domain_cookies = [c for c in cookies
        if (c.get("domain") or "").strip(".") == domain.strip(".")]

    # Send to Chromium via raw CDP
    conn = _browser_cdp_connection(browser)
    params = [{"name": c["name"], "value": c["value"], "domain": c.get("domain"),
               "path": c.get("path", "/"), "secure": c.get("secure", False),
               "httpOnly": c.get("httpOnly", False)} for c in domain_cookies]
    await conn.send(_raw_cdp("Storage.setCookies", {"cookies": params}))
```

### save_browser_cookies (line 878)

```python
async def save_browser_cookies(browser, domain):
    conn = _browser_cdp_connection(browser)
    resp = await conn.send(_raw_cdp("Storage.getCookies"))
    cookies = resp.get("cookies", [])

    # Merge with existing file (keep cookies from other domains)
    existing = json.loads(path.read_text()) if path.exists() else []
    existing_by_key = {(c["name"], c.get("domain")): c for c in existing}
    for c in cookies:
        key = (c["name"], c.get("domain"))
        existing_by_key[key] = c

    path.write_text(json.dumps(list(existing_by_key.values()), indent=2))
```

### Finding the CDP Connection

```python
def _browser_cdp_connection(browser):
    """Iterates browser tabs to find a tab with a 'target' attribute for CDP."""
    for tab in browser:
        if hasattr(tab, 'target') and tab.target is not None:
            return tab
    raise RuntimeError("No active CDP connection found")
```

Cookies can only be sent via a tab that has an active CDP WebSocket connection.
The browser's `browser.websocket` is the management connection, but cookie
commands require a page-level connection.

## JavaScript Evaluation via CDP Runtime.evaluate (crawler.py:602-648)

### The Three-Path Strategy

nodriver's `tab.evaluate()` behavior varies between versions. We use three paths:

```python
async def evaluate_value(tab, script):
    # Path 1: return_by_value=True (works most of the time)
    result = await tab.evaluate(script, return_by_value=True, await_promise=False)
    val = _normalize_evaluate_result(result)
    if val is not None:
        return val

    # Path 2: return_by_value=False (returns RemoteObject)
    result = await tab.evaluate(script, return_by_value=False, await_promise=False)
    val = _normalize_evaluate_result(result)
    if val is not None:
        return val

    # Path 3: Raw CDP Runtime.evaluate (fallback with deep serialization)
    result = await tab.send(cdp_evaluate(
        script,
        await_promise=True,
        return_by_value=True,
        user_gesture=True,
        allow_unsafe_eval_blocked_by_csp=True,
        serialization_options={"serialization": "deep"}
    ))
    return _normalize_evaluate_result(result)
```

### Why Path 3 (raw CDP) exists

- Path 1: `tab.evaluate(return_by_value=True)` — nodriver 0.50.3 sometimes returns
  a `RemoteObject` instead of a value for complex objects
- Path 2: `tab.evaluate(return_by_value=False)` — returns RemoteObject reliably but
  requires extracting from the CDP result manually
- Path 3: `tab.send(cdp_evaluate(...))` — direct CDP call with explicit
  `serialization_options={"serialization": "deep"}` to force deep value extraction

### CDP Runtime.evaluate Parameters

```python
cdp_evaluate(
    expression=script,                              # JS code string
    await_promise=True,                             # wait for Promise resolution
    return_by_value=True,                           # return value, not RemoteObject
    user_gesture=True,                              # treat as user-initiated
    allow_unsafe_eval_blocked_by_csp=True,          # bypass CSP restrictions
    serialization_options={"serialization": "deep"}  # deep-serialize to plain value
)
```

## Result Normalization (`_normalize_evaluate_result`, line 550-575)

Handles the many possible shapes nodriver/CDP can return:

```python
def _normalize_evaluate_result(result):
    # CDP: {"result": {"value": ...}}
    if isinstance(result, dict) and "result" in result:
        result = result["result"]

    # ContraDict (nodriver's dict-with-attribute-access)
    value = getattr(result, "value", None)
    obj_type = getattr(result, "type", None)

    # RemoteObject with deep_serialized_value
    if hasattr(result, "deep_serialized_value") and result.deep_serialized_value:
        return result.deep_serialized_value

    # RemoteObject with plain value
    if value is not None and obj_type not in ("object", "function"):
        return value

    # Tuple (unpacked from early nodriver)
    if isinstance(result, tuple):
        return _normalize_evaluate_result(result[0])

    # Direct value
    if result is not None and not _is_remote_like(result):
        return result

    return None
```

## Page Metrics via DOM Evaluation (crawler.py:2230-2250)

```python
async def _page_metrics(tab):
    return await evaluate_value(tab, """
        () => {
            const doc = document.documentElement || {};
            const body = document.body || {};
            return {
                title: document.title || '',
                url: location.href || '',
                viewport: {
                    width: window.innerWidth,
                    height: window.innerHeight
                },
                scroll_height: Math.max(
                    body.scrollHeight || 0,
                    doc.scrollHeight || 0
                ),
                scroll_y: window.pageYOffset || 0
            };
        }
    """)
```

## DOM Content Extraction

| Function | CDP/JS | Purpose |
|----------|--------|---------|
| `get_tab_content()` | `tab.get_content()` → `DOM.getOuterHTML` | Full rendered HTML |
| `get_tab_content()` fallback | `evaluate_value("document.documentElement.outerHTML")` | HTML via JS |
| `_query_visible_images()` | `tab.evaluate(JS)` | DOM query for visible loaded `<img>` elements |
| `extract_text_from_html()` | BeautifulSoup | Plain text from cleaned HTML |
| `clean_html_for_content()` | BeautifulSoup | Remove scripts, styles, nav for content extraction |

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| Path 1 returns RemoteObject not value | nodriver 0.50.x `return_by_value` doesn't deep-serialize | Falls through to Path 2, then Path 3 |
| `serialization_options` parameter error | Older CDP version doesn't support it | Remove the parameter or upgrade Chromium |
| `Storage.getCookies` returns empty | No CDP connection available | Check `_browser_cdp_connection()` finds a tab with `target` |
| `tab.find()` returns None for consent text | Button text is in child element or shadow DOM | Use Gemini vision fallback (detects by pixel position) |
| Cookies not restoring on cold start | `setCookies` requires page domain to match | Ensure `domain` matches cookie's domain field exactly |
