# NODRIVER-AGENTS.MD — Nodriver Integration in Crawler App

> **For LLM agents coding with nodriver in this project.**
> Source: `D:\Docker\crawler\documentation\nodriver-source-gitlab\nodriver\`

## Architecture

```
CrawlerWorker (server.py:247)
  └── daemon thread running uc.loop()
       └── all crawl operations submitted via asyncio.run_coroutine_threadsafe()
            └── browser.get(url) → Tab → HTML, screenshots, images, interactions
```

The `CrawlerWorker` owns the **single** nodriver event loop in a daemon thread.
FastAPI handlers call `await worker.submit(coroutine)`. Never call
`uc.loop().run_until_complete()` or `asyncio.run()` inside a request handler.

## Package: `nodriver` (v0.50.3)

```python
import nodriver as uc

# Key exports:
uc.start()          # launch Chromium → Browser
uc.loop()           # new asyncio event loop
uc.Browser          # browser instance
uc.Tab              # tab instance (main API surface)
uc.cdp              # all CDP domains
```

## Top-Level Imports Used

```python
import nodriver as uc                                          # crawler.py, server.py
from nodriver.cdp.runtime import evaluate as cdp_evaluate      # crawler.py, server.py, browser_sessions.py
from nodriver.cdp.page import get_frame_tree                   # crawler.py
from nodriver.cdp.network import load_network_resource, LoadNetworkResourceOptions  # crawler.py
from nodriver.cdp.io import read as io_read, close as io_close  # crawler.py
```

## Module Map

```
nodriver/
├── __init__.py              # uc.start(), uc.loop(), Browser, Tab, cdp, Config
├── core/
│   ├── browser.py           # Browser class — launch, get(url), stop(), tabs, cookies
│   ├── tab.py               # Tab class — get(), evaluate(), send(), save_screenshot(), sleep(), find(), select(), get_content(), mouse_click(), close()
│   ├── element.py           # Element — click(), mouse_click(), save_screenshot(), flash()
│   ├── config.py            # Config — chromium args, port, user_data_dir
│   ├── connection.py        # Connection base — send(cdp_command), add_handler()
│   ├── _contradict.py       # ContraDict — dict with attribute access (for CDP results)
│   └── util.py              # start(), loop() helpers
└── cdp/                     # 50+ typed CDP domain command generators
    ├── runtime.py           # Runtime.evaluate
    ├── page.py              # Page.getFrameTree, Page.captureScreenshot
    ├── network.py           # Network.loadNetworkResource
    ├── io.py                # IO.read, IO.close
    ├── storage.py           # Storage.getCookies, Storage.setCookies
    ├── input_.py            # Input.dispatchMouseEvent
    ├── dom.py               # DOM.getOuterHTML, DOM.performSearch
    ├── target.py            # Target.createTarget, Target.closeTarget
    ├── browser.py           # Browser.setWindowBounds, Browser.setDownloadBehavior
    └── emulation.py         # Emulation.setDeviceMetricsOverride
```

## Browser Lifecycle (crawler.py)

```python
# 1. LAUNCH
browser = await uc.start(
    headless=headless,
    browser_args=["--no-sandbox", "--disable-dev-shm-usage", ...],
    browser_executable_path=chrome_path,
    user_data_dir=profile_dir,          # persistent profile for cookies
)

# 2. CONSENT HANDLING (first tab)
tab = await browser.get(first_url)
await set_and_verify_viewport(tab, 1920, 1080)
await cookie_consent_flow(tab)          # see NODRIVER-CONSENT.md
save_browser_cookies(browser, domain)

# 3. CRAWL PAGES (new tabs per URL)
for url in urls:
    tab = await browser.get(url, new_tab=True)   # fresh tab
    await _fetch_and_save(tab, url, out_dir, options)
    await close_tab_safely(tab, url)

# 4. SHUTDOWN
browser.stop()
```

## Tab Methods Used by Frequency

| Method | Count | Primary Purpose |
|--------|-------|-----------------|
| `tab.sleep(s)` | 30+ | DOM settle, consent wait, scroll settle, image loading |
| `tab.evaluate(js, return_by_value=True)` | 20+ | JS execution: DOM queries, scroll, metrics, interaction discovery |
| `tab.send(cdp_command)` | 15+ | Direct CDP: Runtime.evaluate, Network.loadNetworkResource, Storage.* |
| `tab.save_screenshot(path)` | 8+ | Consent before/after, viewport tiles, full-page |
| `tab.get(url)` | 3+ | Primary navigation, page restore |
| `browser.get(url, new_tab=True)` | 5+ | New tab creation for crawl |
| `tab.get_content()` | 1+ | HTML capture via DOM.getOuterHTML |
| `tab.find(text)` | 1 | Consent button detection |
| `tab.mouse_click(x, y)` | 2 | Consent at Gemini-detected coords, interaction click |
| `tab.set_window_state/size()` | 2 | Viewport configuration |
| `tab.close()` | 1 | Tab cleanup |

## CDP Commands Sent via tab.send()

| CDP Domain | Method | Purpose |
|------------|--------|---------|
| Runtime | `evaluate` | JS execution with serialization options (Path 3 fallback) |
| Page | `getFrameTree` | Get frame ID for Network.loadNetworkResource |
| Network | `loadNetworkResource` | Read resource from browser cache (experimental API) |
| IO | `read`, `close` | Stream resource bytes from cache |
| Storage | `getCookies`, `setCookies` | Cookie persistence (via raw CDP generator, not typed) |

## Key Patterns

### Pattern 1: Three-Path JS Evaluation (evaluate_value)

```python
# crawler.py:602-648
async def evaluate_value(tab, script: str):
    # Path 1: tab.evaluate(return_by_value=True, await_promise=False)
    result = await tab.evaluate(script, return_by_value=True, await_promise=False)
    normalized = _normalize_evaluate_result(result)
    if normalized is not None: return normalized

    # Path 2: tab.evaluate(return_by_value=False) — get RemoteObject
    result = await tab.evaluate(script, return_by_value=False, await_promise=False)
    normalized = _normalize_evaluate_result(result)
    if normalized is not None: return normalized

    # Path 3: Raw CDP Runtime.evaluate with serialization_options
    result = await tab.send(cdp_evaluate(
        script, await_promise=True, return_by_value=True,
        user_gesture=True, allow_unsafe_eval_blocked_by_csp=True,
        serialization_options={"serialization": "deep"}
    ))
    return _normalize_evaluate_result(result)
```

Why three paths: nodriver's `tab.evaluate()` behavior varies between versions.
Path 1 (`return_by_value=True`) works most of the time but can return `RemoteObject`
instead of values. Path 2 returns `RemoteObject` which `_normalize_evaluate_result`
unpacks. Path 3 is the CDP-level fallback with explicit `serialization="deep"`.

### Pattern 2: Browser-Cached Resource Fetch

```python
# crawler.py:1635-1721 — see NODRIVER-PAGE-CACHING.md for full details
frame_id = await _get_main_frame_id(tab)           # Page.getFrameTree
result = await tab.send(load_network_resource(      # Network.loadNetworkResource
    full_url, LoadNetworkResourceOptions(disable_cache=False), frame_id))
while not eof:
    _, data, eof = await tab.send(io_read(stream))  # IO.read chunks
    chunk = base64.b64decode(data)                   # base64 → bytes
await tab.send(io_close(stream))                     # IO.close
```

### Pattern 3: Raw CDP for Cookies

```python
# crawler.py:814-897 — cookies don't use typed CDP, they use raw generator
def _raw_cdp(method, params=None):
    response = yield {"method": method, "params": params or {}}
    return response

# Read cookies:
resp = await tab.send(_raw_cdp("Storage.getCookies"))

# Write cookies:
resp = await tab.send(_raw_cdp("Storage.setCookies", {"cookies": params}))
```

This avoids the typed CDP wrappers because nodriver's `storage.py` module
doesn't export `getCookies`/`setCookies` as function generators.

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| `tab.evaluate()` returns `RemoteObject` | nodriver version mismatch | Use Path 3 raw CDP fallback |
| `tab.save_screenshot(full_page=True)` fails | Old nodriver signature | Fall back to `format="png"` only |
| `Network.loadNetworkResource` returns `success=False` | Resource not in cache, experimental API | Fall back to `urllib.request` |
| `await tab` hangs forever | Session dead | Use `asyncio.wait_for()` with timeout |
| `latin-1 codec error` in `io_read` | CSS/JS contains Unicode | Changed to `data.encode("utf-8")` in `crawler.py:1692` |
