# NODRIVER.md - crawler-specific nodriver notes

This file records nodriver usage rules for the crawler so future edits do not drift back to outdated raw-CDP patterns.

## Official references

- Main docs: https://ultrafunkamsterdam.github.io/nodriver/
- Browser class: https://ultrafunkamsterdam.github.io/nodriver/nodriver/classes/browser.html
- Tab class: https://ultrafunkamsterdam.github.io/nodriver/nodriver/classes/tab.html
- PyPI package: https://pypi.org/project/nodriver/


## Browser startup and viewport

Use documented nodriver startup options before falling back to lower-level CDP patterns:

```python
browser = await uc.start(
    headless=headless,
    browser_args=[
        f"--window-size={viewport_width},{viewport_height}",
        "--window-position=0,0",
        "--force-device-scale-factor=1",
    ],
)
```

The official nodriver quickstart documents `uc.start(...)` with `browser_args`, and the Tab API documents `set_window_size`, `set_window_state`, `save_screenshot`, `get_content`, `find`, and `mouse_click`. Keep crawler code on these public helpers unless a feature is not exposed there.

## Browser role in this crawler

Nodriver is used only for the live browser phase:

```text
open page -> accept consent -> wait for rendered content -> get rendered HTML -> capture screenshots
```

The crawler does not use local `file://` HTML as source for screenshots or Markdown.

## Cookies

Use browser cookie helpers:

```python
await browser.cookies.load(path)
await browser.cookies.save(path)
cookies = await browser.cookies.get_all(requests_cookie_format=True)
```

Avoid raw CDP dictionaries:

```python
await tab.send({"method": "Network.getCookies", "params": {}})
await tab.send({"method": "Network.setCookie", "params": {...}})
```

## Navigation

Prefer documented helpers:

```python
tab = await browser.get(url)
await tab.get(url)
await tab.sleep(1)
```

## Consent lookup and click

Preferred DOM text click:

```python
button = await tab.find("ALLE AKZEPTIEREN", best_match=True, timeout=2)
await button.click()
```

Fallback viewport pixel click:

```python
await tab.mouse_click(x, y)
```

Pixel coordinates must come from a viewport screenshot, not a full-page screenshot.

## Fresh nodriver docs checked 2026-06-09

The official docs currently document:

```python
browser = await uc.start(headless=headless, browser_args=[...])
tab = await browser.get(url)
await browser.cookies.load(path)
await browser.cookies.save(path)
await tab.get(url)
html = await tab.get_content()
await tab.save_screenshot(path, format="png", full_page=False)
await tab.set_window_size(left=0, top=0, width=1920, height=1080)
await tab.set_window_state(left=0, top=0, width=1920, height=1080, state="normal")
```

Use those helpers first. Only use `tab.evaluate(...)` as a logged fallback when a documented helper times out or returns unusable data.

## Screenshots

Before screenshots:

```python
await tab.set_window_state(left=0, top=0, width=1920, height=1080, state="normal")
await tab.set_window_size(left=0, top=0, width=1920, height=1080)
```

Then verify:

```python
await tab.evaluate("() => ({width: window.innerWidth, height: window.innerHeight})", return_by_value=True)
```

Use viewport screenshots for consent detection:

```python
await tab.save_screenshot("consent_before.png", format="png", full_page=False)
```

For page capture, the crawler takes viewport tiles after consent is gone. If `save_screenshot(full_page=False)` returns a PNG whose pixel size does not match the verified viewport, log the mismatch, retry once after `set_window_size`/`set_window_state`, and log the retry result.

## Evaluate

For simple expressions (no arrow functions, no promises):
```python
result = await tab.evaluate("document.title", return_by_value=True)
```

For arrow functions and complex scripts that return values, use raw CDP runtime.evaluate:
```python
from nodriver.cdp.runtime import evaluate as cdp_evaluate
raw = await tab.send(cdp_evaluate(script, await_promise=True, return_by_value=True))
result = raw.get("result", {}).get("value") if isinstance(raw, dict) else None
```

**GOTCHA**: `tab.evaluate("() => expr", return_by_value=True)` returns the function
RemoteObject instead of executing it. Always use `cdp_evaluate` with `await_promise=True`
for arrow functions and complex JS that returns values.

Avoid raw Runtime CDP dictionaries outside of the cdp_evaluate helper.

## Image downloads

The crawler can use browser-side `fetch(..., {credentials: "include"})` from `tab.evaluate(...)` because some images may depend on the current session/cookies.

Only image assets are in scope for the current target. CSS and JS mirroring is not required.

## Current guarantees

- Browser session is reused across all pages in one crawl.
- Consent is verified from screenshots before final artifacts.
- Cookies are saved immediately after verified consent success.
- Final HTML/TXT/Markdown are created from post-consent rendered DOM.
- Screenshots are captured after consent.
- Images referenced by Markdown are downloaded under `images/` and rewritten to local paths.
