# NODRIVER-PAGE-CACHING.md — Browser Cache Resource Fetch

> How the crawler reads images, CSS, JS, and fonts from Chromium's cache via CDP.
> Source: `crawler.py:1616-1753` (cache fetch) + `crawler.py:2049-2200` (asset download)

## Overview

The crawler needs resources (images, CSS, fonts, JS) that the page loaded during
rendering. Instead of making HTTP requests from Python, it reads them directly
from Chromium's in-memory/disk cache via the CDP experimental API
`Network.loadNetworkResource`. This is:

- **Faster**: sub-millisecond for cached resources (no network)
- **Self-contained**: same session, same cookies, same headers
- **Already loaded**: the page triggered the downloads during rendering

## Pipeline

```
Page Render
    ↓
Images/CSS/Fonts/JS loaded by Chromium → Browser Cache
    ↓
crawler extracts URLs from HTML DOM
    ↓
_fetch_resource_bytes_cached(tab, url)
    ├── Page.getFrameTree → frame_id
    ├── Network.loadNetworkResource(url, options, frame_id)
    │     └── reads from Chromium cache (not network)
    ├── IO.read(stream) → base64 chunks
    └── IO.close(stream)
    ↓
bytes written to disk, HTML rewritten with local paths
```

## Step-by-Step

### 1. Get Frame ID (`_get_main_frame_id`, line 1623)

```python
from nodriver.cdp.page import get_frame_tree

async def _get_main_frame_id(tab):
    frame_tree = await tab.send(get_frame_tree())
    return frame_tree.frame.id_
```

Caches the frame ID per tab. Needed as `frameId` parameter for `Network.loadNetworkResource`.

### 2. Load from Cache (`_fetch_resource_bytes_cached`, line 1635)

```python
from nodriver.cdp.network import load_network_resource, LoadNetworkResourceOptions
from nodriver.cdp.io import read as io_read, close as io_close

async def _fetch_resource_bytes_cached(tab, full_url,
    allowed_mime_prefixes=("image/",),
    allowed_extensions=(".png", ".jpg", ...),
) -> tuple[bytes, str] | None:

    frame_id = await _get_main_frame_id(tab)
    options = LoadNetworkResourceOptions(disable_cache=False, include_credentials=True)

    # CDP call — reads from browser cache, NOT from network
    result = await tab.send(load_network_resource(full_url, options, frame_id))
    # result: { success: bool, stream: StreamHandle, httpStatusCode: int,
    #            headers: dict, net_error_name: str | None }

    if not result.success:
        return None  # cache miss — caller falls back to network

    if result.stream is None:
        return None

    # Read stream chunks
    chunks = []
    while True:
        base64_encoded, data, eof = await tab.send(io_read(result.stream))
        chunk = base64.b64decode(data) if base64_encoded else data.encode("utf-8")
        chunks.append(chunk)
        if eof:
            break

    await tab.send(io_close(result.stream))
    body = b"".join(chunks)

    # Content-type filtering
    content_type = result.headers.get("content-type", "")
    if allowed_mime_prefixes and content_type:
        if not any(content_type.lower().startswith(p) for p in allowed_mime_prefixes):
            if allowed_extensions and not any(url.lower().endswith(ext) for ext in allowed_extensions):
                return None

    return body, content_type
```

### 3. Fallback: Network Download (`_fetch_resource_bytes`, line 1724)

When the cache fetch returns `None`:

```python
async def _fetch_resource_bytes(tab, full_url,
    allowed_mime_prefixes=("image/",),
    allowed_extensions=(".png", ".jpg", ...),
) -> tuple[bytes, str] | None:

    resp = await asyncio.to_thread(
        lambda: urllib.request.urlopen(
            urllib.request.Request(full_url, headers={"User-Agent": "Mozilla/5.0"}),
            timeout=15,
        )
    )
    content_type = resp.headers.get("Content-Type", "")
    # ... same content-type filtering ...
    return resp.read(), content_type
```

## Content-Type Filtering

The `allowed_mime_prefixes` and `allowed_extensions` parameters control what's accepted:

| Resource | allowed_mime_prefixes | allowed_extensions |
|----------|----------------------|--------------------|
| Images | `("image/",)` | `(".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".avif", ".bmp")` |
| CSS | `("text/css",)` | `(".css",)` |
| JS | `("text/javascript", "application/javascript")` | `(".js",)` |
| Fonts | `None` (all types) | `None` (all types) |

Setting both to `None` accepts ALL content types (used for fonts).

## Content-Type Aware Parameters (added June 2026)

The original function was **hardcoded** for images only:

```python
# BEFORE (broken for CSS/JS):
if content_type and not content_type.lower().startswith("image/") and not any(
    full_url.lower().endswith(ext) for ext in (".png", ".jpg", ...)
):
    return None
```

**Fixed** with parameterized filters (see `crawler.py:1635`).

## Latin-1 → UTF-8 Bug (fixed June 2026)

The CDP `IO.read` returns text resources as strings. The original code did:

```python
# BROKEN — CSS with Unicode arrows/BOM crashed:
chunk = data.encode("latin-1")  # 'latin-1' codec can't encode character '\u2190'
```

Fixed by changing to UTF-8:

```python
# FIXED:
chunk = data.encode("utf-8")
```

Binary resources (images) are always `base64_encoded=True`, so they go through
`base64.b64decode(data)` and are unaffected by this change.

## Static Resource URL Filtering Bug (fixed June 2026)

`_normalized_crawl_url()` at `crawler.py:2437` filters out URLs ending in
`.css`, `.js`, `.woff`, `.ttf`, etc. because it's designed for **page link
discovery** (you don't want to "crawl" CSS files as pages). This silently
blocked the asset extraction functions from finding any CSS/JS URLs.

**Fixed** by adding `allow_static_resources=True` parameter:

```python
def _normalized_crawl_url(href: str, base_url: str, *, allow_static_resources: bool = False):
    ...
    if not allow_static_resources:
        if suffix in _STATIC_RESOURCE_EXTENSIONS:  # .css, .js, .woff, etc.
            return None
```

All extraction functions (`extract_css_links`, `extract_js_links`,
`extract_font_urls_from_css`) now pass `allow_static_resources=True`.

## Asset Download Pipeline (crawler.py:2049-2200)

```python
async def _download_and_rewrite_assets(tab, html, page_url, out_dir, options):
    # 1. Extract URLs from HTML
    css_urls = extract_css_links(html, page_url)     # <link rel=stylesheet>
    js_urls = extract_js_links(html, page_url)       # <script src=>

    for url in css_urls:
        # 2. Try browser cache first
        resource = await _fetch_resource_bytes_cached(tab, url,
            allowed_mime_prefixes=("text/css",), allowed_extensions=(".css",))

        # 3. Retry without query string (?v=...) — cache-buster bypass
        if resource is None and "?" in url:
            resource = await _fetch_resource_bytes_cached(tab, url.split("?")[0], ...)

        # 4. Fallback to network
        if resource is None:
            resource = await _fetch_resource_bytes(tab, url, ...)

        # 5. Save with content-hash name
        fname = _asset_name_from_url_content(url, content, "css")
        (assets_dir / "css" / fname).write_bytes(content)

        # 6. Extract fonts from downloaded CSS (@font-face url(...))
        font_urls = extract_font_urls_from_css(css_text, url)
        for font_url in font_urls:
            font_resource = await _fetch_resource_bytes_cached(tab, font_url, ...)
            (assets_dir / "fonts" / fname).write_bytes(fcontent)
            css_text = css_text.replace(font_url, f"../fonts/{ffname}")

        # 7. Rewrite HTML (both absolute URL and relative href)
        html = html.replace(url, f"assets/css/{fname}")
        html = html.replace(rel_href, f"assets/css/{fname}")

    # 8. Save asset manifest for cross-page dedup
    write_json(assets_dir / "_asset_manifest.json", manifest)
    return html
```

## Asset Manifest Format

```json
{
  "https://www.prolicht.at/frontend.kw/dist/assets/css/application.bundle.css": {
    "file": "css/application.bundle.4fb72ad5.css",
    "size": 415383
  },
  "https://www.prolicht.at/frontend.kw/dist/assets/fonts/GothamNarrow-Book.woff": {
    "file": "fonts/GothamNarrow-Book.64a3ab43.woff",
    "size": 32056
  }
}
```

Deduplication: on subsequent page crawls in the same domain, if a URL is already
in the manifest, the resource is NOT re-downloaded — the local path is reused.

## CDP API Notes

### `Network.loadNetworkResource` (Experimental)

- **CDP level**: Experimental (may change between Chromium versions)
- **Not supported**: returns `"unknown method"` on older Chrome → `_cached_fetch_disabled = True`
- **Recoverable errors**: `net::ERR_FAILED`, `net::ERR_ABORTED` — return None silently
- **Success=false with no netError**: resource simply not in cache — fall back to network
- **Credentials**: `include_credentials=True` passes cookies, which is needed for authenticated origins

### `IO.read` / `IO.close`

- Returns chunks as `(base64_encoded: bool, data: str, eof: bool)`
- Binary data (images): `base64_encoded=True` → `base64.b64decode(data)`
- Text data (CSS, JS): `base64_encoded=False` → `data.encode("utf-8")`
- Must call `IO.close` after reading to release the stream handle

## Troubleshooting

| Symptom | Cause | Solution |
|---------|-------|----------|
| Cache fetch returns success=False | Resource not in Chromium cache | Falls back to urllib network fetch |
| `unknown method` error | Chrome too old for `Network.loadNetworkResource` | `_cached_fetch_disabled` global set to True, all falls back to network |
| Empty body from cache | Resource stream was closed | Falls back to network |
| `latin-1 codec error` (FIXED) | CSS contains Unicode characters | Changed to `data.encode("utf-8")` |
| CSS not extracted (FIXED) | `_normalized_crawl_url` filtered `.css` as static resource | Added `allow_static_resources=True` parameter |
| HTML not rewritten for CSS (FIXED) | Code replaced absolute URLs but HTML had relative paths | Also replace `rel_href` (path+query from original HTML) |
