# IMPLEMENTATION PLAN — Nice Local HTML View (CSS + JS + Font Assets)

## Problem (verified against actual crawled page)

Inspected `prolicht__at__de__inspiration__projekte__btv-innsbruck.html`:

### What works
- `<img src="images/...">` paths correctly rewritten to local
- Base HTML structure intact

### What fails (page looks broken locally)

| Missing | Example | Impact |
|---------|---------|--------|
| **CSS** | `href="/frontend.kw/dist/assets/css/application.bundle.css"` | Page completely unstyled |
| **Fonts** | 14x `<link rel="prefetch">` (.otf/.woff/.ttf) | Wrong fallback font |
| **JS** | None visible (injected by CSS bundle dynamically) | No nav, no carousel |
| **Lazy images** | `bttrlazyloading-wrapper` placeholders | `data-bttr-src` never swapped |

### Root cause: page not scrolled + assets not cached long enough

The crawler captures HTML *immediately* after DOM settles. Prolicht's site:
1. CSS bundle injects `@font-face` — font downloads may not finish
2. Carousel images use `data-bttr-src` swapped by JS — crawler doesn't scroll
3. Dynamic `<script>` injection from the CSS bundle

## Goal

Local double-click renders visually close to live. CSS/fonts/JS from browser cache.

## Phase 0 — One Scroll Serves Both (NOT separate)

Current order in `_fetch_and_save()`:
```
_fetch_live_page()          ← HTML captured HERE, no scroll
download_and_rewrite_images()
capture_page_screenshots()  ← scroll happens HERE, too late for HTML
```

Fix: scroll before `get_tab_content()` at `crawler.py:2833`. The scroll in
`capture_page_screenshots()` (line 2093) becomes a harmless no-op.

```python
# In _fetch_live_page(), after wait_for_rendered_content_stable():
metrics = await wait_for_rendered_content_stable(tab, ...)

# Scroll triggers lazy images (bttrlazyloading, carousels) + loads fonts
await _scroll_to(tab, 99999)
await tab.sleep(2.0)
await _scroll_to(tab, 0)
await tab.sleep(0.5)

html = await get_tab_content(tab, ...)  # DOM now fully loaded
```

Total change: ~5 lines in `_fetch_live_page()`. No changes to screenshot code.

## Phase 1 — Browser Cache Fetch for All Resources

The existing `_fetch_resource_bytes_cached()` at `crawler.py:1635` uses CDP
`Network.loadNetworkResource` to read ANY resource from the browser cache.
Currently filtered to images only at lines 1708-1712.

**Change**: Add `allowed_mime_prefixes` / `allowed_extensions` parameters:

```python
async def _fetch_resource_bytes_cached(
    tab, full_url: str,
    allowed_mime_prefixes: tuple[str, ...] = ("image/",),
    allowed_extensions: tuple[str, ...] = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".avif", ".bmp"),
) -> tuple[bytes, str] | None:
```

When called for CSS: `allowed_mime_prefixes=("text/css",)`, `allowed_extensions=(".css",)`
When called for fonts: no mime filter, `allowed_extensions=(".woff2", ".woff", ".otf", ".ttf", ".eot")`

Same change to `_fetch_resource_bytes()` (urllib fallback).

## Phase 2 — Extraction Functions

```python
def extract_css_links(html: str, base_url: str) -> list[str]:
    """All <link rel=stylesheet href=...> URLs."""
    soup = BeautifulSoup(html, "html.parser")
    urls = []
    for link in soup.find_all("link"):
        rel = link.get("rel") or []
        if isinstance(rel, str):
            rel = [rel]
        if "stylesheet" not in [r.lower() for r in rel]:
            continue
        href = link.get("href", "")
        if not href:
            continue
        absolute = _normalized_crawl_url(href, base_url)
        if absolute:
            urls.append(absolute)
    return urls


def extract_font_urls_from_css(css_content: str, source_url: str) -> list[str]:
    """url(...) references inside @font-face that point to font files."""
    FONT_EXTS = {".woff2", ".woff", ".ttf", ".otf", ".eot"}
    urls = []
    for m in re.finditer(r'url\(\s*["\']?([^"\')\s]+)["\']?\s*\)', css_content, re.I):
        ref = m.group(1)
        absolute = _normalized_crawl_url(ref, source_url)
        if absolute and Path(urlparse(absolute).path).suffix.lower() in FONT_EXTS:
            urls.append(absolute)
    return urls


def extract_js_links(html: str, base_url: str) -> list[str]:
    """All <script src=...> URLs (including async/defer/module)."""
    soup = BeautifulSoup(html, "html.parser")
    urls = []
    for script in soup.find_all("script", src=True):
        absolute = _normalized_crawl_url(script["src"], base_url)
        if absolute:
            urls.append(absolute)
    return urls
```

## Phase 3 — Download + Rewrite Loop

Insert after image download + rewrite (after line ~1900 in `crawler.py`),
before markdown generation:

```python
if options.get("download_assets", False):
    t_assets = time.perf_counter()
    try:
        html = await _download_and_rewrite_assets(tab, html, url, out_dir, options)
    except Exception as exc:
        log.warning("Asset download failed for %s: %s", url, exc)
    _timing_record(url, "asset_download", time.perf_counter() - t_assets)
```

```python
async def _download_and_rewrite_assets(tab, html, page_url, out_dir, options):
    assets_dir = out_dir / "assets"
    manifest_path = assets_dir / "_asset_manifest.json"
    manifest = _load_json(manifest_path) if manifest_path.exists() else {}

    max_css = int(options.get("max_css_per_page", 10))
    max_js = int(options.get("max_js_per_page", 5))
    max_bytes = int(options.get("max_asset_bytes", 5242880))

    # --- CSS ---
    css_urls = extract_css_links(html, page_url)[:max_css]
    for url in css_urls:
        resource = await _fetch_resource_bytes_cached(
            tab, url, allowed_mime_prefixes=("text/css",), allowed_extensions=(".css",))
        if not resource:
            resource = await _fetch_resource_bytes(tab, url, ...)
        if not resource:
            continue
        content, ct = resource
        if len(content) > max_bytes: continue
        fname = _asset_name(url, content, "css")
        (assets_dir / "css").mkdir(parents=True, exist_ok=True)
        (assets_dir / "css" / fname).write_bytes(content)
        html = html.replace(url, f"assets/css/{fname}")

        # --- Fonts from CSS ---
        css_text = content.decode("utf-8", errors="replace")
        font_urls = extract_font_urls_from_css(css_text, url)
        for font_url in font_urls:
            font_res = await _fetch_resource_bytes_cached(tab, font_url, ...)
            if not font_res: continue
            fcontent, _ = font_res
            if len(fcontent) > max_bytes: continue
            fname = _asset_name(font_url, fcontent, "font")
            (assets_dir / "fonts").mkdir(parents=True, exist_ok=True)
            (assets_dir / "fonts" / fname).write_bytes(fcontent)
            css_text = css_text.replace(font_url, f"../fonts/{fname}")
        (assets_dir / "css" / fname).write_text(css_text, encoding="utf-8")
        manifest[url] = {"file": f"css/{fname}", "size": len(content)}

    # --- JS ---
    js_urls = extract_js_links(html, page_url)[:max_js]
    for url in js_urls:
        resource = await _fetch_resource_bytes_cached(
            tab, url, allowed_mime_prefixes=("text/javascript", "application/javascript"),
            allowed_extensions=(".js",))
        if not resource:
            resource = await _fetch_resource_bytes(tab, url, ...)
        if not resource: continue
        content, _ = resource
        if len(content) > max_bytes: continue
        fname = _asset_name(url, content, "js")
        (assets_dir / "js").mkdir(parents=True, exist_ok=True)
        (assets_dir / "js" / fname).write_bytes(content)
        html = html.replace(url, f"assets/js/{fname}")
        manifest[url] = {"file": f"js/{fname}", "size": len(content)}

    _save_json(manifest_path, manifest)
    return html
```

## Phase 4 — Content-Hash Naming

```python
def _asset_name(url: str, content: bytes, category: str) -> str:
    stem = Path(urlparse(url).path).stem or category
    h = hashlib.sha256(content).hexdigest()[:8]
    ext = Path(urlparse(url).path).suffix or {"css":".css","js":".js","font":".woff2"}[category]
    return f"{stem}.{h}{ext}"
```

## Phase 5 — Storage Layout

```
sites_crawled/www.prolicht.at/
  ├── index.html                    # rewritten href/src
  ├── assets/
  │   ├── _asset_manifest.json
  │   ├── css/application.a1b2c3d4.css
  │   ├── fonts/GothamNarrow-Book.e5f6g7h8.woff
  │   └── js/app.m3n4o5p6.js
  ├── images/                       # existing
  └── screenshots/                  # existing
```

## Phase 6 — API / CLI

```python
download_assets: bool = Field(default=False)
max_css_per_page: int = Field(default=10, ge=0)
max_js_per_page: int = Field(default=5, ge=0)
max_asset_bytes: int = Field(default=5242880, ge=1)  # 5 MB
```

```powershell
py crawler_client.py "https://www.prolicht.at/de/" --recursive \
  --download-assets --keep-crawl
```

## Phase 7 — Fix Lazy Images via Page Scroll

Before HTML capture in `_fetch_and_save()`, add mandatory scroll:

```python
# Scroll to trigger lazy-loaded images (carousels, bttrlazyloading)
await _scroll_to(tab, 99999)
await tab.sleep(2.0)
await _scroll_to(tab, 0)
await tab.sleep(0.5)
```

This ensures `data-bttr-src` attributes are swapped to real `<img src>` by
the page's JS before the HTML snapshot is taken. Currently done only for
screenshots, not for HTML capture.

## Files to Modify

| File | Change |
|------|--------|
| `crawler.py:1635` | Make `_fetch_resource_bytes_cached` content-type aware |
| `crawler.py:1718` | Same for `_fetch_resource_bytes` (urllib fallback) |
| `crawler.py` (new) | `extract_css_links`, `extract_font_urls_from_css`, `extract_js_links` |
| `crawler.py` (new) | `_download_and_rewrite_assets` |
| `crawler.py` (~line 1900) | Call asset download after image rewrite, before markdown |
| `crawler.py` (~line 2860) | Add scroll-to-bottom before HTML capture |
| `server.py` | Add `download_assets`, `max_css_per_page`, `max_js_per_page` |
| `crawler_client.py` | Add `--download-assets` flag |
