# AGENTS.md - rendered webpage to Markdown crawler

## THE TARGET

This project is a webpage-to-Markdown tool for LLM workflows. It must support one URL, a URL list, or a whole-site crawl, and it is intended to be callable from an MCP server, for example Open WebUI fetching webpages, or from other programmatic API-style tools.

The primary product output is fast usable Markdown for an LLM. The fast path should produce rendered post-consent Markdown with only useful browser-loaded visible images copied from the browser cache. Better but slower outputs should be optional/background work: full-page scrolling, richer `__fullpage.md`, quality classification, and Gemini vision Markdown only when the programmatic HTML-to-Markdown result is bad.

Do not make `--markdown true` imply every expensive operation. Keep fast Markdown, full-page enrichment, quality checks, and vision fallback as separate modes.

## Purpose

This sidequest crawls rendered webpages and documentation pages into AI-ready Markdown.

The current fast-path priority is:

```text
live page after consent -> rendered HTML -> browser-cache visible images -> Markdown
```

The saved `.html` is a debug/source artifact, not the main target. Do not spend effort on mirroring CSS/JS unless explicitly requested again.

## Required outputs

Per page, the crawler should write:

```text
page.html                 debug/source snapshot from post-consent rendered DOM
page.txt                  plain text from post-consent rendered DOM
page.md                   primary AI-facing Markdown when --markdown true
page.images.json          local image manifest
page.crawl.json           crawl metadata
screenshots/page__*.png   post-consent screenshots when --screenshots true
```

Domain output is:

```text
sites_crawled/<real-host>/
```

Example:

```text
sites_crawled/www.prolicht.at/prolicht__at.md
sites_crawled/www.prolicht.at/images/prolicht__at__original-image-name.webp
sites_crawled/www.prolicht.at/screenshots/prolicht__at__tile_001.png
```

## Cookie consent contract

Consent must happen on the online/live page before final artifacts are written.

Process:

```text
1. Open online URL.
2. Set and verify viewport.
3. Wait, default 3 seconds.
4. Capture viewport screenshot.
5. Send screenshot to gemini_detect.
6. If cookie_consent_shown=true, click the accept button.
7. Capture a second viewport screenshot.
8. Send the second screenshot to gemini_detect.
9. Success means cookie_consent_shown=false.
10. Save cookies immediately.
11. Only then save final HTML/TXT/screenshots/Markdown.
```

Click strategy:

```text
1. Use Gemini-read accept_button_text with tab.find(..., best_match=True).
2. If DOM text click fails, use accept_button_box_2d and tab.mouse_click(x, y).
```

Do not treat "clicked something" as success. Verify the overlay is gone.

## Markdown behavior

Default:

```text
--markdown true --vision false
rendered HTML -> image rewrite -> markdownify -> .md
```

Vision mode:

```text
--markdown true --vision true
screenshots -> Gemini webpage/v1 JSON -> .md
```

The `.md` file must not be built from pre-consent HTML, local `file://` HTML, or the old `.txt` extraction.

## Images

When `--images true`, download only useful browser-loaded visible images from the post-consent live DOM and rewrite references before Markdown conversion. Prefer the browser cache/session over Python network refetches.

Current fast image source is:

```text
document.images -> img.currentSrc/img.src -> Network.loadNetworkResource browser cache stream -> local images/
```

CSS and JS files are intentionally not downloaded in the current target architecture.

## Logging

Every run must recreate `app/crawler.log`. Console and file logs must include both clock time and elapsed `T+...s` timing. Any fallback path, timeout, failed cookie save/load, failed HTML capture, failed image candidate, or skipped image response must be visible in logs.

## Link discovery

Documentation/Writerside pages should remain restricted to same-folder `.html` documentation pages.

Normal websites should crawl same-host internal links, including extensionless CMS routes such as `/de/...`.

`--limit` is a hard cap for downloaded pages, including the start page. Do not keep discovering/enqueuing links after the limit is filled.

## nodriver usage rules

Follow `NODRIVER.md`. Important rules:

```text
- use browser.cookies.save/load for cookies
- avoid raw tab.send({"method": ...}) dictionaries
- use tab.get / browser.get for navigation
- use tab.find(..., best_match=True) for visible button text
- use tab.mouse_click(x, y) for viewport pixel fallback
- use tab.save_screenshot for screenshots
- verify viewport with window.innerWidth/window.innerHeight
```

## Safe test command

```bat
py crawler.py --fetch-sitemap "https://www.prolicht.at" --limit 6 --headless --viewport-width 1920 --viewport-height 1080 --markdown true --verbose
```
