# Rendered webpage crawler with Markdown output

This crawler opens live webpages in Chromium through `nodriver`, handles cookie consent on the online page, waits for rendered content, downloads images referenced by the rendered HTML, rewrites image URLs to local `images/...` paths, and writes AI-facing Markdown.

The saved HTML is a debug/source artifact. The main target output is Markdown plus downloaded images.

## Files

```text
crawler.py
webpage_vision_markdown.py   optional screenshot-to-structured-Markdown pipeline
gemini_detect.py             cookie-consent verification from screenshots
requirements.txt
README.md
AGENTS.md
NODRIVER.md
check_links.txt
```

## Install

```bat
pip install -r requirements.txt
```

For cookie-consent screenshot verification and optional vision Markdown:

```bat
set GEMINI_API_KEY=your_key_here
```

## Recommended PROLICHT test

Rendered HTML to Markdown, screenshots captured by default, no vision Markdown:

```bat
py crawler.py --fetch-sitemap "https://www.prolicht.at" --limit 6 --headless --viewport-width 1920 --viewport-height 1080 --markdown true --verbose
```

Optional screenshot vision Markdown:

```bat
py crawler.py --fetch-sitemap "https://www.prolicht.at" --limit 6 --headless --viewport-width 1920 --viewport-height 1080 --markdown true --vision true --force-vision --verbose
```

## Current artifact flow

```text
online page
  -> wait for cookie modal / lazy rendering
  -> screenshot to gemini_detect
  -> click consent if needed
  -> second screenshot to gemini_detect
  -> continue only after consent is gone or log a warning
  -> wait for rendered content stability
  -> get rendered HTML from browser
  -> download and rewrite images to images/...
  -> write .html debug snapshot
  -> write .txt extracted from rendered HTML
  -> write screenshots by default
  -> write .md from rendered HTML when --markdown true --vision false
  -> optionally write .md + .json from screenshot vision when --vision true
```

## Output layout

For `https://www.prolicht.at`:

```text
sites_crawled/www.prolicht.at/
├── images/
│   ├── prolicht__at__original-image-name.webp
│   └── ...
├── screenshots/
│   ├── prolicht__at__consent_check_01_before.png
│   ├── prolicht__at__consent_check_01_after.png
│   ├── prolicht__at__tile_001.png
│   └── prolicht__at__screenshots.json
├── prolicht__at__before_cookie.html
├── prolicht__at__after_cookie.html
├── prolicht__at.html
├── prolicht__at.txt
├── prolicht__at.md
├── prolicht__at.images.json
├── prolicht__at.crawl.json
├── _consent_snapshots.json
├── _crawl_run.json
├── _discovered_urls.json
├── _failed_urls.json
└── _resource_map.json
```

For a deeper URL:

```text
https://www.prolicht.at/de/produktkonfigurator/massgeschneiderte-losungen/
-> prolicht__at__de__produktkonfigurator__massgeschneiderte-losungen.md
-> images/prolicht__at__de__produktkonfigurator__massgeschneiderte-losungen__original-image-name.jpg
```

## CLI modes

Exactly one fetch mode is required:

```text
--url URL                    alias for --fetch-single URL
--fetch-single URL           fetch exactly one page
--fetch-page-urls FILE       fetch one URL per non-comment line
--fetch-sitemap URL          discover and fetch internal pages
```

Important options:

```text
--limit N                    sitemap mode: max total pages including start page; 0 = unlimited
--headless                   run Chromium headless
--viewport-width N           requested viewport width
--viewport-height N          requested viewport height
--screenshots true|false     capture post-consent screenshots; default true
--markdown true|false        create .md; default false
--vision true|false          use screenshot vision pipeline for Markdown; default false
--consent-check true|false   verify cookie consent through Gemini screenshots; default true
--consent-wait SECONDS       wait before first consent screenshot; default 3
--content-timeout SECONDS    wait for rendered content stability; default 20
--navigation-timeout SECONDS wait for nodriver navigation; default 30
--html-timeout SECONDS       wait for rendered HTML capture; default 15
```

## Markdown behavior

Default Markdown path:

```text
post-consent rendered HTML -> local image rewrite -> markdownify -> .md
```

Vision Markdown path:

```text
post-consent screenshots -> Gemini -> webpage/v1 JSON -> .md
```

The old `.txt` extraction remains available but is not the source for Markdown.

## Images

Only images needed by Markdown are downloaded. CSS and JavaScript mirroring is intentionally out of scope. Images are saved under the domain output directory in `images/` and filenames are based on the page stem plus the cleaned original image filename.

The crawler downloads and rewrites:

```text
img[src]
img[srcset]
source[srcset]
lazy attributes such as data-src and data-srcset
poster
inline background-image URLs when visible in rendered HTML
direct image links
common metadata images for manifest/debugging
```
