# AGENTS.md - crawler Docker/FastAPI deployment

## THE TARGET

This project is a webpage-to-Markdown tool for LLM workflows. It supports one URL, a URL list, or a whole-site crawl, and is callable from an MCP server, Open WebUI, or other programmatic API-style tools.

The primary product output is fast usable Markdown for an LLM. The fast path produces rendered post-consent Markdown with only useful browser-loaded visible images copied from the browser cache. Better but slower outputs (full-page scrolling, `__fullpage.md`, quality classification, Gemini vision Markdown) are optional/background work.

Do not make `--markdown true` imply every expensive operation. Keep fast Markdown, full-page enrichment, quality checks, and vision fallback as separate modes.

## Project Boundary

- `D:\Docker\crawler\` is the **deployment directory** (Docker host root).
- `D:\repos\crawler\` is the **source repository**. Copy changes from there, not the reverse.
- `.env` is **not committed** to git; it contains secrets generated from the host environment.
- `sites_crawled\` and `chrome_profile\` are runtime host volumes, not part of the image.
- `nodriver` is installed from `app/requirements.txt`; the vendored `D:\repos\crawler\nodriver\` checkout is not copied into the container.

## Directory Layout

```text
D:\Docker\crawler\
├── .env                          secrets + runtime vars (not committed)
├── .dockerignore                 build-context exclusions
├── docker-compose.yaml           service definition
├── docker-entrypoint.sh          Xvfb launch + uvicorn start
├── Dockerfile                    Python 3.12 + chromium + xvfb
├── analyze_image.py              Gemini vision tool for agent image analysis (host-side)
├── app/                          (mounted from host, copied from repos\crawler\app)
│   ├── crawler.py                core crawler (CLI + fetch_single_result API)
│   ├── server.py                 FastAPI app + CrawlerWorker + endpoints
│   ├── file_server.py            HMAC-signed token delivery + short links for markdown/images
│   ├── gemini_detect.py          Gemini Flash Lite consent detection
│   ├── webpage_vision_markdown.py optional screenshot-to-structured-Markdown
│   ├── requirements.txt          all Python deps including fastapi, uvicorn
│   ├── cookies/                  saved consent cookies per domain
│   ├── timing_logs/              per-run .csv + .log timing data
│   ├── NODRIVER.md               nodriver usage rules for crawler
│   ├── AGENTS.md                 full crawler contracts
│   └── README.md                 CLI modes and output layout
├── scripts/
│   └── export-current-env.ps1    exports Windows env vars into .env
├── client_scripts/               host-side REST API clients (not in container)
│   ├── crawler_client.py         Python client: crawl + download markdown/images
│   ├── crawl_and_download.ps1    PowerShell client: crawl + download
│   ├── crawl_urls.ps1            PowerShell batch crawl
│   └── crawled_pages/            default download output
├── sites_crawled/                (host volume) crawl output
└── chrome_profile/               (host volume) persistent Chromium profile
```

### API vs CLI defaults

| Option | API default (`request_options()`) | CLI default (`argparse`) |
|---|---|---|
| `max_images` | 5 | 50 |
| `screenshots` | false | false |
| `markdown` | true | false |
| `images` | true | false |
| `threads` | 4 | 4 |
| `consent_check` | true | true |
| `consent_wait` | 3.0 | 3.0 |
| `content_timeout` | 10.0 | 10.0 |
| `navigation_timeout` | 30.0 | 30.0 |
| `html_timeout` | 15.0 | 15.0 |

Viewport is hardcoded to 1920x1080 in the API (not exposed as a request field). In CLI mode it is configurable via `--viewport-width`/`--viewport-height`.

### Source Files vs Docker Volume

The `app/` directory is a bind-mount volume (`./app:/crawler/app`). Code inside the image is built once from `COPY app/` but at runtime the volume overlay replaces it. When changing code:

1. Edit in `D:\repos\crawler\app\`
2. Copy to `D:\Docker\crawler\app\`
3. Restart container (volume picks up changes without rebuild — unless `requirements.txt` changed)

## Environment Variables

| Variable | Default | Purpose |
|---|---|---|
| `GEMINI_API_KEY` | — | Consent screenshot verification + `--vision true` Markdown |
| `DEEPSEEK_API_KEY` | — | `--quality-check true` semantic checks |
| `OPENAI_API_KEY` | — | `check_image.py` screenshot-quality checks |
| `CRAWLER_HOST` | `0.0.0.0` | uvicorn listen address |
| `CRAWLER_PORT` | `8114` | uvicorn port |
| `CRAWLER_PUBLIC_BASE_URL` | `https://crawler.prototype.prolicht.digital` | reverse-proxy base URL |
| `CRAWLER_HEADLESS` | `true` | Chromium headless mode (`false` needs Xvfb) |
| `CRAWLER_DOCKER` | `true` | enables `--no-sandbox`, `--disable-dev-shm-usage` |
| `CRAWLER_USE_XVFB` | `true` | start Xvfb when `CRAWLER_HEADLESS=false` |
| `DISPLAY` | `:99` | X11 display for Xvfb |
| `CRAWLER_CHROME_BINARY` | — | override Chromium path inside container |
| `CRAWLER_OUT` | `/crawler/sites_crawled` | output root override |
| `PYTHONUNBUFFERED` | `1` | unbuffered Python output |
| `CRAWLER_FILE_SECRET` | `crawler-file-signing-key-2026` | HMAC secret for signed file/image URLs |
| `CRAWLER_FILE_TTL_SECONDS` | `1800` | token expiry for markdown/image URLs (30 min) |
| `CRAWLER_DB_HOST` | `mariadb` | MariaDB host for cache/discovery |
| `CRAWLER_DB_PORT` | `3306` | MariaDB port |
| `CRAWLER_DB_USER` | `crawler` | DB username |
| `CRAWLER_DB_PASS` | `crawler-secret` | DB password |
| `CRAWLER_DB_NAME` | `crawler_cache` | DB name |
| `CRAWLER_CACHE_TTL_SECONDS` | `86400` | freshness window for cache-first (24h) |
| `CRAWLER_TIMING_DIR` | `../data/logs/timing_logs` | timing CSV/LOG output |
| `GEMINI_MODEL` | `gemini-flash-lite-latest` | model for page classification |
| `GEMINI_ANALYSIS_ENABLED` | `true` | auto-trigger Gemini on low-confidence |

## Commands

Run from `D:\Docker\crawler`:

```powershell
# Build the image
docker compose build

# Start detached
docker compose up -d

# View status
docker compose ps

# View logs
docker logs crawler --tail 100

# Rebuild and restart after code changes
docker compose up -d --build

# Stop
docker compose down

# Generate .env from current Windows environment
& "D:\Docker\crawler\scripts\export-current-env.ps1"
```

## API Endpoints

Base URL: `http://localhost:8114`. Full Swagger docs with request examples and response schemas at `/docs`.
Public URL: `https://crawler.prototype.prolicht.digital`.

All crawl endpoints use hardcoded viewport 1920x1080 (not request-configurable).
`headless` defaults to `null` (uses `CRAWLER_HEADLESS` env var).

### Shared crawl options

| Option | Values | Default | Purpose |
|---|---|---|---|
| `cache_policy` | cache_first, revalidate, refresh, force | revalidate | cache_first: 0.02s, no browser |
| `crawl_scope` | all, new_only | all | new_only skips known pages/products |
| `query_mode` | semantic, full, path_only | semantic | preserves tab/filter/MountingType |
| `interactive_mode` | off, discover_only, full | off | clicks JS buttons/cards, captures modals |
| `include_page_data` | bool | false | returns inline markdown + images + links |
| `max_inline_chars` | int | 0 | 0 = unlimited, full markdown returned |

### `GET /health`

Worker and browser readiness.

```json
{"status": "ok", "worker_started": true}
```

### `POST /crawl/single`

Crawls one URL. Returns artifact paths + public markdown download URL.
With `include_page_data=true`, returns inline markdown, image URLs, and discoverable links.

**Request:**
```json
{
  "url": "https://www.prolicht.at",
  "markdown": true,
  "images": true,
  "max_images": 5,
  "cache_policy": "cache_first",
  "query_mode": "semantic",
  "include_page_data": true
}
```

**Response (with include_page_data):**
```json
{
  "url": "https://www.prolicht.at",
  "domain": "www.prolicht.at",
  "status": "ok",
  "fast_markdown_path": "/crawler/sites_crawled/www.prolicht.at/prolicht__at.md",
  "markdown_url": "https://crawler.prototype.prolicht.digital/l/abc123",
  "page": {
    "url": "...",
    "domain": "...",
    "status": "ok",
    "markdown": "# inline markdown...",
    "markdown_url": "https://...",
    "images": [{"name": "hero.webp", "url": "https://...l/xxx", "relative_path": "images/..."}],
    "links": [{"url": "https://...", "text": "Link text"}],
    "paths": {"markdown": "...", "html": "...", "txt": "...", "images_manifest": "...", "crawl_metadata": "..."},
    "errors": []
  },
  "consent_status": "skipped_cookies_loaded",
  "elapsed_seconds": 4.0
}
```

### `POST /crawl/urls`

Crawl a list of URLs spanning one or more domains. URLs grouped by domain, one browser session per domain. Equivalent to CLI `--fetch-urls`.

**Request:**
```json
{
  "urls": ["https://example.com/page1", "https://example.com/page2"],
  "markdown": true, "images": true, "max_images": 5, "limit": 0, "threads": 4,
  "include_page_data": true
}
```

**Response:** `{"status": "ok"|"partial"|"error", "total": N, "succeeded": N, "failed": N, "failed_urls": [...], "results": {...}, "markdown_urls": {...}, "pages": [...]}`

### `POST /crawl/sitemap`

Recursive same-folder crawl. For Writerside/help sites, only `.html` pages in the same directory are followed. Product detail pages are skipped if the product already exists in the database.

**Request:**
```json
{
  "url": "https://www.jetbrains.com/help/youtrack/devportal/youtrack-rest-api.html",
  "markdown": true, "images": true, "max_images": 5,
  "limit": 30, "max_depth": 0, "recursive": true, "threads": 4,
  "include_page_data": true
}
```

**Response:** Same shape as `/crawl/urls`. Includes `markdown_urls` mapping each page URL to its signed markdown download URL and `pages[]` array.

### `POST /artifacts/page`

Read already-crawled page from disk without opening the browser. No browser activity.

**Request:** `{"url": "https://www.prolicht.at", "max_inline_chars": 0}`

**Response:** Same page object as `/crawl/single` with `include_page_data`.

### `POST /artifacts/pages`

Batch-read already-crawled pages. No browser activity.

**Request:** `{"urls": ["https://a.com", "https://b.com"], "max_inline_chars": 0}`

**Response:** `{"status": "ok", "total": N, "pages": [...]}`

### `POST /browser/restart`

Stub endpoint (returns `"not_implemented_yet"`).

### `GET /crawler/markdown/{token}`

Serves a crawled `.md` file with all `images/...` references rewritten to signed short-link image URLs. Token is HMAC-signed with expiry (default 30 min).

### `GET /crawler/image/{token}`

Serves a crawled image binary with correct Content-Type. `Cache-Control: private, max-age=300`. Token is HMAC-signed with expiry.

### `GET /l/{code}`

Redirects (307) a 6-character BASE62 short code to the full signed markdown URL. Codes are generated automatically when `markdown_url` is returned in crawl responses.

### Page classification

After every crawl, the rule-based classifier determines: `page_class` (catalogue_context_page, product_family_page, product_configurator_entry, product_detail_page, normal_page), `product` (bool), `product_page_role`, and classification `confidence`. Low-confidence results auto-trigger Gemini Flash Lite analysis when `GEMINI_API_KEY` is set.

### Product deduplication

Sitemap crawler checks `products` DB table before enqueuing product detail URLs (paths matching `/products/p/` or `/product-configurator/product-families/`). Already-known products are skipped — only the reference to the new context is recorded.

## Service Architecture

```text
Host: Windows
  └── Docker Engine
       └── Container: crawler (hostname: crawler.prototype.prolicht.digital)
            ├── FastAPI on 0.0.0.0:8114 (uvicorn)
            │   ├── GET  /health
            │   ├── POST /crawl/single
            │   ├── POST /crawl/urls
            │   ├── POST /crawl/sitemap
            │   ├── POST /browser/restart
            │   ├── GET  /crawler/markdown/{token}
            │   ├── GET  /crawler/image/{token}
            │   └── GET  /l/{code}
            ├── CrawlerWorker (separate thread, nodriver uc.loop())
            │   └── submit(crawler_coroutine) -> await asyncio.wrap_future(future)
            ├── Chromium (headless by default, --no-sandbox)
            ├── Xvfb on DISPLAY=:99 (only when CRAWLER_HEADLESS=false)
            └── Volumes:
                ├── ./app:/crawler/app
                ├── ./sites_crawled:/crawler/sites_crawled
                └── ./chrome_profile:/crawler/chrome_profile
```

The `CrawlerWorker` runs nodriver's `uc.loop()` in a dedicated daemon thread. FastAPI request handlers call `await worker.submit(crawler.fetch_single_result(...))` to bridge between the uvicorn event loop and the nodriver event loop without nested-loop errors.

## Crawl Contracts

See `app/AGENTS.md` for the full crawl contracts (consent flow, required outputs, Markdown behavior, image downloads, link discovery, logging).

See `app/NODRIVER.md` for nodriver usage rules (cookies, navigation, consent clicking, screenshots, evaluate patterns).

Key reminder: in CLI mode, nodriver is driven with `uc.loop().run_until_complete(...)`. In the FastAPI service, the `CrawlerWorker` owns the nodriver loop — never call `uc.loop().run_until_complete()` or `asyncio.run()` inside a request handler.

## Timing Targets

With saved PROLICHT cookies and warm browser:

```text
content_wait < 1.0s
consent_prepare < 1.5s
total_page < 5.0s (CLI cold start)
warm API request < 3.0s (container reuse)
```

## Headless vs. Non-Headless

Default production mode is headless:

```text
CRAWLER_HEADLESS=true     no Xvfb needed, no visible desktop
```

Debug mode (non-headless):

```text
CRAWLER_HEADLESS=false    requires Xvfb
CRAWLER_USE_XVFB=true     docker-entrypoint.sh starts Xvfb on :99
DISPLAY=:99               ensure Chromium finds the virtual display
```

## Code Change Flow

1. Make changes in `D:\repos\crawler\app\`
2. Test with CLI: `py crawler.py --fetch-single "https://www.prolicht.at" --headless --markdown true --images true --screenshots false --quality-check false --max-images 5 --verbose`
3. Copy to Docker host: `Copy-Item -Path "D:\repos\crawler\app\*" -Destination "D:\Docker\crawler\app" -Recurse -Force -Exclude "__pycache__", ".venv", "venv", "crawler.zip"`
4. Rebuild and restart: `docker compose up -d --build`
5. Test API: `Invoke-RestMethod -Uri "http://localhost:8114/crawl/single" -Method Post -ContentType "application/json" -Body '{"url":"https://www.prolicht.at","markdown":true,"images":true,"max_images":5}'`

## Runtime Gotchas

- Nodriver `await_promise=True` may cause `tab.evaluate()` to return the function RemoteObject instead of executing it. Prefer `return_by_value=True` without `await_promise` as the primary evaluation path.
- The `CrawlerWorker` daemon thread owns the nodriver event loop. Never call `uc.loop().run_until_complete()` or `asyncio.run()` inside a FastAPI request handler.
- Container Chromium needs `--no-sandbox` and `--disable-dev-shm-usage` (handled automatically when `CRAWLER_DOCKER=true`).
- `shm_size: "2gb"` in docker-compose.yaml prevents Chromium crashes caused by tiny `/dev/shm`.
- The `.env` file is excluded from the Docker build context via `.dockerignore`; it is loaded at runtime via `env_file` in docker-compose.yaml.
  - When updating `app/requirements.txt`, rebuild the image to install new packages.

## Nodriver Documentation

The nodriver Python package (v0.50.3) is the core browser automation library.
Source code is available at `documentation/nodriver-source-gitlab/`.

Detailed reference docs for LLM agents coding in this project:

- **`documentation/nodriver/NODRIVER-AGENTS.MD`** — Architecture overview: imports,
  module map, browser lifecycle, tab methods by frequency, 3-path JS evaluation,
  browser-cached resource fetch, raw CDP cookies, troubleshooting
- **`documentation/nodriver/NODRIVER-PAGE-CACHING.md`** — Deep dive: CDP
  `Network.loadNetworkResource` pipeline for images/CSS/fonts/JS, content-type
  filtering parameters, Latin-1→UTF-8 fix, static resource URL filtering fix,
  asset manifest format, CDP API notes, troubleshooting
- **`documentation/nodriver/NODRIVER-CDP-DIRECT.md`** — Direct CDP:
  typed wrappers vs raw generators, cookie persistence via `Storage.getCookies`/
  `Storage.setCookies`, 3-path `evaluate_value()` with `serialization="deep"`,
  result normalization, page metrics, DOM extraction, troubleshooting

- `_image_md_pattern` in `file_server.py` strips markdown title attributes (`"Title"`) from image references before building tokens. If images return 404 after a crawl, check that filenames don't contain quote characters embedded by `markdownify`.

## Agent Image Analysis Tool

`analyze_image.py` is a host-side Gemini vision tool that lets the LLM agent inspect screenshots, consent popups, error pages, and crawl artifacts without native image support. Uses `GEMINI_API_KEY` from host environment variables and model `models/gemini-3.1-flash-lite-preview`.

**Always use this tool when you need to look at an image.** You cannot view images natively. Copy images out of the container first with `docker cp`, then analyze.

### Quick reference

```powershell
# Copy an image out of the container first
docker cp crawler:/crawler/sites_crawled/www.example.com/screenshots/example__consent_check_01_before.png D:\temp\opencode\before.png

# Preset modes
py analyze_image.py D:\temp\opencode\screenshot.png                         # general describe
py analyze_image.py D:\temp\opencode\screenshot.png --mode consent           # cookie consent check
py analyze_image.py D:\temp\opencode\screenshot.png --mode cloudflare        # bot-protection check
py analyze_image.py D:\temp\opencode\screenshot.png --mode login             # login wall check
py analyze_image.py D:\temp\opencode\screenshot.png --mode content           # page load quality

# Compare before/after (two images)
py analyze_image.py D:\temp\opencode\before.png D:\temp\opencode\after.png --mode compare

# Custom question
py analyze_image.py D:\temp\opencode\screenshot.png -p "Are product images loaded or showing placeholders?"
```

### Modes

| Mode | Use when |
|---|---|
| `describe` | General: what does this page look like? (default) |
| `consent` | Did cookie consent popup appear / get dismissed? |
| `cloudflare` | Is Cloudflare/Akamai/CAPTCHA blocking the page? |
| `login` | Is a login wall, paywall, or auth gate visible? |
| `content` | Is the page fully loaded, empty, errored, or spinning? |
| `compare` | Two images: what changed between before and after? |

### Typical agent workflow

1. Crawl returns — check if consent worked:
   ```powershell
   docker cp crawler:/crawler/sites_crawled/DOMAIN/screenshots/STEM__consent_check_01_before.png D:\temp\opencode\before.png
   docker cp crawler:/crawler/sites_crawled/DOMAIN/screenshots/STEM__consent_check_01_after.png D:\temp\opencode\after.png
   py analyze_image.py D:\temp\opencode\before.png D:\temp\opencode\after.png --mode compare
   ```
2. Crawl blocked — check what's on screen:
   ```powershell
   docker cp crawler:/crawler/sites_crawled/DOMAIN/screenshots/STEM__challenge_check.png D:\temp\opencode\challenge.png
   py analyze_image.py D:\temp\opencode\challenge.png --mode cloudflare
   ```
3. Page looks wrong — check content quality:
   ```powershell
   docker cp crawler:/crawler/sites_crawled/DOMAIN/screenshots/STEM__viewport_initial.png D:\temp\opencode\page.png
   py analyze_image.py D:\temp\opencode\page.png --mode content
   ```
