# TEST_PLAN.md — Crawler API + MCP

Run from host: `D:\Docker\crawler`

## Prerequisites

```powershell
docker compose ps              # both running
curl http://localhost:8114/health  # {"status":"ok","worker_started":true}
curl http://localhost:8115/health  # {"status":"ok","crawler_api_ok":true}
```

---

## 1. API Smoke Tests

### 1.1 Health

```powershell
Invoke-RestMethod http://localhost:8114/health
```
Expected: `status=ok, worker_started=true`

### 1.2 Single page crawl + rich page data

```powershell
$body = '{"url":"https://www.prolicht.at","markdown":true,"images":true,"max_images":3,"include_page_data":true,"max_inline_chars":0}'
$r = Invoke-RestMethod http://localhost:8114/crawl/single -Method Post -ContentType application/json -Body $body -TimeoutSec 120
$r.page.markdown.Length -gt 0
$r.page.links.Count -gt 0
$r.page.images.Count -gt 0
$r.page.markdown_url -match "^https://"
```
Checks: markdown non-empty, links present, image URLs have short links, markdown_url is valid.

### 1.3 Artifact read (no re-crawl)

```powershell
$body = '{"url":"https://www.prolicht.at","max_inline_chars":0}'
$r = Invoke-RestMethod http://localhost:8114/artifacts/page -Method Post -ContentType application/json -Body $body -TimeoutSec 10
$r.status -eq "ok"
$r.links.Count -gt 0
```
Checks: reads already-crawled page, no browser activity.

### 1.4 Sitemap crawl with page data

```powershell
$body = '{"url":"https://www.prolicht.at","markdown":true,"images":true,"max_images":2,"limit":3,"include_page_data":true,"max_inline_chars":0}'
$r = Invoke-RestMethod http://localhost:8114/crawl/sitemap -Method Post -ContentType application/json -Body $body -TimeoutSec 180
$r.pages.Count -eq 3
```
Checks: limit respected, pages returned with full data.

### 1.5 Multiple artifacts read

```powershell
$body = '{"urls":["https://www.prolicht.at","https://www.heise.de"],"max_inline_chars":1000}'
$r = Invoke-RestMethod http://localhost:8114/artifacts/pages -Method Post -ContentType application/json -Body $body -TimeoutSec 10
$r.pages.Count -gt 0
```
Checks: batch artifact read works.

### 1.6 Markdown serving (token URL)

```powershell
# Get markdown_url from crawl response, then:
Invoke-WebRequest $r.page.markdown_url -MaximumRedirection 5 | % { $_.StatusCode -eq 200 }
```
Checks: signed token URL delivers markdown.

### 1.7 Image serving (token URL)

```powershell
# Get image URL from page.images[0].url, then:
$img = Invoke-WebRequest $r.page.images[0].url -MaximumRedirection 5
$img.StatusCode -eq 200 -and $img.Headers["Content-Type"] -match "^image/"
```
Checks: signed image token delivers binary with correct Content-Type.

### 1.8 Short link redirect

```powershell
$shortUrl = $r.page.markdown_url  # e.g. https://host/l/abc123
Invoke-WebRequest $shortUrl -MaximumRedirection 0 -SkipHttpErrorCheck | % StatusCode
```
Checks: returns 307 redirect.

### 1.9 Error case — missing page

```powershell
$body = '{"url":"https://nonexistent.example.com/404"}'
$r = Invoke-RestMethod http://localhost:8114/crawl/single -Method Post -ContentType application/json -Body $body -TimeoutSec 60
$r.status -eq "error"
```
Checks: handles unreachable URLs gracefully.

### 1.10 Response shape contract

```powershell
$body = '{"url":"https://www.prolicht.at","include_page_data":true}'
$r = Invoke-RestMethod http://localhost:8114/crawl/single -Method Post -ContentType application/json -Body $body -TimeoutSec 120
@("url","domain","status","title","markdown","markdown_url","images","links","paths","errors") | % { $r.page.PSObject.Properties[$_] -ne $null }
```
Checks: page object has all 10 required fields.

---

## 2. MCP Smoke Tests

### 2.1 MCP health

```powershell
Invoke-RestMethod http://localhost:8115/health
```
Expected: `status=ok, crawler_api_ok=true`

### 2.2 crawl_page (recrawl=true)

```powershell
# MCP handshake
$h = @{"Accept"="application/json, text/event-stream"}
$init = @{jsonrpc="2.0";id=1;method="initialize";params=@{protocolVersion="2025-06-18";capabilities=@{};clientInfo=@{name="test";version="1.0"}}} | ConvertTo-Json -Depth 3
$sid = (Invoke-WebRequest http://localhost:8115/mcp -Method Post -Headers $h -ContentType "application/json" -Body $init -TimeoutSec 10).Headers["Mcp-Session-Id"]

# Tool call
$h["Mcp-Session-Id"] = $sid
$call = @{jsonrpc="2.0";id=2;method="tools/call";params=@{name="crawl_page";arguments=@{url="https://www.prolicht.at";max_images=2;max_inline_chars=0;recrawl=$true}}} | ConvertTo-Json -Depth 4
$r = Invoke-RestMethod http://localhost:8115/mcp -Method Post -Headers $h -ContentType "application/json" -Body $call -TimeoutSec 180
$page = ($r.result.content[0].text | ConvertFrom-Json)
$page.status -eq "ok" -and $page.page.markdown.Length -gt 0
```
Checks: full MCP protocol flow, tool returns rich page.

### 2.3 crawl_page (recrawl=false, artifacts)

```powershell
$call = @{jsonrpc="2.0";id=3;method="tools/call";params=@{name="crawl_page";arguments=@{url="https://www.prolicht.at";max_images=1;max_inline_chars=500;recrawl=$false}}} | ConvertTo-Json -Depth 4
$r = Invoke-RestMethod http://localhost:8115/mcp -Method Post -Headers $h -ContentType "application/json" -Body $call -TimeoutSec 30
$page = ($r.result.content[0].text | ConvertFrom-Json)
$page.source -eq "artifact_api"
```
Checks: uses artifact endpoint, no crawl triggered.

### 2.4 research (full pipeline)

```powershell
$call = @{jsonrpc="2.0";id=4;method="tools/call";params=@{name="research";arguments=@{question="downlights technical data";site="prolicht.at";max_pages=3;max_images=2;max_inline_chars=0;recrawl=$true}}} | ConvertTo-Json -Depth 4
$r = Invoke-RestMethod http://localhost:8115/mcp -Method Post -Headers $h -ContentType "application/json" -Body $call -TimeoutSec 600
$data = ($r.result.content[0].text | ConvertFrom-Json)
$data.search_results -gt 0
$data.pages.Count -ge 0
```
Checks: serper.dev → DeepSeek filter → crawl → structured result.

### 2.5 research with api down (resilience)

Manual test: stop crawler container, call research, verify error message instead of hang.

---

## 3. Integration Tests

### 3.1 API → MCP → LLM flow

```
1. MCP /health: crawler_api_ok=true
2. MCP crawl_page: returns page with status=ok, source=crawl_api
3. MCP research: returns pages with search_results > 0
4. All image URLs in MCP responses match ^https://.*/(l|crawler/image)/
5. No tracebacks in any response
```

### 3.2 Data persistence on host

```powershell
Get-ChildItem D:\Docker\crawler\data\sites_crawled -Recurse -File | Measure-Object | % Count
Get-ChildItem D:\Docker\crawler\data\logs\crawler.log
```
Checks: crawl output + logs visible on host.

### 3.3 Docker separation

```powershell
docker exec crawler-mcp python -c "import crawler"  # must FAIL
docker exec crawler python -c "import fastmcp"       # must FAIL
```
Checks: MCP cannot import crawler, API has no fastmcp.

---

## 4. Quick One-Liner

```powershell
# Full health check of both services
Write-Host "API:"; try { (Invoke-RestMethod http://localhost:8114/health -TimeoutSec 5).status } catch { "DOWN" }
Write-Host "MCP:"; try { (Invoke-RestMethod http://localhost:8115/health -TimeoutSec 5).status } catch { "DOWN" }
```

---

## 5. Regressions to Check After Code Changes

- [ ] `POST /crawl/single` returns `markdown_url` 
- [ ] `POST /crawl/sitemap` returns `pages[]`
- [ ] `POST /artifacts/page` works without crawling
- [ ] `GET /l/{code}` returns 307 redirect
- [ ] `GET /crawler/image/{token}` returns binary + correct Content-Type
- [ ] `GET /crawler/markdown/{token}` returns markdown with rewritten image URLs
- [ ] `discover_children_with_text()` returns `{url, text}` dicts
- [ ] MCP `/health` shows `crawler_api_ok=true`
- [ ] `max_inline_chars=0` returns full markdown
- [ ] Uncrawled same-domain links stay intact in markdown
