# IMPLEMENTATION_PLAN_API_FIRST.md

API-first implementation plan for the crawler research project.

This is the first job to run. The goal is to make the existing FastAPI crawler service produce a stable, rich page-artifact contract. The later MCP job must only call this API and must not import `crawler.py`, manage browsers, or read crawler output files directly.

---

## 0. Decision

Build the API layer first.

The MCP layer will rely on these API guarantees:

1. The API performs all browser/crawler work.
2. The API owns artifact discovery from disk.
3. The API returns markdown, public markdown URLs, image URLs, link maps, file paths, and errors in one stable response shape.
4. The MCP server only calls HTTP endpoints on the API.

This avoids the risky original idea where MCP imports `crawler.py` and starts its own browser lifecycle.

---

## 1. Current source facts to preserve

The current source already has:

```text
app/server.py
  FastAPI app
  /health
  /crawl/single
  /crawl/urls
  /crawl/sitemap
  /browser/restart
  file_server.register_routes(app)

app/crawler.py
  fetch_single_result(...)
  fetch_urls(...)
  fetch_sitemap(...)
  discover_children_from_html(...)

app/file_server.py
  build_markdown_token(...)
  build_image_token(...)
  build_public_markdown_url(...)
  build_public_image_url(...)
  shorten_public_url(...)
```

Important current behavior:

```text
fetch_single_result() returns a rich dict with artifact paths.
fetch_urls() returns dict[str, bool], not page objects.
fetch_sitemap() returns dict[str, bool], not page objects.
/crawl/urls currently returns markdown_urls, but not inline markdown, image URL list, or link map.
```

Do not refactor browser ownership in this API job.

---

## 2. Final API contract required by MCP

The MCP job will expect this page object from the API:

```json
{
  "url": "https://example.com/page",
  "domain": "example.com",
  "status": "ok",
  "title": "",
  "markdown": "# inline markdown, possibly truncated",
  "markdown_url": "https://crawler.prototype.prolicht.digital/l/abc123",
  "images": [
    {
      "name": "hero.webp",
      "url": "https://crawler.prototype.prolicht.digital/l/def456",
      "relative_path": "images/example__hero.webp"
    }
  ],
  "links": [
    {
      "url": "https://example.com/child-page",
      "text": "Child page"
    }
  ],
  "paths": {
    "markdown": "/crawler/sites_crawled/example.com/page.md",
    "html": "/crawler/sites_crawled/example.com/page.html",
    "txt": "/crawler/sites_crawled/example.com/page.txt",
    "images_manifest": "/crawler/sites_crawled/example.com/page.images.json",
    "crawl_metadata": "/crawler/sites_crawled/example.com/page.crawl.json"
  },
  "errors": []
}
```

Rules:

```text
status must be "ok" only when the crawl succeeded and the markdown file exists.
markdown may be empty only when include_markdown is false or the file is missing.
markdown must be truncated to max_inline_chars.
markdown_url must be built by file_server, never guessed manually.
images must use file_server image-token URLs.
links must come from rendered/saved HTML, not from raw requests.
paths may be absolute container paths because MCP will not use them directly.
errors must contain readable strings, not tracebacks.
```

---

## 3. Files to change in API job

```text
app/crawler.py          add one helper: discover_children_with_text(...)
app/server.py           add artifact-reader helpers and new/extended endpoints
app/requirements.txt    no change needed for this API job unless current install is missing dependencies
app/AGENTS.md           document the API artifact contract
AGENTS.md               document API-first architecture and MCP dependency
```

Do not change these in the API job:

```text
app/mcp_research.py     not created here
Dockerfile              not needed for API job
browser lifecycle       no changes
crawler parallelism     no changes
consent handling        no changes
screenshot behavior     no changes
vision behavior         no changes
```

---

## 4. Step A1 - Add link-text helper to crawler.py

Location: directly below `discover_children_from_html()`.

Add this new function only. Do not modify `discover_children_from_html()`.

```python
def discover_children_with_text(html: str, base_url: str, start_area_url: str | None = None) -> list[dict[str, str]]:
    """Return crawlable child links with visible anchor text.

    This uses the same discovery rules as discover_children_from_html(), but it
    returns dictionaries for API/MCP consumers.
    """
    soup = BeautifulSoup(html, "html.parser")
    effective_base = _canonical_or_base_url_from_html(html, base_url)
    selector_groups = [
        "nav a[href], header a[href], .navigation a[href], .navbar a[href], .menu a[href]",
        ".app__nav a[href], .app__toc a[href]",
        "main a[href], article a[href], .main-content a[href], .content a[href]",
        "footer a[href]",
        "a[href]",
    ]
    found: list[dict[str, str]] = []
    seen: set[str] = set()
    for selectors in selector_groups:
        for a in soup.select(selectors):
            candidate = _normalized_crawl_url(a.get("href", ""), effective_base)
            if not candidate or not _is_crawlable_child(candidate, effective_base, start_area_url=start_area_url):
                continue
            key = candidate.rstrip("/")
            if key in seen:
                continue
            seen.add(key)
            text = a.get_text(" ", strip=True).strip() or candidate
            found.append({"url": candidate, "text": text})
    log.info("Discovered %d child links with text from HTML", len(found))
    return found
```

Compile check:

```bat
cd D:\Docker\crawler\app
py -m py_compile crawler.py
```

Done when the compile check passes.

---

## 5. Step A2 - Add API request models in server.py

Add these models below the existing request models.

```python
class ArtifactPageRequest(BaseModel):
    url: str = Field(description="Full URL of the already-crawled page")
    include_markdown: bool = Field(default=True, description="Return inline markdown text")
    include_links: bool = Field(default=True, description="Return rendered HTML link map")
    include_images: bool = Field(default=True, description="Return public image URLs")
    max_images: int = Field(default=5, ge=0, description="Maximum images to return; 0 = unlimited")
    max_inline_chars: int = Field(default=20000, ge=0, description="Max inline markdown chars; 0 = no inline markdown")


class ArtifactPagesRequest(BaseModel):
    urls: list[str] = Field(description="Full URLs of already-crawled pages", min_length=1)
    include_markdown: bool = Field(default=True)
    include_links: bool = Field(default=True)
    include_images: bool = Field(default=True)
    max_images: int = Field(default=5, ge=0)
    max_inline_chars: int = Field(default=12000, ge=0)
```

Extend existing crawl request models with optional page-data fields.

Add to `CrawlSingleRequest`:

```python
include_page_data: bool = Field(default=False, description="Return rich page artifact payload after crawling")
max_inline_chars: int = Field(default=20000, ge=0, description="Max inline markdown chars when include_page_data=true")
```

Add to `CrawlUrlsRequest` and `CrawlSitemapRequest`:

```python
include_page_data: bool = Field(default=False, description="Return rich page artifact payloads after crawling")
max_inline_chars: int = Field(default=12000, ge=0, description="Max inline markdown chars per page when include_page_data=true")
```

Reason:

```text
Existing API callers remain compatible because include_page_data defaults to false.
MCP can request rich payloads when needed.
```

---

## 6. Step A3 - Add artifact helper functions in server.py

Place these helpers below `_build_markdown_urls()`.

Required helpers:

```text
_safe_read_text(path, max_chars)
_safe_read_json(path)
_page_artifact_paths(url)
_build_public_markdown_url_for_path(domain, md_path)
_build_public_image_urls_for_manifest(domain, manifest_path, max_images)
_discover_links_for_html(html_path, url)
_build_page_artifact(url, ok=None, crawl_result=None, include_markdown=True, include_links=True, include_images=True, max_images=5, max_inline_chars=20000)
```

Implementation requirements:

```text
Use crawler._real_host(url) for domain.
Use crawler._url_output_stem(url) for output file stem.
Use crawler.SITES_ROOT / domain / filename for paths.
Never guess public URLs; always use file_server token helpers.
If a file does not exist, return None/empty values and add a readable error.
Do not raise for missing markdown/html/images manifests.
Do not expose raw tracebacks in API responses.
```

Skeleton:

```python
def _safe_read_text(path: Path, max_chars: int) -> tuple[str, str | None]:
    if max_chars <= 0:
        return "", None
    if not path.exists() or not path.is_file():
        return "", f"missing file: {path}"
    try:
        text = path.read_text(encoding="utf-8", errors="replace")
        return text[:max_chars], None
    except Exception as exc:
        return "", f"failed to read {path}: {exc}"


def _safe_read_json(path: Path) -> tuple[dict[str, Any], str | None]:
    if not path.exists() or not path.is_file():
        return {}, f"missing file: {path}"
    try:
        return json.loads(path.read_text(encoding="utf-8", errors="replace")), None
    except Exception as exc:
        return {}, f"failed to read json {path}: {exc}"
```

Add `import json` at the top of `server.py` if missing.

Image URL helper requirements:

```text
The manifest file is usually <stem>.images.json.
Look for manifest["images"] as a list.
For every image, use relative_path when available.
Build token with file_server.build_image_token(domain, relative_path).
Build public URL with file_server.build_public_image_url(token).
Shorten with file_server.shorten_public_url(...).
Return: {"name": Path(relative_path).name, "url": short_or_full, "relative_path": relative_path}
```

Link helper requirements:

```text
Read saved HTML from <stem>.html.
Call crawler.discover_children_with_text(html, url).
If helper is missing or fails, return [] and add an error.
```

Page artifact helper requirements:

```text
If crawl_result contains artifact paths, prefer those paths.
Otherwise derive paths from URL using SITES_ROOT/domain/stem.
If ok is None, derive ok from whether markdown path exists.
status = "ok" only when ok is true and markdown exists.
Return the exact page object shown in section 2.
```

---

## 7. Step A4 - Add artifact-only endpoints

Add these endpoints to `server.py`.

```python
@app.post("/artifacts/page")
async def artifact_page(req: ArtifactPageRequest) -> dict[str, Any]:
    return _build_page_artifact(
        req.url,
        include_markdown=req.include_markdown,
        include_links=req.include_links,
        include_images=req.include_images,
        max_images=req.max_images,
        max_inline_chars=req.max_inline_chars,
    )


@app.post("/artifacts/pages")
async def artifact_pages(req: ArtifactPagesRequest) -> dict[str, Any]:
    pages = [
        _build_page_artifact(
            url,
            include_markdown=req.include_markdown,
            include_links=req.include_links,
            include_images=req.include_images,
            max_images=req.max_images,
            max_inline_chars=req.max_inline_chars,
        )
        for url in req.urls
    ]
    return {
        "status": "ok",
        "total": len(pages),
        "pages": pages,
    }
```

Done when:

```bat
cd D:\Docker\crawler\app
py -m py_compile server.py
```

passes.

---

## 8. Step A5 - Extend existing crawl endpoints to optionally include page data

Modify `/crawl/single`.

After the existing markdown URL logic, add:

```python
    if req.include_page_data:
        result["page"] = _build_page_artifact(
            req.url,
            ok=result.get("status") == "ok",
            crawl_result=result,
            include_markdown=True,
            include_links=True,
            include_images=req.images,
            max_images=req.max_images,
            max_inline_chars=req.max_inline_chars,
        )
```

Modify `/crawl/urls`.

Before `return`, build the response dict first:

```python
    response = {
        "status": overall,
        "total": len(results),
        "succeeded": len(succeeded),
        "failed": len(failed),
        "failed_urls": failed,
        "results": results,
        "markdown_urls": _build_markdown_urls(results),
        "elapsed_seconds": round(time.perf_counter() - started, 3),
    }
    if req.include_page_data:
        response["pages"] = [
            _build_page_artifact(
                url,
                ok=ok,
                include_markdown=True,
                include_links=True,
                include_images=req.images,
                max_images=req.max_images,
                max_inline_chars=req.max_inline_chars,
            )
            for url, ok in results.items()
        ]
    return response
```

Apply the same pattern to `/crawl/sitemap`.

Important:

```text
Do not remove existing response fields.
Do not change defaults.
Do not make page-data mandatory.
```

---

## 9. Step A6 - Add API smoke tests

Manual health check:

```bat
curl http://localhost:8114/health
```

Single crawl with rich page data:

```bat
curl -X POST http://localhost:8114/crawl/single ^
  -H "Content-Type: application/json" ^
  -d "{\"url\":\"https://www.prolicht.at\",\"markdown\":true,\"images\":true,\"max_images\":3,\"include_page_data\":true,\"max_inline_chars\":8000}"
```

Artifact-only read after crawl:

```bat
curl -X POST http://localhost:8114/artifacts/page ^
  -H "Content-Type: application/json" ^
  -d "{\"url\":\"https://www.prolicht.at\",\"max_images\":3,\"max_inline_chars\":8000}"
```

Multiple URLs with rich page data:

```bat
curl -X POST http://localhost:8114/crawl/urls ^
  -H "Content-Type: application/json" ^
  -d "{\"urls\":[\"https://www.prolicht.at\",\"https://www.prolicht.at/en\"],\"markdown\":true,\"images\":true,\"threads\":2,\"include_page_data\":true,\"max_inline_chars\":6000}"
```

Expected checks:

```text
Response has status ok or partial.
Single crawl response has page.markdown.
Single crawl response has page.markdown_url when public base URL is configured.
Artifact-only response returns the same markdown without re-crawling.
links is always a list.
images is always a list.
errors is always a list.
```

---

## 10. Step A7 - Docker compose for API remains mostly unchanged

Current service can stay as `crawler` on port `8114`.

Only ensure environment contains placeholders like this:

```yaml
environment:
  CRAWLER_HOST: "0.0.0.0"
  CRAWLER_PORT: "8114"
  CRAWLER_DOCKER: "true"
  CRAWLER_HEADLESS: "true"
  CRAWLER_PUBLIC_BASE_URL: "https://crawler.prototype.prolicht.digital"
  CRAWLER_FILE_SECRET: "${CRAWLER_FILE_SECRET}"
  CRAWLER_FILE_TTL_SECONDS: "1800"
```

Do not hard-code real API keys in compose.

---

## 11. Done definition for API job

The API job is done only when all of these are true:

```text
py -m py_compile crawler.py passes.
py -m py_compile server.py passes.
Docker service starts.
GET /health returns status ok.
POST /crawl/single with include_page_data=true returns page.markdown.
POST /artifacts/page returns the already-crawled page without launching a new crawl.
POST /crawl/urls with include_page_data=true returns pages[].
Every page object has: url, domain, status, title, markdown, markdown_url, images, links, paths, errors.
No MCP code exists yet or is required for this test.
```

---

## 12. Handoff contract to MCP job

When API job is complete, give the MCP implementer this contract:

```text
CRAWLER_API_BASE_URL=http://crawler:8114 inside Docker network
CRAWLER_API_BASE_URL=http://localhost:8114 on host machine

POST /crawl/single crawls one page.
POST /crawl/urls crawls multiple pages.
POST /artifacts/page reads already-created artifacts.
POST /artifacts/pages reads already-created artifacts for multiple URLs.

Use include_page_data=true for crawl endpoints when MCP needs inline markdown immediately.
Use artifact endpoints when MCP only needs to read pages that were already crawled.
```

The MCP job must not import crawler.py.
