# IMPLEMENTATION_PLAN_MCP_SECOND.md

MCP implementation plan for the crawler research project.

Run this only after `IMPLEMENTATION_PLAN_API_FIRST.md` is complete and tested. The MCP server must depend on the API contract created by that plan. It must not import `crawler.py`, start browsers, manage tabs, read crawler output files, or build file tokens itself.

---

## 0. Decision

Build MCP as a thin research/orchestration layer.

MCP responsibilities:

```text
Accept LLM tool calls.
Search Google via serper.dev.
Filter candidate URLs with DeepSeek.
Call the crawler API for crawling/artifact retrieval.
Return structured page data to the LLM.
```

API responsibilities:

```text
Browser lifecycle.
Cookie consent.
Parallel crawling.
Markdown generation.
Image downloading/token URL generation.
Link extraction from rendered HTML.
Artifact reading from disk.
```

This separation is mandatory.

---

## 1. MCP depends on this API contract

The API job must already provide:

```text
GET  /health
POST /crawl/single
POST /crawl/urls
POST /artifacts/page
POST /artifacts/pages
```

The MCP server expects `POST /crawl/single` with `include_page_data=true` to return:

```json
{
  "status": "ok",
  "page": {
    "url": "https://example.com/page",
    "domain": "example.com",
    "status": "ok",
    "title": "",
    "markdown": "# inline markdown",
    "markdown_url": "https://crawler.prototype.prolicht.digital/l/abc123",
    "images": [],
    "links": [],
    "paths": {},
    "errors": []
  }
}
```

The MCP server expects `POST /crawl/urls` with `include_page_data=true` to return:

```json
{
  "status": "ok",
  "total": 2,
  "succeeded": 2,
  "failed": 0,
  "failed_urls": [],
  "results": {
    "https://example.com/a": true,
    "https://example.com/b": true
  },
  "pages": [
    {
      "url": "https://example.com/a",
      "status": "ok",
      "markdown": "..."
    }
  ]
}
```

Do not start this MCP job until those responses work.

---

## 2. Files to change in MCP job

```text
app/mcp_research.py       new FastMCP server
app/requirements.txt      add fastmcp and requests if missing
docker-compose.yaml       add mcp-research service on port 8115
.env.example or docs      document env variables without real keys
app/AGENTS.md             document MCP uses API only
AGENTS.md                 document two-service architecture
```

Do not change in MCP job:

```text
app/crawler.py            no changes
app/server.py             no changes unless API tests failed
app/file_server.py        no changes
browser lifecycle         no changes
crawler parallelism       no changes
```

---

## 3. Environment variables

Required:

```text
CRAWLER_API_BASE_URL=http://crawler:8114
SERPER_API_KEY=<serper key>
DEEPSEEK_API_KEY=<deepseek key>
```

Optional:

```text
DEEPSEEK_MODEL=deepseek-chat
MCP_HOST=0.0.0.0
MCP_PORT=8115
MCP_HTTP_TIMEOUT_SECONDS=180
MCP_MAX_SEARCH_RESULTS=20
MCP_DEFAULT_MAX_PAGES=10
```

Rules:

```text
Do not hard-code real keys.
Do not commit .env with real keys.
Inside Docker use http://crawler:8114.
On Windows host tests use http://localhost:8114.
```

---

## 4. Tool contracts

### Tool 1: crawl_page

```python
crawl_page(url: str, max_images: int = 5, max_inline_chars: int = 20000, recrawl: bool = True) -> dict
```

Behavior:

```text
When recrawl=true:
  POST /crawl/single with include_page_data=true.

When recrawl=false:
  POST /artifacts/page.

Return the API page object plus wrapper metadata.
```

Return shape:

```json
{
  "url": "https://example.com/page",
  "status": "ok",
  "source": "crawl_api",
  "page": {
    "url": "https://example.com/page",
    "domain": "example.com",
    "status": "ok",
    "markdown": "# inline markdown",
    "markdown_url": "https://...",
    "images": [],
    "links": [],
    "paths": {},
    "errors": []
  },
  "errors": []
}
```

### Tool 2: research

```python
research(question: str, site: str, max_pages: int = 10, max_images: int = 3, max_inline_chars: int = 12000, recrawl: bool = True) -> dict
```

Pipeline:

```text
1. Search Google through serper.dev using query: site:{site} {question}
2. Keep only results from the requested site/domain.
3. Ask DeepSeek whether each URL is likely relevant.
4. Keep yes results up to max_pages.
5. If recrawl=true, call POST /crawl/urls with include_page_data=true.
6. If recrawl=false, call POST /artifacts/pages.
7. Attach relevance reason to each returned page.
8. Return structured result.
```

Return shape:

```json
{
  "question": "downlights technical data",
  "site": "xal.com",
  "status": "ok",
  "search_results": 20,
  "relevant_found": 8,
  "pages_requested": 8,
  "pages_crawled": 7,
  "pages_failed": 1,
  "pages": [
    {
      "url": "https://xal.com/...",
      "status": "ok",
      "markdown": "# inline markdown",
      "markdown_url": "https://...",
      "relevance_reason": "Likely contains product specifications."
    }
  ],
  "failed_pages": [
    {
      "url": "https://xal.com/broken",
      "error": "crawl failed"
    }
  ],
  "errors": []
}
```

---

## 5. Step M1 - Add dependencies

Update `app/requirements.txt`:

```text
fastmcp
requests
```

`requests` may already be available through another dependency, but list it explicitly because MCP uses HTTP calls to serper.dev, DeepSeek, and the crawler API.

---

## 6. Step M2 - Create app/mcp_research.py

Create a new file.

Required imports:

```python
import os
import json
from urllib.parse import urlparse

import requests
from fastmcp import FastMCP, Context
```

Do not import:

```python
import crawler
import file_server
import nodriver
```

Required constants:

```python
CRAWLER_API_BASE_URL = os.getenv("CRAWLER_API_BASE_URL", "http://crawler:8114").rstrip("/")
SERPER_API_KEY = os.getenv("SERPER_API_KEY", "")
DEEPSEEK_API_KEY = os.getenv("DEEPSEEK_API_KEY", "")
DEEPSEEK_MODEL = os.getenv("DEEPSEEK_MODEL", "deepseek-chat")
HTTP_TIMEOUT = float(os.getenv("MCP_HTTP_TIMEOUT_SECONDS", "180"))
```

FastMCP setup:

```python
mcp = FastMCP(
    "Crawler Research",
    instructions=(
        "Use crawl_page for a single URL. Use research for multi-page site research. "
        "This MCP server calls the crawler API and does not crawl directly."
    ),
)
```

Health route:

```python
@mcp.custom_route("/health", methods=["GET"])
async def health_check(request) -> dict:
    api_ok = False
    api_error = ""
    try:
        resp = requests.get(f"{CRAWLER_API_BASE_URL}/health", timeout=10)
        api_ok = resp.ok
        if not resp.ok:
            api_error = resp.text[:500]
    except Exception as exc:
        api_error = str(exc)
    return {
        "status": "ok" if api_ok else "degraded",
        "crawler_api_base_url": CRAWLER_API_BASE_URL,
        "crawler_api_ok": api_ok,
        "crawler_api_error": api_error,
    }
```

Run block:

```python
if __name__ == "__main__":
    host = os.getenv("MCP_HOST", "0.0.0.0")
    port = int(os.getenv("MCP_PORT", "8115"))
    mcp.run(transport="http", host=host, port=port)
```

---

## 7. Step M3 - Add small helper functions

Implement these helpers before the tools.

```text
_ctx_info(ctx, message)
_ctx_progress(ctx, progress, total)
_require_env()
_normalize_site(site)
_same_or_subdomain(url, site)
_api_post(path, payload, timeout=HTTP_TIMEOUT)
_google_search(question, site, num=20)
_deepseek_ask(prompt)
_filter_relevant(question, candidates, max_pages)
_relevance_by_url(relevant)
_attach_relevance(pages, relevant)
```

Helper requirements:

```text
_ctx_info must do nothing when ctx is None.
_ctx_progress must do nothing when ctx is None.
_require_env must check SERPER_API_KEY and DEEPSEEK_API_KEY.
_normalize_site must strip protocol, path, query, and trailing slash.
_same_or_subdomain must reject unrelated domains from search results.
_api_post must raise a clear RuntimeError with endpoint and response text when API returns non-2xx.
_filter_relevant must return dictionaries containing url, title, snippet, position, relevance_answer, relevance_reason.
```

Use this relevance output format from DeepSeek:

```json
{"relevant": true, "reason": "Likely contains technical product specifications."}
```

Prompt for DeepSeek:

```text
You decide if a Google result is useful for answering a user question.
Return only compact JSON with keys: relevant, reason.

Question: {question}
URL: {url}
Title: {title}
Snippet: {snippet}
```

DeepSeek parsing rule:

```text
Try json.loads first.
If parsing fails, fall back to yes/no detection.
Never crash the whole research call because one relevance check failed.
Mark that candidate as not relevant and store the error in reason.
```

---

## 8. Step M4 - Implement crawl_page tool

Implementation logic:

```python
@mcp.tool(annotations={"readOnlyHint": True})
async def crawl_page(
    url: str,
    max_images: int = 5,
    max_inline_chars: int = 20000,
    recrawl: bool = True,
    ctx: Context | None = None,
) -> dict:
    await _ctx_info(ctx, f"Preparing page: {url}")
    errors: list[str] = []

    try:
        if recrawl:
            data = _api_post("/crawl/single", {
                "url": url,
                "markdown": True,
                "images": True,
                "screenshots": False,
                "quality_check": False,
                "max_images": max_images,
                "include_page_data": True,
                "max_inline_chars": max_inline_chars,
            })
            page = data.get("page") or {}
            source = "crawl_api"
        else:
            page = _api_post("/artifacts/page", {
                "url": url,
                "include_markdown": True,
                "include_links": True,
                "include_images": True,
                "max_images": max_images,
                "max_inline_chars": max_inline_chars,
            })
            source = "artifact_api"
    except Exception as exc:
        errors.append(str(exc))
        return {"url": url, "status": "error", "source": "api", "page": {}, "errors": errors}

    page_errors = page.get("errors") if isinstance(page, dict) else []
    if isinstance(page_errors, list):
        errors.extend(str(e) for e in page_errors)

    return {
        "url": url,
        "status": page.get("status", "error") if isinstance(page, dict) else "error",
        "source": source,
        "page": page,
        "errors": errors,
    }
```

Important:

```text
Do not use crawl_page.fn from research.
Use shared helper _api_post inside research instead.
```

---

## 9. Step M5 - Implement research tool

Implementation logic:

```python
@mcp.tool(annotations={"readOnlyHint": True})
async def research(
    question: str,
    site: str,
    max_pages: int = 10,
    max_images: int = 3,
    max_inline_chars: int = 12000,
    recrawl: bool = True,
    ctx: Context | None = None,
) -> dict:
    errors: list[str] = []
    site_norm = _normalize_site(site)
    max_pages = max(1, min(int(max_pages), 30))

    try:
        _require_env()
    except Exception as exc:
        return {
            "question": question,
            "site": site_norm,
            "status": "error",
            "search_results": 0,
            "relevant_found": 0,
            "pages_requested": 0,
            "pages_crawled": 0,
            "pages_failed": 0,
            "pages": [],
            "failed_pages": [],
            "errors": [str(exc)],
        }

    await _ctx_info(ctx, f"Searching site:{site_norm} for: {question}")
    try:
        search_results = _google_search(question, site_norm, num=min(max_pages * 2, 20))
    except Exception as exc:
        return {
            "question": question,
            "site": site_norm,
            "status": "error",
            "search_results": 0,
            "relevant_found": 0,
            "pages_requested": 0,
            "pages_crawled": 0,
            "pages_failed": 0,
            "pages": [],
            "failed_pages": [],
            "errors": [f"search failed: {exc}"],
        }

    search_results = [r for r in search_results if _same_or_subdomain(r.get("url", ""), site_norm)]

    await _ctx_info(ctx, f"Filtering {len(search_results)} search results")
    relevant = _filter_relevant(question, search_results, max_pages=max_pages)
    urls = [r["url"] for r in relevant]

    if not urls:
        return {
            "question": question,
            "site": site_norm,
            "status": "ok",
            "search_results": len(search_results),
            "relevant_found": 0,
            "pages_requested": 0,
            "pages_crawled": 0,
            "pages_failed": 0,
            "pages": [],
            "failed_pages": [],
            "errors": [],
        }

    await _ctx_info(ctx, f"Requesting {len(urls)} pages from crawler API")
    try:
        if recrawl:
            api_data = _api_post("/crawl/urls", {
                "urls": urls,
                "markdown": True,
                "images": True,
                "screenshots": False,
                "quality_check": False,
                "max_images": max_images,
                "threads": 4,
                "include_page_data": True,
                "max_inline_chars": max_inline_chars,
            })
        else:
            api_data = _api_post("/artifacts/pages", {
                "urls": urls,
                "include_markdown": True,
                "include_links": True,
                "include_images": True,
                "max_images": max_images,
                "max_inline_chars": max_inline_chars,
            })
    except Exception as exc:
        return {
            "question": question,
            "site": site_norm,
            "status": "error",
            "search_results": len(search_results),
            "relevant_found": len(relevant),
            "pages_requested": len(urls),
            "pages_crawled": 0,
            "pages_failed": len(urls),
            "pages": [],
            "failed_pages": [{"url": u, "error": str(exc)} for u in urls],
            "errors": [str(exc)],
        }

    pages = api_data.get("pages", []) if isinstance(api_data, dict) else []
    pages = _attach_relevance(pages, relevant)
    ok_pages = [p for p in pages if p.get("status") == "ok"]
    failed_pages = [
        {"url": p.get("url", ""), "error": "; ".join(p.get("errors", [])) or "crawl failed"}
        for p in pages
        if p.get("status") != "ok"
    ]

    return {
        "question": question,
        "site": site_norm,
        "status": "ok" if not failed_pages else "partial",
        "search_results": len(search_results),
        "relevant_found": len(relevant),
        "pages_requested": len(urls),
        "pages_crawled": len(ok_pages),
        "pages_failed": len(failed_pages),
        "pages": pages,
        "failed_pages": failed_pages,
        "errors": errors,
    }
```

---

## 10. Step M6 - Add docker-compose service

Add a second service. Keep API service unchanged.

```yaml
  mcp-research:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: crawler-mcp
    restart: unless-stopped
    env_file:
      - .env
    depends_on:
      - crawler
    ports:
      - "8115:8115"
    environment:
      MCP_HOST: "0.0.0.0"
      MCP_PORT: "8115"
      CRAWLER_API_BASE_URL: "http://crawler:8114"
      SERPER_API_KEY: "${SERPER_API_KEY}"
      DEEPSEEK_API_KEY: "${DEEPSEEK_API_KEY}"
      DEEPSEEK_MODEL: "${DEEPSEEK_MODEL:-deepseek-chat}"
    volumes:
      - ./app:/crawler/app
    entrypoint: ["python", "mcp_research.py"]
```

Important:

```text
No sites_crawled volume is required by MCP because MCP does not read files.
No chrome_profile volume is required by MCP because MCP does not run a browser.
No shm_size is required by MCP because MCP does not run Chromium.
```

---

## 11. Step M7 - Test MCP after API is running

Start API first:

```bat
cd D:\Docker\crawler
docker compose up -d crawler
```

Test API health:

```bat
curl http://localhost:8114/health
```

Start MCP:

```bat
docker compose up -d mcp-research
```

Test MCP health:

```bat
curl http://localhost:8115/health
```

Expected:

```json
{
  "status": "ok",
  "crawler_api_ok": true
}
```

Test MCP with a direct tool call from your MCP client:

```text
crawl_page(url="https://www.prolicht.at", max_images=3, max_inline_chars=8000, recrawl=true)
```

Expected:

```text
status = ok
page.markdown is non-empty
page.markdown_url is present if public base URL is configured
page.links is a list
page.images is a list
```

Test research:

```text
research(question="technical data downlights", site="prolicht.at", max_pages=3, max_images=2)
```

Expected:

```text
search_results > 0
relevant_found >= 0
pages is a list
No browser errors in MCP logs
Only API container has crawler/browser logs
```

---

## 12. Common mistakes to avoid

Do not do this:

```python
import crawler
import file_server
browser, tab, domain, out_dir, consent = await crawler._prepare_browser(...)
```

Reason:

```text
The API already owns browser lifecycle and artifact URLs.
Duplicating that in MCP reintroduces the broken shared-browser/parallelism risk.
```

Do not call API artifact endpoints before crawling unless `recrawl=false` is intentional.

Do not let DeepSeek relevance failures stop the whole research request.

Do not let one failed page stop the whole batch.

Do not return raw API keys, environment values, or stack traces to the LLM.

---

## 13. Done definition for MCP job

The MCP job is done only when all of these are true:

```text
MCP container starts.
GET /health on MCP returns crawler_api_ok=true.
MCP code does not import crawler.py, file_server.py, or nodriver.
crawl_page recrawl=true calls /crawl/single and returns page data.
crawl_page recrawl=false calls /artifacts/page and returns already-created page data.
research searches via Serper, filters via DeepSeek, calls /crawl/urls, and returns pages.
When API is down, MCP returns clear API error messages instead of hanging.
When DeepSeek fails for one URL, research continues.
When one crawl page fails, research returns partial result.
```

---

## 14. Final architecture after both plans

```text
LLM / Open WebUI / Claude
  -> MCP tool call on port 8115
     -> app/mcp_research.py
        -> serper.dev search
        -> DeepSeek relevance filtering
        -> HTTP call to crawler API on port 8114
           -> app/server.py
              -> crawler.py
              -> file_server.py
              -> /crawler/sites_crawled artifacts
        -> returns structured research result to LLM
```

This is the intended final separation.
