# TODO_ANALYISIS_MCP.md

## Purpose

This file defines what belongs in the MCP layer for LLM-assisted parser creation.

The MCP is the LLM-facing orchestration layer. It should expose simple tools that call the REST API. It must not own browser automation, direct Gemini SDK calls, database persistence, parser execution internals or crawler internals.

The current MCP style is correct: it should call the crawler API and not import crawler/browser modules.

---

## 1. Core MCP rule

The MCP should provide thin, safe, structured tools for an LLM agent:

- Inspect a URL.
- Get screenshot/DOM/action summaries.
- Ask the API to run Gemini analysis.
- Click an action through the API.
- Test selectors and extractors through the API.
- Save parser manifests through the API.
- Validate parser manifests through the API.
- Run validated parsers through the API.

The MCP should not contain:

- Browser automation code.
- Playwright/nodriver imports.
- Screenshot generation code.
- Gemini SDK code.
- SQL write logic.
- Cache/deduplication logic.
- Parser execution logic.

---

## 2. MCP environment variables

Add/ensure:

```env
CRAWLER_API_BASE_URL=http://crawler:8114
MCP_HTTP_TIMEOUT_SECONDS=180
```

Keep search-related keys if the research MCP uses them:

```env
SERPER_API_KEY=...
DEEPSEEK_API_KEY=...
DEEPSEEK_MODEL=deepseek-chat
```

The MCP does not need `GEMINI_API_KEY`. Gemini belongs in the API.

---

## 3. MCP tool group: page inspection

### TODO 3.1 - `inspect_page`

Purpose:

Open a URL in the API browser session and return a compact inspection package.

Tool signature:

```python
async def inspect_page(
    url: str,
    capture_screenshot: bool = True,
    capture_network: bool = True,
    max_actions: int = 80,
    max_dom_nodes: int = 300,
    max_inline_chars: int = 12000,
    ctx: Context | None = None,
) -> dict:
    ...
```

API calls:

```text
POST /browser/sessions
GET  /browser/sessions/{session_id}/snapshot
GET  /browser/sessions/{session_id}/dom
GET  /browser/sessions/{session_id}/actions
```

Return shape:

```json
{
  "status": "ok",
  "session_id": "sess_...",
  "url": "...",
  "final_url": "...",
  "snapshot": {},
  "dom_summary": {},
  "actions": []
}
```

MCP behavior:

- Return compact data only.
- Include artifact paths from the API.
- Do not inline huge HTML.
- Report progress through `ctx`.

---

### TODO 3.2 - `get_page_snapshot`

Purpose:

Fetch the latest snapshot for an existing session.

Tool signature:

```python
async def get_page_snapshot(
    session_id: str,
    max_inline_chars: int = 12000,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
GET /browser/sessions/{session_id}/snapshot
```

---

### TODO 3.3 - `get_page_actions`

Purpose:

Fetch current clickable actions from a live browser session.

Tool signature:

```python
async def get_page_actions(
    session_id: str,
    max_actions: int = 150,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
GET /browser/sessions/{session_id}/actions
```

---

## 4. MCP tool group: Gemini-assisted analysis

### TODO 4.1 - `analyze_page_with_gemini`

Purpose:

Ask the API to send screenshot + visible text + DOM summary + actions to Gemini and return structured page/parser discovery analysis.

Tool signature:

```python
async def analyze_page_with_gemini(
    session_id: str,
    analysis_type: str = "page_parser_discovery",
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /analysis/gemini/page
```

Request passed to API:

```json
{
  "session_id": "sess_...",
  "analysis_type": "page_parser_discovery",
  "include_screenshot": true,
  "include_visible_text": true,
  "include_dom_summary": true,
  "include_actions": true,
  "required_json_schema": "site_parser_discovery_v1"
}
```

MCP behavior:

- Do not call Gemini directly.
- Do not require `GEMINI_API_KEY` in MCP.
- Return the API result unchanged except for optional compacting.

---

### TODO 4.2 - `classify_page`

Purpose:

Ask the API to classify the currently loaded page.

Tool signature:

```python
async def classify_page(
    session_id: str,
    use_gemini: bool = True,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /analysis/classify-page
```

Expected classifications:

- `normal_page`
- `catalogue_context_page`
- `product_listing_page`
- `product_family_page`
- `product_detail_page`
- `product_configurator_entry`
- `product_configurator_state`
- `product_modal`
- `navigation_overlay`
- `unknown_interactive`

---

## 5. MCP tool group: action testing

### TODO 5.1 - `click_action`

Purpose:

Click a known action returned by `inspect_page` or `get_page_actions`.

Tool signature:

```python
async def click_action(
    session_id: str,
    action_id: str,
    wait_strategy: str = "dom_or_network_change",
    timeout_seconds: int = 10,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /browser/sessions/{session_id}/actions/{action_id}/click
```

MCP behavior:

- Report the normalized outcome.
- Include before/after snapshot IDs.
- Include diff hints from API if available.

---

### TODO 5.2 - `click_selector`

Purpose:

Let the LLM test a proposed selector.

Tool signature:

```python
async def click_selector(
    session_id: str,
    selector: str,
    wait_strategy: str = "dom_or_network_change",
    timeout_seconds: int = 10,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /browser/sessions/{session_id}/click-selector
```

---

### TODO 5.3 - `repeat_action`

Purpose:

Repeat a button/action, e.g. XAL `Show more`, until stable.

Tool signature:

```python
async def repeat_action(
    session_id: str,
    action_id: str,
    until: str = "row_count_stable",
    max_clicks: int = 50,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /browser/sessions/{session_id}/actions/{action_id}/repeat
```

Use cases:

- XAL `Show more`.
- Lazy-loaded product catalogue lists.
- Product-code tables with pagination/load-more controls.

---

## 6. MCP tool group: network and diff inspection

### TODO 6.1 - `get_network_events`

Purpose:

Return captured XHR/fetch responses and their API-side classifications.

Tool signature:

```python
async def get_network_events(
    session_id: str,
    only_relevant: bool = True,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
GET /browser/sessions/{session_id}/network
```

Use cases:

- Find JSON responses behind XAL `Show more`.
- Find PROLICHT configurator state responses.
- Discover download APIs.

---

### TODO 6.2 - `get_snapshot_diff`

Purpose:

Ask the API what changed after a click.

Tool signature:

```python
async def get_snapshot_diff(
    session_id: str,
    before_snapshot_id: str,
    after_snapshot_id: str,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
GET /browser/sessions/{session_id}/diffs/{before_snapshot_id}/{after_snapshot_id}
```

---

## 7. MCP tool group: selector and extractor testing

### TODO 7.1 - `test_selector`

Purpose:

Let the LLM test selector ideas against the live browser session.

Tool signature:

```python
async def test_selector(
    session_id: str,
    selector: str,
    return_text: bool = True,
    limit: int = 20,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /parser/test-selector
```

---

### TODO 7.2 - `test_extractor`

Purpose:

Let the LLM test an extractor JSON/YAML object before saving it into a parser manifest.

Tool signature:

```python
async def test_extractor(
    session_id: str,
    extractor: dict,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /parser/test-extractor
```

MCP behavior:

- Pass extractor object through to the API.
- Return extracted sample records and errors.
- Do not execute extraction logic inside MCP.

---

## 8. MCP tool group: parser projects

### TODO 8.1 - `create_parser_project`

Purpose:

Create or fetch a parser project for a domain.

Tool signature:

```python
async def create_parser_project(
    domain: str,
    parser_name: str,
    parser_type: str = "product_discovery",
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /parser/projects
```

---

### TODO 8.2 - `save_parser_manifest`

Purpose:

Save generated parser rules.

Tool signature:

```python
async def save_parser_manifest(
    project_id: str,
    manifest: dict,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
PUT /parser/projects/{project_id}/manifest
```

MCP behavior:

- Validate that the manifest is JSON-serializable.
- Do not run browser logic.
- Return API validation errors if the manifest is malformed.

---

### TODO 8.3 - `get_parser_manifest`

Purpose:

Retrieve existing parser rules for review or editing.

Tool signature:

```python
async def get_parser_manifest(
    project_id: str,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
GET /parser/projects/{project_id}/manifest
```

---

## 9. MCP tool group: parser validation and execution

### TODO 9.1 - `validate_parser`

Purpose:

Run a parser against test URLs and return validation errors/hints.

Tool signature:

```python
async def validate_parser(
    project_id: str,
    test_urls: list[str],
    required_fields: list[str] | None = None,
    min_products: int = 1,
    min_variants: int = 0,
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /parser/projects/{project_id}/validate
```

---

### TODO 9.2 - `run_parser`

Purpose:

Run a validated parser against one or multiple URLs.

Tool signature:

```python
async def run_parser(
    project_id: str,
    urls: list[str],
    persist: bool = True,
    cache_policy: str = "cache_first",
    ctx: Context | None = None,
) -> dict:
    ...
```

API call:

```text
POST /parser/projects/{project_id}/run
```

MCP behavior:

- This is for controlled parser execution, not general crawling.
- Return result summary and artifact references.

---

## 10. MCP tool group: existing research flow

The existing research functionality should remain separate from parser-creation tools.

Keep tools like:

- `crawl_page`
- `research`

But update them to support new API options where useful:

```python
async def crawl_page(
    url: str,
    max_images: int = 5,
    max_inline_chars: int = 0,
    recrawl: bool = True,
    mode: str = "page_crawl",
    cache_policy: str = "cache_first",
    ctx: Context | None = None,
) -> dict:
    ...
```

Possible API payload additions:

```json
{
  "mode": "page_crawl",
  "cache_policy": "cache_first",
  "javascript_navigation": true,
  "open_modals": true,
  "include_page_data": true
}
```

---

## 11. Recommended LLM parser creation workflow via MCP

### Step 1 - Inspect page

Tool:

```text
inspect_page(url)
```

The LLM receives:

- Session ID.
- Screenshot path.
- Visible text preview.
- DOM summary.
- Action list.

### Step 2 - Analyze with Gemini

Tool:

```text
analyze_page_with_gemini(session_id)
```

The LLM receives:

- Page class.
- Candidate extraction regions.
- Recommended actions.
- Candidate selectors.

### Step 3 - Test actions

Tools:

```text
click_action(session_id, action_id)
repeat_action(session_id, action_id)
get_snapshot_diff(session_id, before, after)
```

The LLM learns:

- Which buttons reveal rows.
- Which buttons open modals.
- Which buttons change configurator states.

### Step 4 - Test selectors/extractors

Tools:

```text
test_selector(session_id, selector)
test_extractor(session_id, extractor)
```

The LLM iterates until fields are extracted correctly.

### Step 5 - Save parser manifest

Tools:

```text
create_parser_project(domain, parser_name)
save_parser_manifest(project_id, manifest)
```

### Step 6 - Validate parser

Tool:

```text
validate_parser(project_id, test_urls)
```

### Step 7 - Run parser

Tool:

```text
run_parser(project_id, urls)
```

---

## 12. Example MCP flow for XAL

1. `inspect_page("https://www.xal.com/en/products/p/...")`
2. `analyze_page_with_gemini(session_id)`
3. Identify page as `product_family_page`.
4. `click_action` for `Products and codes` if required.
5. `repeat_action` for `Show more`.
6. `test_selector` for product title `h1`.
7. `test_extractor` for `Products and codes` table.
8. Save parser manifest with:
   - URL pattern `/products/p/`
   - Markers `Quickinfo`, `Products and codes`, `Downloads & links`
   - Repeat action `Show more`
   - Extractor for variant rows.
9. Validate on 3-5 XAL product URLs.

---

## 13. Example MCP flow for PROLICHT

1. `inspect_page("https://www.prolicht.at/en/product-configurator/product-families/...")`
2. `analyze_page_with_gemini(session_id)`
3. Identify page as `product_configurator_entry`.
4. `get_page_actions(session_id)`.
5. Click configurator step actions:
   - `01 Design`
   - `02 Technical Data`
   - `03 Light`
6. Test extractors for:
   - Current article number.
   - Selected options.
   - Option groups.
   - Downloads.
7. Check network events for configurator state JSON.
8. Save parser manifest.
9. Validate on several PROLICHT product-family configurator URLs.

---

## 14. MCP error handling

Every MCP tool should return:

```json
{
  "status": "ok|partial|error",
  "errors": [],
  "warnings": []
}
```

When API call fails:

- Include endpoint path.
- Include HTTP status.
- Include first 500 chars of response body.
- Do not expose stack traces unless debug mode is enabled.

---

## 15. MCP implementation pattern

Use a shared API helper:

```python
def _api_get(path: str, timeout: float | None = None) -> dict:
    ...


def _api_post(path: str, payload: dict, timeout: float | None = None) -> dict:
    ...


def _api_put(path: str, payload: dict, timeout: float | None = None) -> dict:
    ...


def _api_delete(path: str, timeout: float | None = None) -> dict:
    ...
```

All MCP tools should call these helpers.

---

## 16. Final MCP boundary

MCP should be the LLM tool interface.

API should be the browser/parser engine.

Generated site parsers should be stored and executed by the API.

This separation keeps the MCP safe, simple and stable while allowing the API to evolve into a full parser-building platform.
