# TODO_ANALYSIS_API.md

## Purpose

This file defines what belongs in the REST API layer so an LLM agent can build reliable, reusable site parsers for websites such as `prolicht.at`, `xal.com`, Lightnet and other manufacturer catalogues.

The REST API is the execution engine. It owns browser automation, screenshots, DOM snapshots, JavaScript clicking, network capture, cache/deduplication, Gemini analysis, parser storage, parser validation and production parser runs.

The MCP must stay thin and should call this API instead of importing crawler/browser internals.

---

## 1. Core architecture rule

The API must expose browser state and parser-building primitives to the LLM/MCP, but the LLM must not directly control Playwright/nodriver/browser internals.

The API should answer questions like:

- Open this URL and give me a stable snapshot.
- Show me the simplified DOM.
- Show me clickable elements.
- Click this action and tell me what changed.
- Send screenshot + DOM summary to Gemini.
- Test this selector.
- Test this extractor.
- Save this parser manifest.
- Validate this parser against fixture URLs.
- Run this parser in production mode.

---

## 2. Required environment variables

Add these to the API container/service:

```env
GEMINI_API_KEY=...
GEMINI_MODEL=gemini-flash-lite-latest
```

Optional but recommended:

```env
ANALYSIS_ARTIFACT_DIR=/data/analysis_artifacts
PARSER_PROJECTS_DIR=/data/parser_projects
BROWSER_SESSION_TTL_SECONDS=1800
MAX_SCREENSHOT_BYTES_FOR_GEMINI=8000000
MAX_DOM_NODES_FOR_LLM=500
MAX_ACTIONS_FOR_LLM=150
```

---

## 3. API responsibility overview

The API must own:

- Browser sessions.
- Page loading and rendering.
- Screenshot creation.
- HTML capture.
- Visible text extraction.
- Simplified DOM extraction.
- Clickable/action extraction.
- JavaScript link/button clicking.
- DOM/network diffing after actions.
- Network/XHR capture and response storage.
- Cache and already-seen handling.
- Action registry and action replay.
- Gemini page analysis.
- Selector/extractor testing.
- Parser manifest storage.
- Parser validation.
- Production parser execution.
- Product/catalogue/configurator persistence.

The API must not rely on the LLM for production crawling decisions once a parser is validated.

---

## 4. Browser session endpoints

### TODO 4.1 - Create browser session

Add:

```http
POST /browser/sessions
```

Request:

```json
{
  "url": "https://www.xal.com/en/products/p/...",
  "viewport": { "width": 1920, "height": 1080 },
  "wait_until": "network_idle",
  "capture_screenshot": true,
  "capture_html": true,
  "capture_network": true,
  "mode": "parser_discovery"
}
```

Response:

```json
{
  "session_id": "sess_...",
  "page_key": "sha256...",
  "url": "...",
  "final_url": "...",
  "status": "ok"
}
```

Implementation notes:

- Keep the browser page alive for iterative inspection/clicking.
- Store initial snapshot immediately.
- Capture cookies/session state if required.
- Use cache only for persisted artifacts, not for the live session itself.

---

### TODO 4.2 - Get browser session snapshot

Add:

```http
GET /browser/sessions/{session_id}/snapshot
```

Response:

```json
{
  "session_id": "sess_...",
  "snapshot_id": "snap_...",
  "url": "...",
  "final_url": "...",
  "title": "...",
  "html_path": "...",
  "screenshot_path": "...",
  "visible_text_path": "...",
  "visible_text_preview": "...",
  "dom_digest": "sha256...",
  "content_digest": "sha256...",
  "page_class": "unknown",
  "product_page_role": null,
  "clickable_count": 0,
  "network_event_count": 0
}
```

Implementation notes:

- Save every snapshot to disk and DB.
- Return paths and short previews, not huge inline HTML by default.
- Support optional `max_inline_chars`.

---

### TODO 4.3 - Close browser session

Add:

```http
DELETE /browser/sessions/{session_id}
```

Implementation notes:

- Release browser/page resources.
- Keep artifacts and DB records.

---

## 5. Simplified DOM endpoint

### TODO 5.1 - Expose LLM-friendly DOM

Add:

```http
GET /browser/sessions/{session_id}/dom
```

Response shape:

```json
{
  "session_id": "sess_...",
  "snapshot_id": "snap_...",
  "url": "...",
  "nodes": [
    {
      "node_id": "n_001",
      "tag": "button",
      "text": "Show more",
      "normalized_text": "show more",
      "role": "button",
      "aria_label": null,
      "title": null,
      "href": null,
      "onclick": null,
      "classes": ["..."],
      "data_attrs": {},
      "section_heading": "Products and codes",
      "nearby_text": "...",
      "bbox": { "x": 0, "y": 0, "width": 0, "height": 0 },
      "visible": true,
      "enabled": true,
      "selector_candidates": [
        "button:has-text('Show more')",
        "text=Show more"
      ]
    }
  ]
}
```

Implementation notes:

- Do not return the entire raw DOM to the LLM.
- Include only visible and relevant nodes by default.
- Include headings, buttons, links, cards, tabs, inputs, tables and major sections.
- Generate multiple selector candidates for each node.
- Include stable node IDs that can be used by action endpoints.

---

## 6. Clickable action discovery

### TODO 6.1 - Expose clickable actions

Add:

```http
GET /browser/sessions/{session_id}/actions
```

Response:

```json
{
  "session_id": "sess_...",
  "actions": [
    {
      "action_id": "act_...",
      "node_id": "n_001",
      "action_type": "button",
      "text": "Show more",
      "normalized_text": "show more",
      "role": "button",
      "tag": "button",
      "href": null,
      "section_heading": "Products and codes",
      "bbox": { "x": 0, "y": 0, "width": 0, "height": 0 },
      "selector_candidates": ["button:has-text('Show more')"],
      "action_signature": "sha256...",
      "known_action": true,
      "last_successful_selector": "button:has-text('Show more')",
      "expected_outcome": "table_rows_added"
    }
  ]
}
```

Action candidates must include:

- `<a href>` links.
- `href="javascript:..."` links.
- `href="#"` links.
- Buttons.
- Elements with `onclick`.
- Elements with `role="button"` or `role="link"`.
- Product cards with pointer cursor.
- Tabs.
- Accordions.
- Dropdown controls.
- “Show more” / “Load more” controls.
- Configurator option cards.

---

## 7. Action execution and outcome detection

### TODO 7.1 - Click action by action ID

Add:

```http
POST /browser/sessions/{session_id}/actions/{action_id}/click
```

Request:

```json
{
  "wait_strategy": "dom_or_network_change",
  "timeout_seconds": 10,
  "capture_after": true,
  "register_success": true
}
```

Response:

```json
{
  "status": "ok",
  "outcome": "table_rows_added",
  "url_before": "...",
  "url_after": "...",
  "snapshot_before_id": "snap_1",
  "snapshot_after_id": "snap_2",
  "dom_digest_before": "sha256...",
  "dom_digest_after": "sha256...",
  "visible_text_digest_before": "sha256...",
  "visible_text_digest_after": "sha256...",
  "new_nodes": 12,
  "removed_nodes": 0,
  "added_rows": 24,
  "new_network_events": 2,
  "screenshot_after_path": "...",
  "html_after_path": "..."
}
```

Supported normalized outcomes:

- `url_changed`
- `modal_opened`
- `content_changed`
- `table_rows_added`
- `dropdown_opened`
- `download_started`
- `api_response_received`
- `article_number_changed`
- `configurator_state_changed`
- `no_change`
- `failed`

Implementation notes:

- Always capture a before/after snapshot.
- Compare URL, visible text digest, DOM digest, row counts and network events.
- Register successful actions in the action registry.

---

### TODO 7.2 - Click by selector

Add:

```http
POST /browser/sessions/{session_id}/click-selector
```

Request:

```json
{
  "selector": "button:has-text('Show more')",
  "wait_strategy": "dom_or_network_change",
  "timeout_seconds": 10
}
```

Use this for LLM-generated selector tests.

---

### TODO 7.3 - Repeat action until stable

Add:

```http
POST /browser/sessions/{session_id}/actions/{action_id}/repeat
```

Request:

```json
{
  "until": "row_count_stable",
  "max_clicks": 50,
  "timeout_seconds_per_click": 10
}
```

Response:

```json
{
  "status": "ok",
  "clicks": 7,
  "stop_reason": "row_count_stable",
  "row_count_before": 12,
  "row_count_after": 96
}
```

Use cases:

- XAL `Show more` buttons.
- Catalogue lists with lazy loading.

---

## 8. Action registry and replay

### TODO 8.1 - Create action registry table

```sql
CREATE TABLE action_registry (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    domain VARCHAR(255) NOT NULL,
    action_signature CHAR(64) NOT NULL,

    normalized_text TEXT NULL,
    role VARCHAR(64) NULL,
    tag_name VARCHAR(64) NULL,
    section_heading TEXT NULL,

    selector_primary TEXT NULL,
    selector_fallback_json JSON NULL,
    expected_outcome VARCHAR(64) NULL,

    first_seen_url TEXT NULL,
    last_seen_url TEXT NULL,

    success_count INT NOT NULL DEFAULT 0,
    fail_count INT NOT NULL DEFAULT 0,

    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,

    UNIQUE KEY uq_action_registry (domain, action_signature)
);
```

### TODO 8.2 - Match known actions on future pages

When an action appears again, match by:

1. Exact selector.
2. Visible text + role + section heading.
3. Aria label.
4. Data attributes.
5. DOM neighborhood.
6. Bounding-box region fallback.

### TODO 8.3 - Track action reliability

Store:

- Success count.
- Failure count.
- Last successful selector.
- Expected outcome.
- Last seen URL.

---

## 9. Network/XHR capture

### TODO 9.1 - Capture network events

Add:

```http
GET /browser/sessions/{session_id}/network
```

Response:

```json
{
  "session_id": "sess_...",
  "events": [
    {
      "event_id": "net_001",
      "trigger_action_id": "act_show_more_001",
      "method": "GET",
      "url": "https://...",
      "resource_type": "xhr",
      "status": 200,
      "content_type": "application/json",
      "response_digest": "sha256...",
      "response_preview": "{...}",
      "response_path": "...",
      "classified_as": "variant_rows"
    }
  ]
}
```

Classify responses as:

- `variant_rows`
- `product_details`
- `downloads`
- `filters`
- `search_results`
- `configurator_state`
- `article_number_update`
- `unknown_json`

Implementation notes:

- Store JSON responses as artifacts.
- Link responses to page/session/action.
- Prefer structured API data over DOM scraping when available.

---

## 10. DOM diff endpoint

### TODO 10.1 - Expose snapshot diffs

Add:

```http
GET /browser/sessions/{session_id}/diffs/{before_snapshot_id}/{after_snapshot_id}
```

Response:

```json
{
  "outcome": "table_rows_added",
  "changed_sections": ["Products and codes"],
  "added_rows": 24,
  "removed_rows": 0,
  "new_text": ["090-7Q901C0031 ..."],
  "removed_text": []
}
```

Use this to help the LLM understand whether a click was useful.

---

## 11. Gemini analysis endpoint

### TODO 11.1 - Add Gemini page analysis

Add:

```http
POST /analysis/gemini/page
```

Use:

```text
GEMINI_API_KEY
model = gemini-flash-lite-latest
```

Request:

```json
{
  "session_id": "sess_...",
  "analysis_type": "page_parser_discovery",
  "include_screenshot": true,
  "include_visible_text": true,
  "include_dom_summary": true,
  "include_actions": true,
  "required_json_schema": "site_parser_discovery_v1"
}
```

Expected response:

```json
{
  "page_class": "product_family_page",
  "confidence": 0.92,
  "site": "xal.com",
  "main_regions": [
    {
      "region_name": "products_and_codes",
      "contains": ["filters", "variant_table", "show_more_button"]
    }
  ],
  "product_identity": {
    "product_name": "UNICO Q9 basic recessed",
    "product_family": "UNICO Q9"
  },
  "candidate_extractors": [
    {
      "field": "product_name",
      "method": "selector",
      "selector_hint": "h1"
    }
  ],
  "recommended_actions": [
    {
      "text": "Show more",
      "reason": "Loads additional product-code rows"
    }
  ]
}
```

Implementation notes:

- Send screenshot, visible text, simplified DOM and actions to Gemini.
- Ask Gemini for structured JSON only.
- Do not use Gemini for production extraction of every row.
- Use Gemini to discover page structure and candidate extraction logic.
- Cache Gemini analysis by `screenshot_digest + dom_digest + analysis_type`.

---

## 12. Page classification

### TODO 12.1 - Add classifier endpoint

Add:

```http
POST /analysis/classify-page
```

Input:

```json
{
  "session_id": "sess_...",
  "use_gemini": true
}
```

Output:

```json
{
  "page_class": "catalogue_context_page",
  "product_page_role": "catalogue_context",
  "has_product_catalogue": true,
  "product": false,
  "context_type": "application",
  "context_label": "Sports hall",
  "confidence": 0.88
}
```

Required page classes:

- `normal_page`
- `content_landing_page`
- `catalogue_context_page`
- `product_listing_page`
- `product_family_page`
- `product_detail_page`
- `product_configurator_entry`
- `product_configurator_state`
- `product_modal`
- `navigation_overlay`
- `unknown_interactive`

---

## 13. Selector and extractor test endpoints

### TODO 13.1 - Test selector

Add:

```http
POST /parser/test-selector
```

Request:

```json
{
  "session_id": "sess_...",
  "selector": "section:has-text('Products and codes') table tr",
  "return_text": true,
  "limit": 20
}
```

Response:

```json
{
  "status": "ok",
  "match_count": 24,
  "matches": [
    {
      "index": 0,
      "text": "STATUS ARTICLE NUMBER(S) TYPE DIMMING"
    }
  ]
}
```

### TODO 13.2 - Test extractor

Add:

```http
POST /parser/test-extractor
```

Request:

```json
{
  "session_id": "sess_...",
  "extractor": {
    "name": "xal_product_variants",
    "type": "table",
    "section_heading": "Products and codes",
    "row_selector": "...",
    "columns": {
      "article_number_raw": "ARTICLE NUMBER(S)",
      "type_description": "TYPE",
      "dimming": "DIMMING"
    }
  }
}
```

Response:

```json
{
  "status": "ok",
  "records": [],
  "errors": []
}
```

---

## 14. Parser project and manifest endpoints

### TODO 14.1 - Create parser project

Add:

```http
POST /parser/projects
```

Request:

```json
{
  "domain": "xal.com",
  "parser_name": "xal_product_parser",
  "parser_type": "product_discovery"
}
```

### TODO 14.2 - Save parser manifest

Add:

```http
PUT /parser/projects/{project_id}/manifest
```

Manifest format should support:

- Domain.
- Parser type.
- Page classifiers.
- Action rules.
- Repeat rules.
- Extractors.
- Validation checks.

Example fields:

```yaml
domain: xal.com
parser_type: product_discovery

page_classifiers:
  product_family_page:
    url_patterns:
      - "/products/p/"
    text_markers:
      - "Quickinfo"
      - "Products and codes"
      - "Downloads & links"

actions:
  show_more_variants:
    match:
      text: "Show more"
      section: "Products and codes"
    repeat_until:
      condition: "row_count_stable"
      max_clicks: 50

extractors:
  product_identity:
    fields:
      product_name:
        selector: "h1"
  variants:
    section_heading: "Products and codes"
    type: "table"
    load_more_action: "show_more_variants"
```

---

## 15. Parser validation and run endpoints

### TODO 15.1 - Validate parser

Add:

```http
POST /parser/projects/{project_id}/validate
```

Request:

```json
{
  "test_urls": ["https://www.xal.com/en/products/p/..."],
  "checks": {
    "min_products": 1,
    "min_variants": 10,
    "required_fields": ["product_name", "article_number_raw"]
  }
}
```

Response:

```json
{
  "status": "failed",
  "errors": [
    {
      "check": "min_variants",
      "expected": 10,
      "actual": 4,
      "hint": "Show more button may not have been clicked."
    }
  ],
  "sample_output": {}
}
```

### TODO 15.2 - Run parser

Add:

```http
POST /parser/projects/{project_id}/run
```

Request:

```json
{
  "urls": ["https://www.xal.com/en/products/p/..."],
  "persist": true,
  "cache_policy": "cache_first"
}
```

Response:

```json
{
  "status": "ok",
  "pages_processed": 1,
  "products": [],
  "variants": [],
  "downloads": [],
  "errors": []
}
```

---

## 16. Data model additions

### TODO 16.1 - Pages

Add/ensure:

```sql
ALTER TABLE pages
  ADD COLUMN page_class VARCHAR(64) NULL,
  ADD COLUMN product_page_role VARCHAR(64) NULL,
  ADD COLUMN has_product_catalogue BOOLEAN NOT NULL DEFAULT FALSE,
  ADD COLUMN product BOOLEAN NOT NULL DEFAULT FALSE,
  ADD COLUMN product_entry_url TEXT NULL,
  ADD COLUMN context_type VARCHAR(64) NULL,
  ADD COLUMN context_label TEXT NULL;
```

### TODO 16.2 - Products

```sql
CREATE TABLE products (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    product_key CHAR(64) NOT NULL UNIQUE,
    domain VARCHAR(255) NOT NULL,
    canonical_product_url TEXT NULL,
    product_name TEXT NULL,
    product_code TEXT NULL,
    product_family TEXT NULL,
    product_type TEXT NULL,
    product_page_role VARCHAR(64) NULL,
    identity_confidence DECIMAL(5,2) NULL,
    identity_source VARCHAR(64) NULL,
    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
```

### TODO 16.3 - Product variants

```sql
CREATE TABLE product_variants (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    variant_key CHAR(64) NOT NULL UNIQUE,
    product_key CHAR(64) NOT NULL,
    article_number TEXT NULL,
    article_number_raw TEXT NULL,
    variant_name TEXT NULL,
    type_description TEXT NULL,
    dimming TEXT NULL,
    status TEXT NULL,
    luminous_flux TEXT NULL,
    power TEXT NULL,
    efficacy TEXT NULL,
    color_temperature TEXT NULL,
    cri TEXT NULL,
    ugr TEXT NULL,
    ip_rate TEXT NULL,
    source_url TEXT NULL,
    source_section TEXT NULL,
    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL
);
```

### TODO 16.4 - Page/product references

```sql
CREATE TABLE page_product_references (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    page_key CHAR(64) NOT NULL,
    product_key CHAR(64) NOT NULL,
    context_type VARCHAR(64) NULL,
    context_label TEXT NULL,
    visible_product_name TEXT NULL,
    visible_card_text TEXT NULL,
    product_url TEXT NULL,
    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,
    UNIQUE KEY uq_page_product_reference (page_key, product_key, context_type, context_label)
);
```

### TODO 16.5 - Option groups and values

```sql
CREATE TABLE product_option_groups (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    product_key CHAR(64) NOT NULL,
    group_name TEXT NOT NULL,
    group_type VARCHAR(64) NULL,
    source_section TEXT NULL,
    position_index INT NULL
);
```

```sql
CREATE TABLE product_option_values (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    product_key CHAR(64) NOT NULL,
    group_id BIGINT NOT NULL,
    value_text TEXT NOT NULL,
    value_code TEXT NULL,
    value_unit TEXT NULL,
    selected_by_default BOOLEAN NULL,
    disabled BOOLEAN NULL,
    position_index INT NULL
);
```

### TODO 16.6 - Downloads

```sql
CREATE TABLE product_downloads (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    product_key CHAR(64) NOT NULL,
    variant_key CHAR(64) NULL,
    download_label TEXT NULL,
    download_type VARCHAR(64) NULL,
    url TEXT NOT NULL,
    canonical_url TEXT NULL,
    source_section TEXT NULL,
    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL
);
```

---

## 17. Deduplication keys

Implement these key rules in the API:

```text
page_key = sha256("page:" + canonical_url)
product_key = sha256("product:" + domain + canonical_product_url)
product_key_weak = sha256("product_weak:" + domain + normalized_product_name)
variant_key = sha256("variant:" + domain + article_number)
variant_key_weak = sha256("variant_weak:" + product_key + row_text_digest)
reference_key = sha256("page_product:" + page_key + product_key + context_label)
configurator_state_key = sha256("state:" + product_entry_url + selected_option_path)
action_signature = sha256(domain + normalized_text + role + section_heading + href + stable_dom_context)
```

---

## 18. XAL-specific API support

The API must support discovery of:

- `/products/c/` catalogue context pages.
- `/products/p/` product/family pages.
- Product cards within category/application pages.
- Quickinfo section.
- Colour / mounting set / light distribution options.
- Products and codes variant table.
- `Show more` repeated loading.
- Downloads & links.
- Aggregated downloads.

Required helpers:

- Section finder by heading.
- Table extractor by section heading.
- Row-count tracking.
- Load-more repeat action.
- Product-card extractor.
- Product identity resolver.

---

## 19. PROLICHT-specific API support

The API must support discovery of:

- Product filter card overlays.
- JavaScript CTAs such as Configure now, Most configured, All about product.
- Product configurator entry pages.
- Configurator steps: Design, Technical Data, Light, Complete.
- Option tabs: Colors, Size, Sensor, Plug, Light Emission, Colourtemperature & LED, Diffuser, Indirect lighting coverage.
- Current article number.
- Selected option state.
- Downloads for the active state.
- XHR responses that update the active state.

Required helpers:

- Configurator detector.
- Selected-state extractor.
- Article-number extractor.
- Option taxonomy extractor.
- Network response classifier for configurator state.

---

## 20. Production parser principle

Gemini and the LLM are for parser discovery and validation.

Validated production parsers should run deterministically using:

- Saved parser manifest.
- Saved action signatures.
- Tested selectors.
- API browser actions.
- API extractor helpers.

Do not call Gemini on every production page unless an explicit `use_ai_fallback` option is enabled.

---
