# Discovery Implementation Plan

## 0. Purpose

This document describes how to extend the crawler system so it can handle:

1. Normal page crawling for website search/indexing.
2. JavaScript navigation links, overlays, tabs, modals, and product cards.
3. Product discovery as a separate mode for competitor product extraction.
4. Product catalogue pages where the same product can appear under multiple contexts, for example XAL Sports hall and XAL Recessed spotlights.
5. Product configurators where changing options can change article numbers, downloads, and valid product states, for example PROLICHT.
6. Persistent cache and duplicate handling so already known pages, JavaScript states, product references, products, variants, and downloads are not crawled or inserted repeatedly.
7. Gemini-based content analysis using `GEMINI_API_KEY` and the model `gemini-flash-lite-latest`.
8. A clear split between the REST API server and the MCP research/search server.

The goal is to give a junior developer a step-by-step implementation path.

---

## 1. Existing system assumptions

The system currently has two main parts:

1. A REST crawler API.
2. An MCP research/search server.

The MCP research server currently calls the crawler API and must not import crawler/browser internals directly. Keep this separation.

The MCP server is a user/research interface. The REST API is the execution engine.

---

## 2. High-level architecture

Use this split:

```text
User / LLM client
    |
    v
MCP Research Server
    - search orchestration
    - question-based site research
    - relevance filtering
    - calls REST API
    - returns compact results to LLM
    |
    v
REST Crawler API
    - browser automation
    - JavaScript clicks
    - network/XHR capture
    - cache and dedupe
    - page classification
    - Gemini content analysis
    - product discovery extraction
    - database writes
    - artifact storage
    |
    v
Database + artifacts
    - pages
    - crawl targets
    - products
    - product variants
    - page-product references
    - option groups/values
    - downloads
    - screenshots/html/markdown/json
```

Important rule:

```text
The MCP must not crawl directly.
The MCP must not import browser/crawler modules.
The MCP should call REST API endpoints only.
```

---

## 3. Responsibilities: REST API vs MCP

### 3.1 REST API responsibilities

The REST API owns everything that changes crawler state or requires browser execution.

REST API must handle:

- URL normalization and canonicalization.
- Cache-first lookup before browser navigation.
- Persistent crawl target queue.
- Browser rendering.
- JavaScript navigation clicks.
- Modal and overlay capture.
- Tab and accordion capture.
- XHR/fetch network capture.
- Screenshot capture.
- HTML/Markdown/Text artifact generation.
- Page classification.
- Gemini content/page analysis.
- Product discovery mode.
- Catalogue context extraction.
- Product card extraction.
- Product identity resolution.
- Product detail extraction.
- Product variant/article extraction.
- Product configurator option extraction.
- Product downloads extraction.
- Database writes and deduplication.

The REST API should expose clean endpoints for MCP and other clients.

### 3.2 MCP responsibilities

The MCP owns LLM-facing search and research orchestration.

MCP should handle:

- Searching the web or site-specific results, for example through Serper.
- Filtering search results for relevance.
- Calling REST API to crawl or fetch cached artifacts.
- Calling REST API to start product discovery jobs.
- Fetching existing results from REST API.
- Returning compact, LLM-friendly JSON.
- Reporting progress to the MCP client.

MCP should not handle:

- Browser automation.
- Direct DOM interaction.
- Direct database mutation.
- Direct screenshot processing.
- Direct product extraction from browser pages.
- Direct crawler cache decisions.

### 3.3 Gemini placement

Use Gemini in two places, but with different scopes.

#### REST API Gemini usage

This is the main place for Gemini.

Use Gemini for:

- Page classification when rule-based classification is not enough.
- Extracting structured page roles from rendered content.
- Determining whether a page is:
  - normal page
  - catalogue context page
  - product listing page
  - product detail page
  - product family page
  - product configurator entry
  - product configurator state
  - JavaScript modal/overlay
- Extracting catalogue context metadata.
- Extracting product option groups from complex configurator pages.
- Validating product-card extraction when DOM heuristics are uncertain.
- Normalizing labels from visible text.

#### MCP Gemini usage

Use Gemini only for lightweight research assistance.

Use Gemini for:

- Replacing or supplementing the current DeepSeek relevance filter.
- Ranking search results before asking the REST API to crawl them.
- Creating compact summaries from already-crawled page artifacts.

The MCP should not use Gemini to inspect browser state directly.

---

## 4. Required environment variables

Add these environment variables to the crawler API container:

```env
GEMINI_API_KEY=your_key_here
GEMINI_MODEL=gemini-flash-lite-latest
GEMINI_ANALYSIS_ENABLED=true
GEMINI_ANALYSIS_TIMEOUT_SECONDS=30
GEMINI_MAX_INPUT_CHARS=60000
```

Add these environment variables to the MCP container:

```env
CRAWLER_API_BASE_URL=http://crawler:8114
SERPER_API_KEY=your_key_here
GEMINI_API_KEY=your_key_here
GEMINI_MODEL=gemini-flash-lite-latest
MCP_HTTP_TIMEOUT_SECONDS=180
```

If you keep DeepSeek temporarily, use:

```env
DEEPSEEK_API_KEY=your_key_here
DEEPSEEK_MODEL=deepseek-chat
```

Long term, prefer one AI stack:

```text
MCP relevance filtering: Gemini
REST API page/product analysis: Gemini
```

---

## 5. Requirements update

Current MCP requirements are minimal.

Update the shared or API-specific requirements as needed.

Recommended API requirements:

```text
fastapi
uvicorn
requests
google-genai
pydantic
beautifulsoup4
lxml
pillow
python-multipart
```

Recommended MCP requirements:

```text
fastmcp
requests
google-genai
```

If the API already has browser dependencies, keep them where they are. Do not add browser automation dependencies to the MCP service.

---

## 6. Modes

Add a top-level mode field to crawl requests.

```json
{
  "mode": "page_crawl"
}
```

Supported values:

```text
page_crawl
product_discovery
catalogue_discovery
```

### 6.1 `page_crawl`

Purpose:

- Crawl pages for search/indexing.
- Discover normal links and JavaScript navigation.
- Store pages and artifacts.
- Avoid deep product variant/configurator extraction.

Behavior:

```text
- crawl normal pages
- click JavaScript navigation links
- open modals and overlays
- save tab/accordion content
- detect product-related pages
- mark product metadata
- do not enumerate product variants
- do not expand all catalogue filters
- do not traverse all configurator combinations
```

### 6.2 `product_discovery`

Purpose:

- Extract competitor product information.
- Extract product references, products, variants, options, article numbers, and downloads.

Behavior:

```text
- crawl catalogue context pages
- extract product cards
- deduplicate products across contexts
- open product pages
- extract product detail data
- click Show more buttons
- extract article/variant tables
- extract product downloads
- extract configurator options and selected states
- capture XHR/fetch responses
```

### 6.3 `catalogue_discovery`

Purpose:

- Discover which products exist and where they are referenced.
- Stop before deep product detail extraction.

Behavior:

```text
- crawl category/application/listing pages
- extract filters and product cards
- create page-product references
- enqueue product detail pages but do not necessarily crawl them now
```

---

## 7. Request options

Add these options to the REST API request model.

```json
{
  "mode": "page_crawl",
  "cache_policy": "cache_first",
  "new_pages_only": false,
  "changed_pages_only": false,
  "javascript_navigation": true,
  "open_modals": true,
  "capture_xhr": true,
  "gemini_analysis": true,
  "product_discovery": {
    "enabled": false,
    "extract_catalogue_contexts": true,
    "extract_product_cards": true,
    "extract_product_pages": true,
    "extract_variant_tables": true,
    "click_show_more": true,
    "extract_downloads": true,
    "filter_strategy": "visible_values_only",
    "configurator_strategy": "taxonomy_only",
    "max_show_more_clicks": 50,
    "max_product_variant_rows": 5000,
    "max_configurator_states": 200
  }
}
```

### 7.1 Cache policies

Supported values:

```text
cache_first
revalidate
refresh
force
```

Meaning:

```text
cache_first:
    If a valid artifact exists, return it without opening browser.

revalidate:
    Check whether a known page changed. Use validators if possible.

refresh:
    Open browser/API again, but skip expensive processing if content digest is unchanged.

force:
    Ignore cache and crawl again.
```

### 7.2 `new_pages_only`

This means:

```text
Only crawl targets that have never been seen before.
```

It is different from cache.

If `target_key` already exists:

```text
- do not enqueue
- mark as already_known
- update last_seen_run_id if this is a discovery run
```

### 7.3 `changed_pages_only`

This means:

```text
Known pages may be checked/revalidated, but only saved/processed again if content changed.
```

---

## 8. URL canonicalization

Do not strip all query parameters.

For product/configurator URLs, query parameters can be meaningful.

Example:

```text
https://www.prolicht.at/en/product-configurator/product-families/hypro/hypro-f-160-free-standing/?tab=color&MountingType=freistehend
```

The query values `tab=color` and `MountingType=freistehend` can represent product state.

### 8.1 Drop tracking parameters

Drop:

```text
utm_*
fbclid
gclid
msclkid
rnd
cache
timestamp
```

### 8.2 Keep semantic parameters

Keep:

```text
tab
MountingType
mounting_type
filter
category
ProductFamily
ProductCategory
lang
variant
configuration
```

### 8.3 Canonical URL rule

Use:

```text
lowercase scheme + lowercase host + normalized path + sorted semantic query parameters
```

Do not lowercase query values globally.

---

## 9. Target keys and duplicate handling

The crawler must not only ask:

```text
Did I already crawl this URL?
```

It must ask:

```text
Did I already know this target?
Did I already crawl this page?
Did I already see this product?
Did I already see this product on this page?
Did I already extract this filter state?
Did I already extract this configurator state?
Did I already store this variant/article row?
```

### 9.1 Target key examples

Normal page:

```text
target_key = sha256("page:" + canonical_url)
```

JavaScript action:

```text
action_signature = sha256(parent_canonical_url + visible_text + role + href + aria_label + stable_dom_path)
```

Modal:

```text
target_key = sha256("modal:" + parent_canonical_url + modal_title + modal_text_digest)
```

Content state:

```text
target_key = sha256("state:" + parent_canonical_url + action_signature + visible_content_digest)
```

Product:

```text
product_key = sha256("product:" + domain + canonical_product_url)
```

Weak product fallback:

```text
product_key = sha256("product_weak:" + domain + normalized_product_name)
```

Variant/article:

```text
variant_key = sha256("variant:" + domain + article_number)
```

Weak variant fallback:

```text
variant_key = sha256("variant_weak:" + product_key + row_text_digest)
```

Page-product reference:

```text
reference_key = sha256("page_product:" + page_key + product_key + context_label)
```

Configurator state:

```text
state_key = sha256("configurator_state:" + product_entry_url + selected_option_path)
```

---

## 10. Database schema

### 10.1 `pages`

Add or migrate to these fields.

```sql
CREATE TABLE pages (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    page_key CHAR(64) NOT NULL UNIQUE,
    url TEXT NOT NULL,
    canonical_url TEXT NOT NULL,
    canonical_url_hash CHAR(64) NOT NULL,
    domain VARCHAR(255) NOT NULL,

    page_class VARCHAR(64) NULL,
    product BOOLEAN NOT NULL DEFAULT FALSE,
    has_product_catalogue BOOLEAN NOT NULL DEFAULT FALSE,
    product_page_role VARCHAR(64) NULL,
    product_entry_url TEXT NULL,

    context_type VARCHAR(64) NULL,
    context_label TEXT NULL,

    title TEXT NULL,
    content_digest CHAR(64) NULL,

    screenshot_path TEXT NULL,
    markdown_path TEXT NULL,
    html_path TEXT NULL,
    txt_path TEXT NULL,
    network_manifest_path TEXT NULL,

    cache_policy VARCHAR(64) NULL,
    etag VARCHAR(512) NULL,
    last_modified VARCHAR(512) NULL,

    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,

    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,

    INDEX idx_pages_domain (domain),
    INDEX idx_pages_canonical_url_hash (canonical_url_hash),
    INDEX idx_pages_content_digest (content_digest),
    INDEX idx_pages_page_class (page_class),
    INDEX idx_pages_product_role (product_page_role)
);
```

### 10.2 `crawl_targets`

```sql
CREATE TABLE crawl_targets (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    target_key CHAR(64) NOT NULL UNIQUE,
    target_type VARCHAR(64) NOT NULL,

    url TEXT NULL,
    canonical_url TEXT NULL,
    parent_target_key CHAR(64) NULL,
    parent_page_key CHAR(64) NULL,

    action_text TEXT NULL,
    action_role VARCHAR(64) NULL,
    action_selector TEXT NULL,
    action_signature CHAR(64) NULL,

    page_class VARCHAR(64) NULL,
    product BOOLEAN NOT NULL DEFAULT FALSE,
    product_page_role VARCHAR(64) NULL,

    status VARCHAR(32) NOT NULL DEFAULT 'discovered',
    priority INT NOT NULL DEFAULT 100,
    depth INT NOT NULL DEFAULT 0,

    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,

    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,

    INDEX idx_targets_status (status),
    INDEX idx_targets_type (target_type),
    INDEX idx_targets_parent_page (parent_page_key),
    INDEX idx_targets_depth (depth)
);
```

Supported `target_type` values:

```text
url
javascript_navigation
modal
content_state
catalogue_context
catalogue_filter_state
product_reference
product_detail
product_configurator
api_response
download
```

Supported `status` values:

```text
discovered
queued
crawled
already_known
skipped_existing
skipped_cache
failed
ignored
no_op
referenced_only
```

### 10.3 `products`

```sql
CREATE TABLE products (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    product_key CHAR(64) NOT NULL UNIQUE,
    domain VARCHAR(255) NOT NULL,

    canonical_product_url TEXT NULL,
    product_name TEXT NULL,
    product_code TEXT NULL,
    product_family TEXT NULL,
    product_type TEXT NULL,

    product_page_role VARCHAR(64) NULL,
    identity_confidence DECIMAL(5,2) NULL,
    identity_source VARCHAR(64) NULL,

    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,

    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,

    INDEX idx_products_domain (domain),
    INDEX idx_products_name (product_name(255)),
    INDEX idx_products_code (product_code(255))
);
```

### 10.4 `page_product_references`

```sql
CREATE TABLE page_product_references (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    reference_key CHAR(64) NOT NULL UNIQUE,
    page_key CHAR(64) NOT NULL,
    product_key CHAR(64) NOT NULL,

    reference_type VARCHAR(64) NOT NULL,
    context_type VARCHAR(64) NULL,
    context_label TEXT NULL,

    visible_product_name TEXT NULL,
    visible_product_code TEXT NULL,
    visible_product_family TEXT NULL,
    visible_card_text TEXT NULL,

    product_url TEXT NULL,
    canonical_product_url TEXT NULL,

    position_index INT NULL,
    source_selector TEXT NULL,

    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,

    UNIQUE KEY uq_page_product_context (page_key, product_key, context_type, context_label(255)),
    INDEX idx_ppr_page (page_key),
    INDEX idx_ppr_product (product_key)
);
```

### 10.5 `page_catalogue_filters`

```sql
CREATE TABLE page_catalogue_filters (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    page_key CHAR(64) NOT NULL,
    filter_key CHAR(64) NOT NULL UNIQUE,

    filter_name TEXT NOT NULL,
    filter_type VARCHAR(64) NULL,
    visible_value_count INT NULL,

    raw_text TEXT NULL,
    position_index INT NULL,

    INDEX idx_pcf_page (page_key)
);
```

### 10.6 `page_catalogue_filter_values`

```sql
CREATE TABLE page_catalogue_filter_values (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    filter_key CHAR(64) NOT NULL,
    value_key CHAR(64) NOT NULL UNIQUE,

    page_key CHAR(64) NOT NULL,
    filter_name TEXT NOT NULL,

    value_text TEXT NOT NULL,
    value_code TEXT NULL,
    result_count INT NULL,

    selected BOOLEAN NULL,
    disabled BOOLEAN NULL,

    position_index INT NULL,

    INDEX idx_pcfv_filter (filter_key),
    INDEX idx_pcfv_page (page_key)
);
```

### 10.7 `product_variants`

```sql
CREATE TABLE product_variants (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    variant_key CHAR(64) NOT NULL UNIQUE,
    product_key CHAR(64) NOT NULL,

    article_number TEXT NULL,
    article_number_raw TEXT NULL,

    variant_name TEXT NULL,
    type_description TEXT NULL,
    dimming TEXT NULL,
    status TEXT NULL,

    luminous_flux TEXT NULL,
    power TEXT NULL,
    efficacy TEXT NULL,
    color_temperature TEXT NULL,
    cri TEXT NULL,
    ugr TEXT NULL,
    ip_rate TEXT NULL,

    source_url TEXT NULL,
    source_section TEXT NULL,

    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,

    INDEX idx_variants_product (product_key),
    INDEX idx_variants_article (article_number(255))
);
```

### 10.8 `product_option_groups`

```sql
CREATE TABLE product_option_groups (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    group_key CHAR(64) NOT NULL UNIQUE,
    product_key CHAR(64) NOT NULL,

    step_name TEXT NULL,
    tab_name TEXT NULL,
    group_name TEXT NOT NULL,

    group_type VARCHAR(64) NULL,
    required BOOLEAN NULL,
    multi_select BOOLEAN NULL,

    source_section TEXT NULL,
    position_index INT NULL,

    INDEX idx_option_groups_product (product_key)
);
```

### 10.9 `product_option_values`

```sql
CREATE TABLE product_option_values (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    value_key CHAR(64) NOT NULL UNIQUE,
    product_key CHAR(64) NOT NULL,
    group_key CHAR(64) NOT NULL,

    value_text TEXT NOT NULL,
    value_code TEXT NULL,
    value_unit TEXT NULL,
    value_image_url TEXT NULL,

    selected_by_default BOOLEAN NULL,
    disabled BOOLEAN NULL,

    raw_label TEXT NULL,
    raw_html TEXT NULL,

    position_index INT NULL,

    INDEX idx_option_values_product (product_key),
    INDEX idx_option_values_group (group_key)
);
```

### 10.10 `product_downloads`

```sql
CREATE TABLE product_downloads (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    download_key CHAR(64) NOT NULL UNIQUE,
    product_key CHAR(64) NOT NULL,
    variant_key CHAR(64) NULL,

    download_label TEXT NULL,
    download_type VARCHAR(64) NULL,
    url TEXT NOT NULL,
    canonical_url TEXT NULL,

    source_section TEXT NULL,
    first_seen_run_id CHAR(36) NOT NULL,
    last_seen_run_id CHAR(36) NOT NULL,

    INDEX idx_downloads_product (product_key),
    INDEX idx_downloads_variant (variant_key)
);
```

### 10.11 `ai_page_analysis`

```sql
CREATE TABLE ai_page_analysis (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,

    analysis_key CHAR(64) NOT NULL UNIQUE,
    page_key CHAR(64) NOT NULL,
    model VARCHAR(128) NOT NULL,

    input_digest CHAR(64) NOT NULL,
    output_json JSON NOT NULL,
    confidence DECIMAL(5,2) NULL,

    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,

    INDEX idx_ai_page (page_key),
    INDEX idx_ai_input_digest (input_digest)
);
```

This prevents calling Gemini again for identical page content.

---

## 11. Page classes and product roles

### 11.1 Page classes

Supported values:

```text
normal_page
content_landing_page
catalogue_context_page
product_listing_page
product_family_page
product_detail_page
product_configurator_entry
product_configurator_state
product_modal
navigation_overlay
unknown_interactive
```

### 11.2 Product page roles

Supported values:

```text
none
catalogue_context
product_listing
product_family_page
product_detail_page
product_configurator_entry
product_configurator_state
product_codes_section
```

### 11.3 Examples

XAL Sports hall:

```text
page_class = catalogue_context_page
product = false
has_product_catalogue = true
product_page_role = catalogue_context
context_type = application
context_label = Sports hall
```

XAL Recessed spotlights:

```text
page_class = catalogue_context_page
product = false
has_product_catalogue = true
product_page_role = catalogue_context
context_type = product_category
context_label = Recessed spotlights
```

XAL UNICO Q9 basic recessed:

```text
page_class = product_family_page
product = true
has_product_catalogue = false
product_page_role = product_family_page
```

PROLICHT HYPRO configurator:

```text
page_class = product_configurator_entry
product = true
has_product_catalogue = false
product_page_role = product_configurator_entry
product_entry_url = current URL
```

---

## 12. Rule-based classifier first, Gemini second

Do not call Gemini for every page by default.

Use this order:

```text
1. Run deterministic/rule-based classifier.
2. If confidence >= 0.85, use rule result.
3. If confidence < 0.85 and gemini_analysis=true, call Gemini.
4. Store Gemini result in ai_page_analysis.
5. If Gemini fails, keep rule result with low confidence.
```

### 12.1 Rule-based signals for configurators

Keywords:

```text
customize this product
configure
configurator
technical data
design
light
complete
choose your color
size
sensor
plug
diffuser
light emission
colourtemperature
color temperature
product sketch
article number
```

German equivalents:

```text
konfigurieren
konfigurator
technische daten
farbe
groesse
sensor
stecker
diffusor
lichtaustritt
artikelnr
auswaehlen
```

### 12.2 Rule-based signals for catalogue pages

Keywords:

```text
product categories
application
product families
search products in this category
detail filter
efficiency
length
colour
colour temperature
control
luminous flux
power
```

### 12.3 Rule-based signals for product detail/family pages

Keywords:

```text
quickinfo
products and codes
downloads and links
aggregated downloads
description
simple drawing
light distribution
article number
mounting set
colour
```

---

## 13. Gemini content analysis

### 13.1 Gemini client wrapper

Create a module in the REST API, for example:

```text
app/services/gemini_analysis.py
```

Use environment variables:

```python
GEMINI_API_KEY = os.getenv("GEMINI_API_KEY", "")
GEMINI_MODEL = os.getenv("GEMINI_MODEL", "gemini-flash-lite-latest")
```

Use the official SDK if available:

```python
from google import genai

client = genai.Client(api_key=GEMINI_API_KEY)
```

### 13.2 Gemini input package

Build a compact input object:

```json
{
  "url": "https://www.xal.com/en/products/p/...",
  "domain": "www.xal.com",
  "title": "UNICO Q9 basic recessed",
  "visible_text": "...",
  "links": [
    {"text": "Products and codes", "href": "..."},
    {"text": "Discover family", "href": "..."}
  ],
  "buttons": [
    "UNICO pro Configurator",
    "Products and codes",
    "Show more"
  ],
  "headings": [
    "Quickinfo",
    "Products and codes",
    "Downloads & links"
  ],
  "network_response_summaries": [],
  "screenshot_available": true
}
```

Do not send full raw HTML unless needed. Prefer visible text, semantic DOM snippets, link lists, headings, and small selected HTML fragments.

### 13.3 Gemini page classification prompt

Use a strict JSON-only prompt.

Expected output:

```json
{
  "page_class": "product_family_page",
  "product": true,
  "has_product_catalogue": false,
  "product_page_role": "product_family_page",
  "context_type": null,
  "context_label": null,
  "product_name": "UNICO Q9 basic recessed",
  "product_family": "UNICO Q9",
  "detected_sections": [
    "Quickinfo",
    "Products and codes",
    "Downloads & links",
    "Description"
  ],
  "recommended_extractor": "xal_product_family_dom",
  "confidence": 0.93,
  "reason": "The page shows a product title, quick information, options, product code table, and downloads."
}
```

### 13.4 Gemini extraction prompt for catalogue context pages

Expected output:

```json
{
  "page_class": "catalogue_context_page",
  "context_type": "application",
  "context_label": "Sports hall",
  "has_product_catalogue": true,
  "filter_names": [
    "Efficiency",
    "Length",
    "Colour",
    "Colour temperature",
    "Control",
    "Luminous flux",
    "Power",
    "Melanopic ratio"
  ],
  "visible_products": [
    {
      "name": "PROFILES 60",
      "type": "luminaire & system",
      "confidence": 0.88
    }
  ],
  "confidence": 0.91
}
```

### 13.5 Gemini extraction prompt for configurator pages

Expected output:

```json
{
  "page_class": "product_configurator_entry",
  "product": true,
  "product_name": "HYPRO F 160 Free Standing",
  "current_article_number": "510-0510 03 03 03 03 30 02 01 00",
  "steps": [
    {
      "step_name": "01 Design",
      "groups": [
        {
          "group_name": "Colors",
          "values": ["Sparkling Silver", "Black Velvet", "Crystal White"]
        }
      ]
    },
    {
      "step_name": "02 Technical Data",
      "groups": [
        {"group_name": "Size", "values": ["1510mm", "1655mm", "2200mm", "2220mm"]},
        {"group_name": "Sensor", "values": ["No Sensor", "With smart sensor control"]},
        {"group_name": "Plug", "values": ["Schuko - Plug Europe", "British Plug", "Swiss Plug"]}
      ]
    }
  ],
  "confidence": 0.9
}
```

---

## 14. JavaScript click handling

Use three click profiles.

### 14.1 `navigation_click_profile`

Used in `page_crawl`.

Click:

```text
menus
product cards
JavaScript links
modals
tabs
accordions
read more buttons
configure now
all about
most configured
```

Avoid:

```text
cart
basket
checkout
login
register
account
share
print
wishlist
compare
language switch
cookie settings
newsletter
social media
video play
map
```

### 14.2 `catalogue_click_profile`

Used on XAL-style category/product-list pages.

Click:

```text
product cards
Show more
filter dropdowns to read values
detail filter if needed
```

Avoid by default:

```text
all filter combinations
cart/request buttons
compare/wishlist
login-required actions
```

### 14.3 `configurator_click_profile`

Used on PROLICHT-style configurators.

Click:

```text
configurator steps
option tabs
visible option values
```

Traversal depends on:

```text
configurator_strategy
```

Supported strategies:

```text
selected_state_only
taxonomy_only
sample_dependencies
valid_variants_from_network
exhaustive_combinations
```

Default:

```text
page_crawl: selected_state_only
product_discovery: taxonomy_only + selected_state_only
deep analysis: valid_variants_from_network
```

Do not use `exhaustive_combinations` unless explicitly requested.

---

## 15. JavaScript click outcome detection

After each click, detect what happened.

Possible outcomes:

```text
url_changed
modal_opened
content_changed
api_response_received
download_triggered
no_op
blocked
```

### 15.1 Detection logic

Before click, capture:

```text
current URL
visible text digest
DOM digest of main content
modal count
network request counter
download counter
```

After click, wait for one of:

```text
URL changed
new modal/overlay appeared
visible text digest changed
network response received
new download appeared
timeout
```

Then create the appropriate target:

```text
url_changed -> enqueue canonical URL
modal_opened -> save virtual modal page
content_changed -> save virtual content state
api_response_received -> store API response artifact if useful
download_triggered -> store download target
no_op -> mark action no_op
```

---

## 16. Network/XHR capture

Add XHR/fetch capture in the browser layer.

Capture:

```text
request URL
request method
status code
content type
response body if JSON and under size limit
initiating page_key
initiating action_signature
```

Store useful responses under artifacts and reference them in DB.

Do not store huge binary responses as JSON.

### 16.1 Response classification

Classify API responses as:

```text
product_list
product_detail
variant_list
filter_values
download_list
configurator_state
unknown_json
```

Prefer extraction from structured API responses when available:

```text
structured API response > DOM table > visible text fallback
```

---

## 17. XAL handling

### 17.1 XAL catalogue context pages

Examples:

```text
https://www.xal.com/en/products/c/sports-hall~383825
https://www.xal.com/en/products/c/recessed-spotlights~382913
```

These pages are not products.

They are discovery contexts.

Store:

```text
page_class = catalogue_context_page
product = false
has_product_catalogue = true
product_page_role = catalogue_context
context_type = application or product_category
context_label = visible page heading
```

Extract:

```text
page content/header text
catalogue filters
visible product cards
product links
page-product references
```

Do not create duplicate products when the same product appears under multiple contexts.

Correct model:

```text
products:
    UNICO Q9

page_product_references:
    Sports hall -> UNICO Q9
    Recessed spotlights -> UNICO Q9
    Downlights -> UNICO Q9
```

### 17.2 XAL product/family pages

Example:

```text
UNICO Q9 basic recessed
```

Extract:

```text
title
breadcrumb
quick info
colour options
mounting set options
light distribution options
simple drawing options
downloads and links
aggregated downloads
description
products and codes table
```

### 17.3 XAL Products and codes table

In `product_discovery` mode:

```text
1. Scroll to Products and codes.
2. Extract current visible rows.
3. Click Show more.
4. Wait for row count to increase.
5. Repeat until Show more disappears or row count no longer changes.
6. Stop at max_show_more_clicks or max_product_variant_rows.
7. Deduplicate variants by article number or row digest.
```

Pseudo-code:

```python
previous_count = 0
clicks = 0
while clicks < max_show_more_clicks:
    rows = extract_variant_rows()
    current_count = len(rows)
    if current_count <= previous_count and clicks > 0:
        break
    previous_count = current_count

    button = find_visible_button_text("Show more")
    if not button:
        break

    click(button)
    wait_until(lambda: len(extract_variant_rows()) > current_count, timeout=10)
    clicks += 1
```

Variant row fields:

```text
status
article_number_raw
type_description
dimming
download icons/links
```

---

## 18. PROLICHT handling

### 18.1 PROLICHT product filter page

Product cards can open JavaScript overlays/modals.

When clicking a product card like GLORIOUS:

```text
- detect modal overlay
- save modal as virtual page
- extract text/spec table
- extract CTA buttons
- enqueue Configure now
- enqueue Most configured
- enqueue All about product/family
```

### 18.2 PROLICHT product configurator pages

Product configurators change page state in the background.

Changing options can update:

```text
article number
product sketch
lumen/watt values
downloads
valid next options
selected color/material
```

In `page_crawl` mode:

```text
- save the entry page
- capture screenshot
- mark product=true
- product_page_role=product_configurator_entry
- do not traverse all options
```

In `product_discovery` mode:

```text
1. Open configurator entry URL.
2. Detect product name and current article number.
3. Detect selected options.
4. Detect steps and tabs.
5. Extract option taxonomy.
6. Capture selected state as product variant/configurator state.
7. Capture downloads for selected state.
8. Optionally sample state dependencies.
9. Avoid duplicate states by state_key.
```

---

## 19. Domain adapters

Build generic logic first. Add optional domain adapters second.

Adapter config example:

```json
{
  "domain": "xal.com",
  "product_url_patterns": [
    "/products/p/"
  ],
  "catalogue_url_patterns": [
    "/products/c/"
  ],
  "product_page_keywords": [
    "Quickinfo",
    "Products and codes",
    "Downloads & links"
  ],
  "show_more_texts": [
    "Show more"
  ]
}
```

```json
{
  "domain": "prolicht.at",
  "product_url_patterns": [
    "/product-configurator/product-families/"
  ],
  "catalogue_url_patterns": [
    "/product-configurator/product-filter/"
  ],
  "configurator_keywords": [
    "Customize this product",
    "Technical Data",
    "Light Emission"
  ],
  "ignore_action_texts": [
    "Back to product filter page",
    "Complete"
  ]
}
```

The crawler must still work without an adapter.

---

## 20. REST API endpoints

### 20.1 Crawl one page

```http
POST /crawl/single
```

Payload:

```json
{
  "url": "https://www.xal.com/en/products/p/...",
  "mode": "page_crawl",
  "cache_policy": "cache_first",
  "javascript_navigation": true,
  "open_modals": true,
  "capture_xhr": true,
  "gemini_analysis": true,
  "include_page_data": true
}
```

### 20.2 Crawl many URLs

```http
POST /crawl/urls
```

Payload:

```json
{
  "urls": ["https://www.xal.com/en/products/c/sports-hall~383825"],
  "mode": "page_crawl",
  "cache_policy": "cache_first",
  "new_pages_only": false,
  "threads": 4
}
```

### 20.3 Start product discovery

```http
POST /discovery/products/start
```

Payload:

```json
{
  "seed_urls": [
    "https://www.xal.com/en/products/c/recessed-spotlights~382913"
  ],
  "mode": "product_discovery",
  "cache_policy": "cache_first",
  "product_discovery": {
    "enabled": true,
    "extract_catalogue_contexts": true,
    "extract_product_cards": true,
    "extract_product_pages": true,
    "extract_variant_tables": true,
    "click_show_more": true,
    "extract_downloads": true,
    "filter_strategy": "visible_values_only",
    "configurator_strategy": "taxonomy_only"
  }
}
```

Return:

```json
{
  "job_id": "...",
  "status": "queued"
}
```

### 20.4 Get product discovery job status

```http
GET /discovery/products/jobs/{job_id}
```

### 20.5 Get product discovery results

```http
GET /discovery/products/jobs/{job_id}/results
```

### 20.6 Fetch cached page artifact

```http
POST /artifacts/page
```

Payload:

```json
{
  "url": "https://www.xal.com/en/products/p/...",
  "include_markdown": true,
  "include_links": true,
  "include_images": true,
  "include_product_data": true,
  "max_inline_chars": 12000
}
```

### 20.7 Search local product database

```http
POST /products/search
```

Payload:

```json
{
  "domain": "xal.com",
  "query": "UNICO Q9",
  "include_contexts": true,
  "include_variants": true,
  "include_downloads": true
}
```

---

## 21. MCP tool changes

### 21.1 Keep existing tools but extend payloads

Existing tools can remain:

```text
crawl_page
research
```

Update `crawl_page` so it passes through:

```text
mode
cache_policy
new_pages_only
changed_pages_only
gemini_analysis
```

### 21.2 Add `product_discovery` MCP tool

```python
@mcp.tool(annotations={"readOnlyHint": True})
async def product_discovery(
    site: str,
    seed_urls: list[str],
    max_products: int = 100,
    recrawl: bool = False,
    ctx: Context | None = None,
) -> dict:
    ...
```

This tool should call:

```text
POST /discovery/products/start
GET /discovery/products/jobs/{job_id}
GET /discovery/products/jobs/{job_id}/results
```

It should not crawl directly.

### 21.3 Add `get_product` MCP tool

```python
@mcp.tool(annotations={"readOnlyHint": True})
async def get_product(
    query: str,
    site: str | None = None,
    include_contexts: bool = True,
    include_variants: bool = True,
    include_downloads: bool = True,
) -> dict:
    ...
```

This calls:

```text
POST /products/search
```

### 21.4 Replace DeepSeek relevance filtering with Gemini

Current MCP filtering can be changed from `_deepseek_ask()` to `_gemini_ask()`.

Expected behavior remains the same:

```text
Input: question, URL, title, snippet
Output: compact JSON with relevant=true/false and reason
```

New environment:

```text
GEMINI_API_KEY
GEMINI_MODEL=gemini-flash-lite-latest
```

---

## 22. Implementation phases for junior developer

### Phase 1: Data model and request options

Tasks:

1. Add new request fields:
   - `mode`
   - `cache_policy`
   - `new_pages_only`
   - `changed_pages_only`
   - `gemini_analysis`
   - `product_discovery`
2. Add DB migrations:
   - `crawl_targets`
   - new `pages` columns
   - `products`
   - `page_product_references`
   - `page_catalogue_filters`
   - `page_catalogue_filter_values`
   - `product_variants`
   - `product_option_groups`
   - `product_option_values`
   - `product_downloads`
   - `ai_page_analysis`
3. Add unit tests for target key generation.
4. Add unit tests for URL canonicalization.

Done when:

```text
- requests accept the new fields
- migrations run cleanly
- same URL with tracking params maps to same canonical URL
- semantic query params are preserved
```

### Phase 2: Cache-first and target queue

Tasks:

1. Move cache lookup before browser opening.
2. Implement `cache_first` behavior.
3. Implement `new_pages_only` behavior.
4. Implement `crawl_targets` insert/upsert logic.
5. Add statuses: `already_known`, `skipped_cache`, `queued`, `crawled`.

Done when:

```text
- already crawled pages are returned from cache without browser navigation
- new_pages_only skips known targets
- duplicate targets are not queued twice
```

### Phase 3: Page classifier

Tasks:

1. Add rule-based classifier.
2. Add page classes and product roles.
3. Store classification on `pages`.
4. Add tests with saved text snippets from:
   - XAL Sports hall
   - XAL Recessed spotlights
   - XAL UNICO Q9
   - PROLICHT HYPRO configurator
   - PROLICHT GLORIOUS modal

Done when:

```text
- catalogue pages are not marked as products
- product family/detail pages are marked as products
- configurator pages are marked as product_configurator_entry
```

### Phase 4: Gemini analysis service

Tasks:

1. Add `google-genai` dependency.
2. Create `gemini_analysis.py`.
3. Implement `analyze_page()`.
4. Implement caching in `ai_page_analysis` by input digest.
5. Use model `gemini-flash-lite-latest`.
6. Add strict JSON parsing and fallback handling.

Done when:

```text
- ambiguous pages can be classified by Gemini
- repeated identical content does not call Gemini again
- failed Gemini calls do not break crawling
```

### Phase 5: JavaScript navigation

Tasks:

1. Implement clickable candidate extraction.
2. Implement action signatures.
3. Implement click outcome detection.
4. Save virtual pages for modals and content states.
5. Enqueue URL changes.
6. Add action blacklist.

Done when:

```text
- JavaScript links are clicked
- product modals are captured as virtual pages
- CTA links from modals are discovered
- duplicate JS states are not recrawled
```

### Phase 6: Catalogue extraction

Tasks:

1. Detect catalogue area.
2. Extract visible filter names.
3. Open filter dropdowns to read values.
4. Store filters and values.
5. Extract product cards.
6. Create products or weak products.
7. Create page_product_references.
8. Deduplicate products across pages.

Done when:

```text
- XAL Sports hall creates one catalogue context page
- XAL Recessed spotlights creates one catalogue context page
- same product appearing on both pages creates one product and two references
```

### Phase 7: XAL product page extractor

Tasks:

1. Detect XAL product/family page.
2. Extract title and quick info.
3. Extract visible options:
   - Colour
   - Mounting set
   - Light distribution
   - Simple drawing
4. Scroll to Products and codes.
5. Extract rows.
6. Click Show more until complete or limits reached.
7. Store product variants.
8. Extract downloads and aggregated downloads.

Done when:

```text
- UNICO Q9 page produces a product record
- variant rows are stored once
- Show more loads additional rows
- downloads are linked to product or variants
```

### Phase 8: PROLICHT configurator extractor

Tasks:

1. Detect PROLICHT configurator page.
2. Extract current article number.
3. Extract selected state.
4. Extract steps/tabs.
5. Extract option groups and option values.
6. Store selected state as variant/configurator state.
7. Capture downloads for selected state.
8. Implement `taxonomy_only` strategy.

Done when:

```text
- HYPRO configurator stores product entry
- steps Design / Technical Data / Light are found
- option groups Colors, Size, Sensor, Plug, etc. are stored
- selected article number is stored
```

### Phase 9: Network/XHR extraction

Tasks:

1. Capture XHR/fetch responses.
2. Store useful JSON responses.
3. Classify response types.
4. Use API response data before DOM fallback.
5. Link responses to page/action/state.

Done when:

```text
- Show more API responses can be inspected
- configurator state API responses can be inspected
- JSON responses can enrich product/variant extraction
```

### Phase 10: MCP update

Tasks:

1. Add Gemini client to MCP.
2. Replace/supplement DeepSeek relevance filtering.
3. Add pass-through options to `crawl_page` and `research`.
4. Add `product_discovery` tool.
5. Add `get_product` tool.
6. Keep MCP free of crawler/browser imports.

Done when:

```text
- MCP can trigger product discovery through REST API
- MCP can fetch product results through REST API
- MCP can still do question-based site research
- MCP does not import crawler internals
```

---

## 23. Acceptance tests

### 23.1 Cache test

Input:

```text
Crawl same URL twice with cache_policy=cache_first.
```

Expected:

```text
First call opens browser.
Second call returns cached artifact without browser navigation.
```

### 23.2 New pages only test

Input:

```text
Run full crawl with new_pages_only=true twice.
```

Expected:

```text
First run crawls new targets.
Second run marks targets already_known and crawls nothing new.
```

### 23.3 XAL catalogue dedupe test

Input:

```text
Crawl XAL Sports hall and XAL Recessed spotlights in product_discovery mode.
```

Expected:

```text
Both pages are catalogue_context_page.
Products referenced by both pages exist once in products.
Two page_product_references exist for shared products.
```

### 23.4 XAL Show more test

Input:

```text
Open UNICO Q9 product page in product_discovery mode.
```

Expected:

```text
Products and codes rows are extracted.
Show more is clicked until complete or limit reached.
Rows are deduplicated by article number or row digest.
```

### 23.5 PROLICHT configurator test

Input:

```text
Open HYPRO configurator in product_discovery mode.
```

Expected:

```text
Product is marked true.
Page role is product_configurator_entry.
Article number is extracted.
Option taxonomy is extracted.
Downloads for selected state are captured if visible/available.
```

### 23.6 PROLICHT modal test

Input:

```text
Open PROLICHT product filter page and click GLORIOUS product card.
```

Expected:

```text
Modal is saved as virtual page.
Configure now, Most configured, and All about links/actions are discovered.
Duplicate modal captures are skipped on repeated runs.
```

### 23.7 MCP separation test

Input:

```text
Run MCP research and product_discovery tools.
```

Expected:

```text
MCP calls REST API only.
MCP does not import browser/crawler modules.
MCP returns compact JSON.
```

---

## 24. Suggested file/module layout

REST API:

```text
app/
  api/
    crawl_routes.py
    artifact_routes.py
    product_discovery_routes.py
    product_routes.py
  crawler/
    browser_runner.py
    cache_service.py
    canonical_url.py
    click_profiles.py
    click_outcome.py
    js_discovery.py
    network_capture.py
    page_classifier.py
  product_discovery/
    catalogue_extractor.py
    product_card_extractor.py
    product_identity.py
    product_page_extractor.py
    variant_table_extractor.py
    configurator_extractor.py
    downloads_extractor.py
    adapters/
      generic.py
      xal.py
      prolicht.py
  services/
    gemini_analysis.py
    artifact_store.py
    database.py
  migrations/
    001_discovery_schema.sql
```

MCP:

```text
mcp_research.py
mcp_gemini.py
mcp_api_client.py
```

---

## 25. Junior developer checklist

Start with this exact order:

```text
1. Add request model fields.
2. Add URL canonicalization tests.
3. Add DB migrations.
4. Add cache-first before browser navigation.
5. Add crawl_targets table writes.
6. Add page classifier.
7. Add Gemini analysis service.
8. Add JavaScript click outcome detection.
9. Add catalogue extraction.
10. Add product identity resolver.
11. Add XAL product extractor.
12. Add PROLICHT configurator extractor.
13. Add XHR capture.
14. Add MCP tool updates.
15. Run acceptance tests.
```

Do not start with exhaustive configurator traversal.

Do not start with filter combination crawling.

Do not hardcode everything for PROLICHT or XAL. Implement generic behavior first, then add adapters.

---

## 26. Most important design rule

Product discovery is not page crawling with more clicks.

It is a separate extraction mode with:

```text
- separate tables
- separate deduplication keys
- separate click profiles
- separate limits
- separate acceptance tests
```

Normal crawling should stay fast and safe.

Product discovery can be slower, deeper, and more structured, but only when explicitly requested.
