# Crawler Discovery, Cache, JavaScript Link, and New-Pages-Only Implementation Plan

Prepared for: Junior developer

Project context: Python REST API crawler using `server.py`, `crawler.py`, `crawl_db.py`, and MariaDB persistence.

## 1. Goal

Extend the crawler so it can safely crawl product areas and complete homepages across many domains, including sites such as PROLICHT, XAL, Lightnet, and similar modern JavaScript-heavy websites.

The crawler must:

1. Avoid duplicate crawls within the same run.
2. Avoid duplicate crawls across historical runs.
3. Avoid refetching pages that are already cached when the selected crawl policy allows it.
4. Preserve meaningful query parameters instead of stripping all query strings.
5. Discover normal HTML links.
6. Discover and click JavaScript links, clickable cards, buttons, overlays, and modal CTAs.
7. Store modal/interaction content as searchable virtual pages when no real URL exists.
8. Provide an option to crawl only pages that did not exist in the crawler history before.
9. Keep the implementation generic and domain-configurable, not hardcoded for PROLICHT only.

## 2. Current behavior that must be changed

### 2.1 Query parameters are currently stripped too aggressively

Current code uses `_strip_fragment_and_query(url)` and calls it in several important places, including single-page crawl, sitemap crawl, canonical extraction, and link discovery.

Problem:

```text
https://www.prolicht.at/en/product-configurator/product-families/lullaby/cloud-model-1/?tab=color&MountingType=pendel
```

currently becomes:

```text
https://www.prolicht.at/en/product-configurator/product-families/lullaby/cloud-model-1/
```

That loses the state of configurator pages and can overwrite or skip real crawl states.

### 2.2 JavaScript links are currently ignored

Current `_normalized_crawl_url()` ignores these href values:

```text
javascript:
#
mailto:
tel:
sms:
data:
blob:
```

This is correct for static URL extraction, but it means the crawler never clicks JavaScript-driven product cards, buttons, tabs, overlays, or modal CTAs.

### 2.3 Cache is checked too late

Current `_fetch_and_save()` fetches the live rendered page first, then checks if the digest is unchanged.

That saves processing time, but it does not prevent refetching the page. For true cache-first behavior, the crawler must check the persistent cache before `browser.get()` or `_prepare_browser()` when possible.

### 2.4 Multi/sitemap crawl currently disables keep-crawl

`server.py` currently hardcodes this in `multi_request_options()`:

```python
"keep_crawl": False,
```

This means full crawl jobs refetch known pages unless the code is changed.

### 2.5 In-run and historical dedupe are not strict enough

The sitemap queue currently dedupes mostly with an in-memory `seen` set based on `child.rstrip("/")`.

This is not enough because these URLs may represent the same page:

```text
https://example.com/products?b=2&a=1
https://example.com/products?a=1&b=2
https://example.com/products/?a=1&b=2&utm_source=test
```

The crawler needs one shared canonicalization and hashing strategy for:

- URL extraction
- queue insertion
- DB cache lookup
- artifact filename creation
- page link storage
- new-pages-only decisions

## 3. Core concepts to implement

### 3.1 Raw URL

The URL exactly as discovered from HTML, JavaScript, DOM attributes, browser location, or API.

Example:

```text
/products/?utm_source=x&MountingType=pendel&tab=color
```

### 3.2 Canonical URL

The normalized URL used for dedupe, cache, and persistence.

Rules:

1. Lowercase only scheme and hostname.
2. Preserve path case unless a site profile explicitly allows lowercasing paths.
3. Remove fragments by default.
4. Keep meaningful query parameters.
5. Drop tracking/noise query parameters.
6. Sort query parameters by key and value.
7. Normalize duplicate slashes in paths only if safe.
8. Remove default ports `:80` and `:443`.
9. Normalize trailing slash using a consistent rule.

### 3.3 URL hash

Hash the canonical URL, not the raw URL.

Use SHA-256, not MD5, for new columns:

```python
canonical_url_hash = sha256(canonical_url.encode("utf-8")).hexdigest()
```

Keep the old `url_hash` temporarily for backward compatibility.

### 3.4 Content digest

Hash the cleaned rendered content, not the raw HTML.

Use this to detect changed content after fetching:

```text
content_digest = sha256(cleaned_visible_html_or_markdown_text)
```

### 3.5 Discovery key

A key used before a page is crawled.

For normal URLs:

```text
discovery_key = "url:" + canonical_url_hash
```

For JavaScript/modal actions:

```text
discovery_key = "action:" + sha256(parent_canonical_url + "|" + action_signature)
```

### 3.6 Page types

Add page types so the crawler can store more than normal URLs:

```text
url          - normal browser URL
modal        - overlay/modal content captured after a click
interaction  - DOM state created by JavaScript click without URL change
api          - API response transformed into searchable text
asset        - optional downloaded document or media artifact
```

## 4. New API options

Add these fields to `CrawlSingleRequest`, `CrawlUrlsRequest`, and `CrawlSitemapRequest` where applicable.

```python
cache_policy: str = "cache_first"
```

Allowed values:

```text
cache_first  - return cached artifacts if fresh; do not open browser
revalidate   - use lightweight validators when available; fetch only if changed
refresh      - always fetch page, but skip expensive reprocessing if digest unchanged
force        - always fetch and always rewrite artifacts
```

```python
crawl_scope: str = "all"
```

Allowed values:

```text
all          - crawl known and new pages according to cache_policy
new_only     - only crawl targets never previously crawled or reserved
```

```python
discovery_mode: str = "html_and_interactions"
```

Allowed values:

```text
html_only              - only extract normal HTML links
html_and_interactions  - extract HTML links and click JavaScript/clickable elements
interactions_only      - only run interaction discovery on the seed pages
```

```python
refresh_discovery_pages: bool = True
```

Meaning:

- In `crawl_scope="new_only"`, the crawler still needs to refresh seed/listing/sitemap pages to find new links.
- When this is true, seed pages and listing pages may be fetched for discovery, but already-known product/detail pages are not crawled again.

```python
crawl_js_links: bool = True
```

Controls whether JavaScript links and clickable DOM elements are clicked.

```python
max_interactions_per_page: int = 80
```

Hard safety cap for clicks per page.

```python
max_virtual_pages_per_page: int = 50
```

Hard safety cap for modal/interaction pages captured from one real page.

```python
query_mode: str = "semantic"
```

Allowed values:

```text
semantic     - keep meaningful query parameters and drop tracking parameters
full         - keep all query parameters except known unsafe/noise values
path_only    - old behavior, strips all query parameters
```

Default should be `semantic`.

## 5. Database changes

Create a migration file, for example:

```text
migrations/002_discovery_cache_interactions.sql
```

### 5.1 Extend `pages`

```sql
ALTER TABLE pages
  ADD COLUMN canonical_url VARCHAR(2048) NULL,
  ADD COLUMN canonical_url_hash CHAR(64) NULL,
  ADD COLUMN page_type VARCHAR(32) NOT NULL DEFAULT 'url',
  ADD COLUMN parent_canonical_url_hash CHAR(64) NULL,
  ADD COLUMN action_key CHAR(64) NULL,
  ADD COLUMN action_label VARCHAR(512) NULL,
  ADD COLUMN cache_policy VARCHAR(32) NULL,
  ADD COLUMN first_seen_at DOUBLE NULL,
  ADD COLUMN last_seen_at DOUBLE NULL,
  ADD COLUMN etag VARCHAR(512) NULL,
  ADD COLUMN last_modified VARCHAR(512) NULL;

CREATE UNIQUE INDEX idx_pages_canonical_type_action
  ON pages (canonical_url_hash, page_type, action_key);

CREATE INDEX idx_pages_content_digest
  ON pages (content_digest);

CREATE INDEX idx_pages_parent_hash
  ON pages (parent_canonical_url_hash);
```

Important:

- For normal URL pages, `action_key` can be NULL.
- For modal/interaction pages, `canonical_url` can be a virtual URL and `action_key` must be set.

### 5.2 Add `discovered_targets`

This table records every target before it is crawled.

```sql
CREATE TABLE IF NOT EXISTS discovered_targets (
  id INT AUTO_INCREMENT PRIMARY KEY,
  discovery_key CHAR(71) NOT NULL,
  target_type VARCHAR(32) NOT NULL DEFAULT 'url',
  raw_url VARCHAR(2048) NULL,
  canonical_url VARCHAR(2048) NULL,
  canonical_url_hash CHAR(64) NULL,
  parent_canonical_url_hash CHAR(64) NULL,
  source_url VARCHAR(2048) NULL,
  link_text VARCHAR(1024) NOT NULL DEFAULT '',
  action_signature VARCHAR(2048) NULL,
  first_seen_at DOUBLE NOT NULL,
  last_seen_at DOUBLE NOT NULL,
  seen_count INT NOT NULL DEFAULT 1,
  crawl_status VARCHAR(32) NOT NULL DEFAULT 'discovered',
  reserved_at DOUBLE NULL,
  crawled_at DOUBLE NULL,
  error_text TEXT NULL,
  UNIQUE INDEX idx_discovered_key (discovery_key),
  INDEX idx_discovered_canonical_hash (canonical_url_hash),
  INDEX idx_discovered_parent_hash (parent_canonical_url_hash),
  INDEX idx_discovered_status (crawl_status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

`crawl_status` values:

```text
discovered
reserved
crawled
skipped_known
skipped_policy
failed
```

### 5.3 Add `page_actions`

This table stores click candidates and results.

```sql
CREATE TABLE IF NOT EXISTS page_actions (
  id INT AUTO_INCREMENT PRIMARY KEY,
  parent_canonical_url_hash CHAR(64) NOT NULL,
  action_key CHAR(64) NOT NULL,
  action_type VARCHAR(32) NOT NULL,
  selector VARCHAR(2048) NOT NULL DEFAULT '',
  label VARCHAR(1024) NOT NULL DEFAULT '',
  href VARCHAR(2048) NULL,
  before_url VARCHAR(2048) NULL,
  after_url VARCHAR(2048) NULL,
  result_type VARCHAR(32) NOT NULL DEFAULT 'unknown',
  virtual_url VARCHAR(2048) NULL,
  content_digest CHAR(64) NULL,
  created_at DOUBLE NOT NULL,
  UNIQUE INDEX idx_page_actions_key (action_key),
  INDEX idx_page_actions_parent (parent_canonical_url_hash)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

`result_type` values:

```text
url_navigation
new_tab
modal
interaction
no_change
failed
```

## 6. New persistence functions in `crawl_db.py`

Add these functions.

### 6.1 `canonical_url_hash()`

```python
def canonical_url_hash(canonical_url: str) -> str:
    return hashlib.sha256(canonical_url.encode("utf-8")).hexdigest()
```

### 6.2 `get_page_by_canonical_hash()`

Used for cache-first decisions.

Return:

```python
CacheEntry | None
```

Look up by:

```sql
WHERE canonical_url_hash = %s AND page_type = %s
```

### 6.3 `is_known_target()`

Used for `crawl_scope="new_only"`.

A target is known if it exists in either:

- `pages` with matching `canonical_url_hash` and page type
- `discovered_targets` with matching `discovery_key`

### 6.4 `mark_discovered_target()`

Called immediately when a link/action is found, before queueing.

Use `INSERT ... ON DUPLICATE KEY UPDATE`:

- update `last_seen_at`
- increment `seen_count`
- keep original `first_seen_at`

### 6.5 `reserve_target_for_crawl()`

Called before a worker starts.

This prevents duplicate concurrent crawls.

Rules:

1. If target is already `reserved` by another worker and reservation is fresh, skip.
2. If target is already `crawled` and `crawl_scope="new_only"`, skip.
3. Otherwise set `crawl_status="reserved"` and `reserved_at=now`.

### 6.6 `mark_target_crawled()` and `mark_target_failed()`

Called after crawl success/failure.

## 7. URL normalization implementation

Create a new function in `crawler.py`:

```python
def canonicalize_url(url: str, *, base_url: str | None = None, query_mode: str = "semantic") -> str | None:
    ...
```

### 7.1 Drop query parameters by default

Drop these parameters in `semantic` mode:

```text
utm_*
fbclid
gclid
msclkid
dclid
yclid
_ga
_gid
mc_cid
mc_eid
ref
ref_src
source
campaign
rnd
random
cache
cachebuster
cb
timestamp
_ts
```

### 7.2 Keep meaningful query parameters

Keep these by default:

```text
lang
language
locale
tab
mountingtype
mounting_type
product
productid
product_id
family
productfamily
product_family
category
filter
filters
sort
page
q
search
query
variant
configuration
config
```

Important:

- Compare parameter names case-insensitively.
- Preserve original value case.
- Sort parameters after filtering.

### 7.3 Site profile override

Allow host-specific query rules from config.

Example file:

```text
site_profiles/prolicht.at.yaml
```

Example content:

```yaml
host_patterns:
  - prolicht.at
  - www.prolicht.at
query_keep:
  - tab
  - MountingType
  - ProductFamily
  - ProductCategory
query_drop:
  - utm_*
  - fbclid
path_prefixes:
  - /en/product-configurator/
```

Do not hardcode this for PROLICHT. The same profile system must work for XAL, Lightnet, and other websites.

## 8. Artifact filenames must include semantic query state

Update `_url_output_stem(url)` so semantic query parameters affect the filename.

Example:

```text
xal.com/products/detail?tab=downloads
xal.com/products/detail?tab=configuration
```

must not write to the same `.html`, `.txt`, `.md`, `.crawl.json`, or image manifest file.

Recommended filename rule:

```text
<domain>__<path_tokens>__q_<short_hash>
```

Only add `__q_<hash>` if the canonical URL has a query string.

Example:

```text
prolicht__en__product-configurator__product-families__lullaby__cloud-model-1__q_a13f29c6.md
```

## 9. Cache-first flow

Add a preflight function:

```python
async def maybe_return_cached_result(url: str, out_dir: Path, options: dict) -> dict | None:
    ...
```

Use it before browser startup where possible.

### 9.1 Single page flow

In `fetch_single_result()`:

1. Canonicalize URL.
2. Compute output dir.
3. Check DB cache by canonical hash.
4. Check artifact files exist on disk.
5. If `cache_policy="cache_first"` and cache is fresh, return cached result immediately.
6. Do not call `_prepare_browser()` in this case.
7. Otherwise fetch normally.

### 9.2 Sitemap flow

In `fetch_sitemap()`:

1. Canonicalize start URL.
2. If `crawl_scope="new_only"` and `refresh_discovery_pages=True`, fetch seed/listing pages even if known, but mark them as discovery refreshes.
3. For every discovered target, call `mark_discovered_target()`.
4. Before queueing, check in-run seen set and DB known target state.
5. If `crawl_scope="new_only"` and target is known, do not queue it.
6. If `cache_policy="cache_first"` and target has fresh artifacts, mark result as cache hit and do not open browser.
7. Queue only targets that need a live crawl.

### 9.3 URL-list flow

In `fetch_urls()`:

1. Canonicalize all input URLs before grouping by domain.
2. Deduplicate by canonical hash.
3. In `new_only` mode, remove targets that already exist in DB.
4. In `cache_first` mode, return cache hits without live crawling.
5. Only start browser sessions for domains that still have live targets.

## 10. JavaScript link and interaction discovery

Add this after rendered HTML capture and normal link extraction.

### 10.1 New function

```python
async def discover_interactive_targets(tab, page_url: str, out_dir: Path, options: dict) -> list[dict]:
    ...
```

Return a list of discovered targets:

```python
{
    "target_type": "url" | "modal" | "interaction",
    "raw_url": "...",
    "canonical_url": "...",
    "virtual_url": "...",
    "link_text": "...",
    "selector": "...",
    "action_signature": "...",
    "action_key": "...",
    "result_type": "url_navigation" | "new_tab" | "modal" | "interaction" | "no_change" | "failed",
}
```

### 10.2 Click candidate selectors

Use generic selectors first:

```text
a[href^="javascript:"]
a[href="#"]
[onclick]
button
[role="button"]
[role="link"]
[data-href]
[data-url]
[data-link]
[data-target]
.card
.product-card
.product-tile
.product-item
[class*="product"]
[class*="card"]
[class*="tile"]
```

Then filter candidates in JavaScript:

- visible only
- enabled only
- non-zero bounding box
- not inside cookie banner after consent is handled
- not duplicate by text + href + bounding box + selector
- skip external social share buttons
- skip cart/checkout/login/account unless explicitly allowed by site profile
- skip file downloads unless asset crawling is enabled

### 10.3 Action signature

Build a stable signature:

```text
parent_canonical_url
selector
visible_text
href
onclick hash
data-href/data-url
aria-label/title
rounded bounding box position
```

Hash this into `action_key`.

### 10.4 Click execution

For each candidate:

1. Record `before_url`.
2. Record a lightweight DOM digest before clicking.
3. Click the element.
4. Wait 0.5 to 2.0 seconds, configurable.
5. Detect outcome.
6. Store result.
7. Restore page state before the next click.

### 10.5 Outcome detection

After clicking, check in this order:

1. New browser tab opened.
2. `location.href` changed.
3. Modal or overlay appeared.
4. DOM digest changed.
5. No change.

### 10.6 If URL changed

If the click navigated to a real URL:

1. Canonicalize the new URL.
2. Add it as a discovered URL target.
3. Go back to the original page or reopen original page in the same tab.

### 10.7 If a modal appeared

Capture modal content as a virtual page.

Virtual URL format:

```text
<parent_canonical_url>#modal=<safe_label>&action=<short_action_key>
```

Save artifacts like a normal page:

```text
.html
.txt
.md
.crawl.json
```

Set:

```text
page_type = modal
parent_canonical_url_hash = parent hash
action_key = action key
action_label = visible button/card text
```

Then extract normal links and clickable buttons from inside the modal.

For modal CTAs like:

```text
Configure now
Most configured
All about GLORIOUS
```

handle each CTA as another discovered target.

### 10.8 If DOM changed without a URL or modal

Store it as `page_type="interaction"` only if the new text content is materially different.

Use a threshold:

```text
new_text_length > 300 characters
and content_digest changed
```

Virtual URL format:

```text
<parent_canonical_url>#interaction=<safe_label>&action=<short_action_key>
```

### 10.9 Restoring page state

Preferred restore order:

1. Close modal using visible close button or Escape.
2. If URL changed, use browser back.
3. If DOM state is still changed, reload original page.
4. If restore fails, close the tab and open a fresh tab for the parent page.

## 11. Persistent dedupe algorithm

Every target must pass through the same gate.

### 11.1 Gate function

Add:

```python
async def should_enqueue_target(target: dict, options: dict, run_state: dict) -> tuple[bool, str]:
    ...
```

Return:

```text
(True, "queued")
(False, "seen_in_run")
(False, "known_from_previous_run")
(False, "cache_hit")
(False, "out_of_scope")
(False, "limit_reached")
```

### 11.2 Gate order

1. Canonicalize URL or action key.
2. Check crawl scope and same-domain/scope rules.
3. Check static resource/file extension rules.
4. Check in-run `seen_discovery_keys` set.
5. Call `mark_discovered_target()`.
6. If `crawl_scope="new_only"`, check `is_known_target()`.
7. If cache hit and `cache_policy="cache_first"`, mark result as success with `cache_hit=True` and do not queue.
8. If limit reached, skip.
9. Add to queue.

### 11.3 Reservation before crawling

Before a worker opens a browser tab:

1. Call `reserve_target_for_crawl()`.
2. If reservation fails, skip.
3. Crawl.
4. Mark crawled or failed.

This prevents duplicates when multiple workers discover the same URL at the same time.

## 12. New-pages-only behavior

Add option:

```python
crawl_scope="new_only"
```

Expected behavior:

1. Seed pages may be refreshed for discovery when `refresh_discovery_pages=True`.
2. Known product/detail/content pages are not crawled again.
3. Newly discovered URLs are crawled.
4. Newly discovered JavaScript modal/interaction states are crawled/stored.
5. Existing pages are not overwritten.
6. Link maps and discovery metadata can still be updated.

### 12.1 What counts as existing?

A page exists if:

- its canonical URL hash exists in `pages`, or
- its discovery key exists in `discovered_targets` with status `crawled`, or
- its action key exists in `page_actions` and has a stored virtual page.

### 12.2 What if a known page changed?

In `new_only` mode, do not recrawl it.

Use a separate mode for changed content:

```text
crawl_scope="all"
cache_policy="revalidate"
```

or:

```text
crawl_scope="all"
cache_policy="refresh"
```

## 13. Site profiles for many domains

Do not build one-off code for PROLICHT.

Create a profile loader:

```text
site_profiles/default.yaml
site_profiles/prolicht.at.yaml
site_profiles/xal.com.yaml
site_profiles/lightnet.de.yaml
```

Each profile can define:

```yaml
host_patterns:
  - example.com
allowed_paths:
  - /products
  - /en/products
blocked_paths:
  - /checkout
  - /cart
  - /login
query_keep:
  - tab
  - variant
  - MountingType
query_drop:
  - utm_*
  - fbclid
click_include_selectors:
  - a[href^="javascript:"]
  - button
  - .product-card
click_exclude_selectors:
  - .cookie
  - .newsletter
  - .login
modal_selectors:
  - '[role="dialog"]'
  - '.modal'
  - '.overlay'
  - '.lightbox'
cta_selectors:
  - 'a'
  - 'button'
interaction_wait_ms: 800
max_interactions_per_page: 80
```

Default profile must work without any domain-specific config.

Domain-specific profiles should only improve precision.

## 14. Required code changes by file

### 14.1 `server.py`

Add request model fields:

- `cache_policy`
- `crawl_scope`
- `discovery_mode`
- `refresh_discovery_pages`
- `crawl_js_links`
- `max_interactions_per_page`
- `max_virtual_pages_per_page`
- `query_mode`

Pass them through in:

- `request_options()`
- `multi_request_options()`

Remove the hardcoded:

```python
"keep_crawl": False
```

Replace it with the new policy values.

For backward compatibility:

```python
if req.keep_crawl:
    cache_policy = "cache_first"
```

### 14.2 `crawler.py`

Add or change:

1. `canonicalize_url()`
2. `canonical_url_hash()`
3. update `_url_output_stem()` to include semantic query hash
4. replace `_strip_fragment_and_query()` calls where canonicalization is needed
5. update `_normalized_crawl_url()` so it does not automatically discard all query params
6. add preflight cache lookup before browser startup
7. add `discover_interactive_targets()`
8. add modal/interaction artifact saving
9. add `should_enqueue_target()`
10. add worker reservation logic
11. update sitemap queue to use discovery keys, not `rstrip("/")`
12. update URL-list crawl to canonicalize and dedupe before domain grouping

### 14.3 `crawl_db.py`

Add:

1. migration support or schema auto-upgrade
2. canonical hash lookup
3. discovered target persistence
4. action persistence
5. reservation functions
6. cache lookup by canonical hash and page type

Update old `_url_hash(url)` behavior only after migration is safe.

During transition:

- keep old `url_hash`
- add new `canonical_url_hash`
- read from new first, old second

### 14.4 `database_schema.sql`

Add the migration SQL from section 5.

### 14.5 Tests

Create a small local test site with:

1. normal links
2. duplicated query parameter order
3. tracking parameters
4. JavaScript href link
5. onclick navigation
6. button opening modal
7. modal CTA link
8. card opening modal
9. DOM-only tab switch
10. repeated links to the same target

## 15. Step-by-step implementation order

### Step 1: Add canonicalization without changing crawl behavior

Tasks:

1. Add `canonicalize_url()`.
2. Add unit tests.
3. Log raw URL and canonical URL during discovery.
4. Do not change DB writes yet.

Acceptance criteria:

- Query parameters such as `tab` and `MountingType` are preserved.
- Tracking parameters are dropped.
- Parameter order no longer creates duplicates.

### Step 2: Update filename stems

Tasks:

1. Update `_url_output_stem()`.
2. Add query hash suffix when canonical URL contains query.
3. Confirm no overwrites occur for different semantic query states.

Acceptance criteria:

- Two URLs with different meaningful query params create different artifact files.
- Same URL with only tracking params maps to the same artifact files.

### Step 3: Add DB migration

Tasks:

1. Add new columns and tables.
2. Add read/write functions in `crawl_db.py`.
3. Keep old fields working.

Acceptance criteria:

- Existing DB does not break.
- New crawls write canonical URL, canonical hash, page type, and discovered targets.

### Step 4: Add in-run dedupe by discovery key

Tasks:

1. Replace `seen = {url.rstrip("/")}` with `seen_discovery_keys`.
2. Use the same key for normal URLs and interactions.
3. Use this in sitemap and URL-list crawls.

Acceptance criteria:

- Same URL with reordered query params is queued once.
- Same action/card/button is clicked once per page.

### Step 5: Add persistent discovery table

Tasks:

1. Call `mark_discovered_target()` for every extracted URL/action.
2. Save source page and link text.
3. Update `seen_count` when the same target appears again.

Acceptance criteria:

- Repeated crawls increase `seen_count` instead of creating duplicate targets.
- Historical discovery can be inspected without crawling every target.

### Step 6: Add cache-first before browser startup

Tasks:

1. Add cache preflight in `fetch_single_result()`.
2. Add cache preflight in URL-list workers.
3. Add cache preflight in sitemap workers.
4. Return normal result payload with `cache_hit=True`.

Acceptance criteria:

- `cache_policy="cache_first"` does not start Chrome for a fresh cached single URL.
- Cached pages are counted as success.
- Missing artifact files force a live crawl even if DB row exists.

### Step 7: Add `crawl_scope="new_only"`

Tasks:

1. Add API option.
2. Before queueing, check if target is known from `pages` or `discovered_targets`.
3. Skip known targets.
4. Still allow seed/listing pages to refresh discovery when `refresh_discovery_pages=True`.

Acceptance criteria:

- A second `new_only` crawl of the same site does not recrawl old pages.
- If the seed page now links to a new product page, only the new page is crawled.

### Step 8: Add JavaScript candidate extraction

Tasks:

1. Add JS code to list visible click candidates.
2. Return selector, text, href, onclick hash, data attributes, role, bounding box.
3. Filter obvious unwanted actions.

Acceptance criteria:

- `href="javascript:..."` links appear as action targets.
- Buttons and product cards appear as action targets.
- Cookie/login/cart/social buttons are skipped by default.

### Step 9: Add click execution and outcome detection

Tasks:

1. Click candidates one by one.
2. Detect URL navigation, new tab, modal, DOM interaction, or no change.
3. Restore page state after each click.
4. Persist each action result.

Acceptance criteria:

- JavaScript links that navigate to a URL are added to the crawl queue.
- Product cards that open overlays are captured.
- Modal CTAs are discovered.

### Step 10: Store modal and interaction virtual pages

Tasks:

1. Capture modal HTML and visible text.
2. Save `.html`, `.txt`, `.md`, and `.crawl.json` for modal pages.
3. Save as `page_type="modal"` with `parent_canonical_url_hash` and `action_key`.
4. Extract normal links and CTA buttons from the modal.

Acceptance criteria:

- A product overlay becomes searchable even without a real URL.
- Modal CTA links are queued or skipped according to dedupe/cache/new-only rules.

### Step 11: Add site profile support

Tasks:

1. Add YAML profile loader.
2. Add default profile.
3. Apply query keep/drop rules from profile.
4. Apply click include/exclude selectors from profile.
5. Apply allowed/blocked path rules from profile.

Acceptance criteria:

- The crawler works generically without a profile.
- Adding a profile improves behavior without code changes.

### Step 12: Add tests and logs

Tasks:

1. Unit tests for canonical URLs.
2. Unit tests for discovery keys.
3. Unit tests for new-only decisions.
4. Integration test with local JavaScript demo page.
5. Add crawl summary JSON with counts.

Summary file example:

```json
{
  "input_urls": 1,
  "html_links_found": 120,
  "interaction_candidates": 45,
  "javascript_targets_found": 18,
  "modal_pages_saved": 12,
  "queued": 36,
  "cache_hits": 80,
  "skipped_known": 90,
  "skipped_seen_in_run": 30,
  "failed": 2
}
```

Acceptance criteria:

- Logs explain why a target was queued, skipped, cache-hit, or failed.
- Junior developers can debug duplicate crawl decisions from the summary alone.

## 16. Safety limits

Default limits:

```text
max_interactions_per_page = 80
max_virtual_pages_per_page = 50
max_ctas_per_modal = 20
max_click_depth = 2
same_domain_only = true
allow_login = false
allow_cart = false
allow_checkout = false
allow_form_submit = false
```

Do not click:

```text
login
logout
account
cart
checkout
delete
remove
subscribe
newsletter
share
print
cookie settings
language switcher
```

unless a site profile explicitly allows it.

## 17. Recommended final crawl modes

### Normal full crawl with cache

```json
{
  "cache_policy": "cache_first",
  "crawl_scope": "all",
  "discovery_mode": "html_and_interactions",
  "crawl_js_links": true
}
```

### Only add new pages

```json
{
  "cache_policy": "cache_first",
  "crawl_scope": "new_only",
  "refresh_discovery_pages": true,
  "discovery_mode": "html_and_interactions",
  "crawl_js_links": true
}
```

### Force recrawl everything

```json
{
  "cache_policy": "force",
  "crawl_scope": "all",
  "discovery_mode": "html_and_interactions",
  "crawl_js_links": true
}
```

### Discovery refresh only

```json
{
  "cache_policy": "cache_first",
  "crawl_scope": "new_only",
  "refresh_discovery_pages": true,
  "discovery_mode": "html_and_interactions",
  "crawl_js_links": true,
  "limit": 0
}
```

## 18. Implementation checklist

- [ ] Add canonical URL function.
- [ ] Add canonical URL tests.
- [ ] Update output stems to include semantic query state.
- [ ] Add DB migration.
- [ ] Add discovered target table functions.
- [ ] Add action table functions.
- [ ] Add cache-first preflight.
- [ ] Add `crawl_scope="new_only"`.
- [ ] Replace in-run URL dedupe with discovery-key dedupe.
- [ ] Add persistent reservation before worker crawl.
- [ ] Add JavaScript click candidate extraction.
- [ ] Add click outcome detection.
- [ ] Add modal virtual page saving.
- [ ] Add modal CTA extraction.
- [ ] Add site profile loader.
- [ ] Add generic default profile.
- [ ] Add integration test site.
- [ ] Add crawl summary JSON.
- [ ] Update REST API docs.

## 19. Definition of done

The task is done when:

1. Running the same crawl twice with `cache_policy="cache_first"` does not refetch fresh cached pages.
2. Running the same crawl twice with `crawl_scope="new_only"` only crawls newly discovered pages or newly discovered interactions.
3. URLs with semantic query parameters are not collapsed into path-only URLs.
4. Tracking query parameters do not create duplicate pages.
5. JavaScript links and clickable product cards are discovered.
6. Product overlays/modals are saved as virtual searchable pages.
7. Modal CTA buttons are discovered and queued.
8. The crawler can be adapted for PROLICHT, XAL, Lightnet, and other sites through site profiles, without hardcoding one brand into the main crawler.
9. Logs and summary files clearly show why each target was crawled, skipped, cache-hit, or failed.
