# IMPLEMENTATION PLAN DRAFT — Fix Image Search in Online Knowledge

Task: `crawling-relations-problems`  
Date: 2026-06-17  

## Problem summary

When users search for project photos (e.g. "show me Bethel project photos"), the MCP
returns product family pages but not the actual project reference page with images.
The cause is 7 identified issues in the pipeline.

## Files to modify

### 1. `app/online_knowledge/pipeline_v2.py` — `build_and_embed()`

**Problem:** Creates page-level vector docs (`page_summary`, `page_questions`, etc.)
but does NOT create image-level vector documents from `images.ai_description` or
`image_summaries.summary`.

**Fix:** Add an `image_description` vector doc type. Build text from:
- `image_summaries.summary` or `images.ai_description`
- `images.image_category`
- Source page title, URL, segment
- Product families if page is project_reference

### 2. `app/online_knowledge/mcp_backend_knowledge.py` — `_enrich_results()`

**Problem A:** Enrichment attaches images only to the exact page IDs already returned
by search. It does not traverse from a product-family page to related project pages
and attach their images.

**Fix A:** After fetching `linked_pages`, for product_family results also look up
project_reference pages that reference the same product_family and attach their
representative images.

**Problem B:** Image query filters `analysis_status IN ('classified', 'done')` but
there's no `image_count_raw`, `image_count_after_status_filter`, or debug fields to
diagnose why images are missing.

**Fix B:** Add debug fields to search results:
- `image_count_available` — total images for this page
- `image_count_classified` — images with `classified`/`done` status
- `image_count_returned` — images after all filters
- `image_filter_rejection_reasons` — why images were excluded

### 3. `app/online_knowledge/mcp_backend_knowledge.py` — `_relation_recall()` or equivalent

**Problem:** Relation recall query joins `page_analysis pa` but the schema stores
page analysis fields directly on `pages`. No `page_analysis` table exists. The
exception is caught silently and returns empty recall.

**Fix:** Remove the `page_analysis` join. Query `page_category` and `segment`
directly from `pages` table.

### 4. `app/online_knowledge/enrich.py` — image summaries

**Problem:** Image summaries exist in `image_summaries` table but are not embedded
in the vector store. They're only used as enrichment after search.

**Fix:** No code change needed — image summaries are populated correctly. The fix
is in step 1 (embed them).

### 5. `app/online_knowledge/image_grid_threaded.py` — image classification

**Problem:** Images for some pages are still `analysis_status='pending'` because
the image pipeline ran before those pages were processed.

**Fix:** Always run image classification AFTER text analysis. The `run_all.ps1`
script already does this in the correct order. Ensure `--force` is not needed to
pick up new pages.

### 6. Project-photo retrieval mode

**Problem:** No explicit mode for "show me project photos". The search relies on
FAISS to rank product-family pages above project-reference pages.

**Fix:** Add a `filter_category` parameter to `search_online_knowledge` (already exists).
When user asks for project photos, the LLM should use `filter_category='project_reference'`.
Add to tool description.

### 7. Image descriptions scored but not embedded

**Problem:** Search backend has scoring support for `image_description` and doc-type
boost for `image_description`, but no such vectors exist.

**Fix:** Step 1 creates the vectors. Step 2 ensures they are ranked properly.

## Verification steps

After fixes, run these SQL checks against Bethel:

```sql
-- 1) Bethel page exists with correct category
SELECT id, url, title, page_category, segment, image_count, analysis_status
FROM pages WHERE url LIKE '%bethel%' OR local_stem LIKE '%bethel%';

-- 2) Bethel images are classified
SELECT i.id, i.original_filename, i.analysis_status, i.image_category, i.is_representative
FROM images i JOIN pages p ON p.id = i.page_id
WHERE p.url LIKE '%bethel%' OR p.local_stem LIKE '%bethel%';

-- 3) Image-description vectors exist
SELECT vector_doc_type, COUNT(*) FROM vector_embeddings ve
JOIN pages p ON p.id = ve.page_id
WHERE p.url LIKE '%bethel%' OR p.local_stem LIKE '%bethel%'
GROUP BY vector_doc_type;
```

Expected: at least one `project_reference` result for Bethel with `results[].images[]`
containing `image_id`, `image_url`, description, filename, category, quality_score.
