# PLAN-FILE-STRUCTURE.md

## Goal

Separate API and MCP code into distinct directories with all runtime data
mapped to host folders so content is live-accessible from Windows.

## Target structure

```
D:\Docker\crawler\
├── app/                              # API service only (FastAPI + crawler)
│   ├── crawler.py
│   ├── server.py
│   ├── file_server.py
│   ├── gemini_detect.py
│   ├── webpage_vision_markdown.py
│   ├── consent_texts.json
│   ├── cookies/                      # consent cookies
│   └── requirements.txt              # API deps only
├── app-mcp/                          # MCP service only
│   ├── mcp_research.py
│   └── requirements.txt              # fastmcp, requests
├── data/                             # host-mapped (all volumes)
│   ├── sites_crawled/                # crawl output
│   ├── chrome_profile/               # Chromium profile
│   ├── downloads/                    # client downloads
│   ├── logs/                         # crawler.log + timing_logs/
│   └── db/                           # optional SQLite (short links)
├── client_scripts/                   # PowerShell/Python clients
│   ├── crawler_client.py
│   ├── crawl_and_download.ps1
│   ├── crawl_urls.ps1
│   └── crawled_pages/
├── scripts/
│   └── export-current-env.ps1
├── docs/                             # plans, AGENTS, README copies
│   ├── AGENTS.md
│   ├── README.md (app copy)
│   ├── IMPLEMENTATION_PLAN_API_FIRST.md
│   └── IMPLEMENTATION_PLAN_MCP_SECOND.md
├── docker-compose.yaml
├── Dockerfile                        # API image (chromium + xvfb)
├── Dockerfile.mcp                    # MCP image (python only, lighter)
├── .env
└── .dockerignore
```

## File moves

| From | To |
|---|---|
| `app/AGENTS.md` | `docs/AGENTS.md` (root AGENTS.md stays) |
| `app/README.md` | `docs/README-app.md` |
| `app/NODRIVER.md` | `docs/NODRIVER.md` (or keep in app/) |
| `app/crawler.py` | `app/crawler.py` (stays) |
| `app/server.py` | `app/server.py` (stays) |
| `app/file_server.py` | `app/file_server.py` (stays) |
| `app/gemini_detect.py` | `app/gemini_detect.py` (stays) |
| `app/webpage_vision_markdown.py` | `app/webpage_vision_markdown.py` (stays) |
| `app/requirements.txt` | `app/requirements.txt` (trimmed to API deps only) |
| `app/consent_texts.json` | `app/consent_texts.json` (stays) |
| `app/cookies/` | `app/cookies/` (stays) |
| `app/mcp_research.py` | **`app-mcp/mcp_research.py`** (move) |
| `app/fastmcp` dep in requirements.txt | **`app-mcp/requirements.txt`** (split) |
| `app/graphify-out/` | delete (stale) |
| `app/timing_logs/` | **`data/logs/timing_logs/`** (move) |
| `app/crawler.log` | **`data/logs/crawler.log`** (move, recreate per run) |
| `app/_test_tile_detect.py` | delete |
| `app/_timing_runner.py` | delete |
| `app/test_runner.py` | `docs/` or delete |
| `app/fast_smoke.py` | `docs/` or delete |
| `app/check_markdown.py` | `docs/` or delete |
| `app/check_image.py` | `docs/` or delete |
| `app/analyze_image_file.py` | `docs/` or delete |
| `app/repair_vision.py` | `docs/` or delete |
| `app/reset_output.py` | `docs/` or delete |
| `app/check_links.txt` | delete |
| `app/FAILED_SOLUTIONS.md` | delete |
| `app/FAILED_TRIES_IMAGE_BOUNDINGBOX_EXTRACTION.md` | delete |

## Volume mounts

```yaml
# API service
volumes:
  - ./app:/crawler/app
  - ./data/sites_crawled:/crawler/sites_crawled
  - ./data/chrome_profile:/crawler/chrome_profile
  - ./data/logs:/crawler/logs
  - ./data/downloads:/crawler/downloads

# MCP service (no chromium volumes)
volumes:
  - ./app-mcp:/crawler/app-mcp

# Client scripts access host paths directly (no container)
# Output: ./data/downloads/<domain>/ or ./client_scripts/crawled_pages/<domain>/
```

## Env variable renames

| Old | New | Reason |
|---|---|---|
| `CRAWLER_OUT` = `sites_crawled` | `CRAWLER_OUT` = `/crawler/data/sites_crawled` | unified data dir |
| `crawler.log` at `app/crawler.log` | logs at `/crawler/logs/crawler.log` | host-accessible |
| timing_logs at `app/timing_logs/` | logs at `/crawler/logs/timing_logs/` | host-accessible |

## Container paths

```
API container (/crawler):
  /crawler/app/          → code (volume: ./app)
  /crawler/sites_crawled → crawl output (volume: ./data/sites_crawled)
  /crawler/chrome_profile→ browser profile (volume: ./data/chrome_profile)
  /crawler/logs/         → logs (volume: ./data/logs)
  /crawler/downloads/    → downloads (volume: ./data/downloads)

MCP container (/crawler-mcp):
  /crawler-mcp/app-mcp/  → code (volume: ./app-mcp)
  (calls API via http://crawler:8114)
```

## Dockerfile.mcp (lighter, no chromium)

```dockerfile
FROM python:3.12-slim
ENV PYTHONUNBUFFERED=1
WORKDIR /crawler-mcp/app-mcp
COPY app-mcp/requirements.txt /tmp/requirements-mcp.txt
RUN pip install --no-cache-dir -r /tmp/requirements-mcp.txt
COPY app-mcp/ /crawler-mcp/app-mcp/
EXPOSE 8115
ENTRYPOINT ["python", "mcp_research.py"]
```

## Implementation order

1. Create `app-mcp/` and move `mcp_research.py` there
2. Create `data/` with subdirs: `sites_crawled`, `chrome_profile`, `logs`, `downloads`, `db`
3. Create `docs/` and move non-runtime files there
4. Update `Dockerfile` to use `data/logs/` and `data/sites_crawled/` paths
5. Create `Dockerfile.mcp` for MCP service
6. Update `docker-compose.yaml` with new volume mounts and MCP image
7. Update `app/crawler.py` log paths and timing_log paths to `data/logs/`
8. Update `app/requirements.txt` — remove `fastmcp` and `requests`
9. Create `app-mcp/requirements.txt` — `fastmcp` and `requests`
10. Test API container starts with new paths
11. Test MCP container starts with separate image
12. Remove stale files from `app/`
