# Plan

Status: **ready_for_implement**

Goal: implement Priority 3 of `ROADMAP.md` — fix downloads failing with a false `This video is not available` error caused by yt-dlp having no JavaScript runtime available in the container.

## Scope (Priority 3)

- **T-005 — Install Deno as yt-dlp's JS runtime**:
  - Add Deno (a single static binary, yt-dlp's documented lightweight JS runtime) to the `Dockerfile` so it's on `PATH` at container run time.
  - Point yt-dlp at it explicitly via `ydl_opts` (rather than relying on autodetection) so behavior doesn't silently regress if `deno` ever isn't found on `PATH`.
  - Allow yt-dlp to retrieve its official GitHub-hosted EJS challenge-solver scripts when current YouTube extraction requires them.
  - Verify server logs stop showing `WARNING: ... No supported JavaScript runtime could be found` during a normal download, and that a video URL exhibiting the `This video is not available` failure now succeeds.
  - Document the new build-time/runtime dependency and its purpose in `README.md`.

## Acceptance Criteria

(from `ROADMAP.md` Priority 3)

- `docker build` still completes without errors after adding Deno.
- Re-running a download of a video URL that previously failed with `This video is not available` (due to the missing JS runtime) succeeds and delivers the file.
- Server logs no longer show `No supported JavaScript runtime could be found` during a normal MP3 or Video download.
- Existing Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected.

## Implementation Phases

### Phase 1 — T-005: Add Deno and wire it into yt-dlp

**Files to change:**
- `Dockerfile`
- `app.py`
- `README.md` (document the Deno dependency and why it's needed)

**Changes in `Dockerfile`:**
1. Install Deno in the build stage. Simplest reliable path for a `python:3.12-slim` (Debian) base without adding curl/unzip as extra layers if avoidable — use the official install script, which only needs `curl` and `unzip`:
   ```dockerfile
   RUN apt-get update \
       && apt-get install -y --no-install-recommends ffmpeg curl unzip \
       && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \
       && rm -rf /var/lib/apt/lists/*
   ```
   This places the `deno` binary at `/usr/local/bin/deno`, already on `PATH` for subsequent `RUN`/`CMD` layers and for the app at runtime.
2. Keep the rest of the Dockerfile (`WORKDIR`, `COPY requirements.txt`, `pip install`, `COPY app.py`, `EXPOSE`, `CMD`) unchanged.

**Changes in `app.py`:**
1. In the shared `ydl_opts` construction (around line 317, alongside `outtmpl`/`noplaylist`/`progress_hooks`), explicitly point yt-dlp at the installed runtime instead of relying purely on `PATH` autodetection, so a missing/misconfigured binary fails loudly rather than silently degrading:
   ```python
   ydl_opts: dict[str, object] = {
       "outtmpl": output_template,
       "noplaylist": True,
       "progress_hooks": [progress_hook],
       "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}},
       "remote_components": ["ejs:github"],
   }
   ```
   (Exact option name/shape to be confirmed against the installed yt-dlp version's `--js-runtimes` support at implementation time — yt-dlp exposes this as a CLI flag `--js-runtimes RUNTIME[:PATH]`; the implementer should check `yt_dlp.YoutubeDL` / `yt_dlp.options` for the corresponding `ydl_opts` key in the pinned `yt-dlp` version and use that key, falling back to relying on autodetection via `PATH` only if no explicit option key exists in that version, in which case the Dockerfile's `PATH` install alone satisfies the requirement.)
2. Add `remote_components: ["ejs:github"]` to allow the EJS scripts needed by current yt-dlp releases to be fetched from the official yt-dlp GitHub repository.
3. Add `import shutil` if not already imported and used only for this lookup.

**Validation:**
- `python -m py_compile app.py`
- `docker build -t yt-dl .` — must exit 0.
- `docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl`, then `docker exec yt-dl-test deno --version` to confirm the binary is present and runnable in the final image.
- Live check: paste the video URL that previously failed (`https://www.youtube.com/watch?v=7MrdyaSlOfI`, without playlist params) through the running container for MP3 or Video mode, and confirm it downloads successfully instead of failing with `This video is not available`.
- Inspect `docker logs yt-dl-test` during that download and confirm the `No supported JavaScript runtime could be found` warning is no longer present.
- Regression spot-check: re-run one playlist-link download (`&list=...`) and confirm only the single video downloads, progress bar still updates, and the delivered filename still matches the sanitized video title (Priority 1/2 behavior unaffected).
- `docker stop yt-dl-test`

**Documentation update (`README.md`):**
- Add a short note under "Getting Started" / near the Build section stating that the image installs Deno and permits official GitHub-hosted EJS challenge-solver retrieval so yt-dlp can perform JS-based signature/PO-token deciphering required by current YouTube extraction, and that GitHub access is needed for downloads requiring those scripts.

## Validation (Priority 3)

```bash
python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
docker exec yt-dl-test deno --version
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test
```

Live download checks (previously-failing video URL now succeeds, log warning gone, playlist-link/progress-bar/filename regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced.

---

# Previous Plan (Priority 2 — completed, kept for reference)

Goal: implement Priority 2 of `ROADMAP.md` — single-video-only downloads from playlist links, correctly-named output files, and a live progress bar in the browser UI.

## Scope

- **T-003 — Single video from playlist links + correct filenames** (no architecture change; still one synchronous `POST /download` request/response):
  - Pass yt-dlp's `noplaylist: True` so a playlist URL or a `watch?v=...&list=...` URL only ever downloads the one referenced video.
  - Add a `sanitize_filename(title, fallback_id)` helper: strip characters outside letters (any language)/digits/spaces/`- _ . ( ) , '`, collapse resulting whitespace, trim, and fall back to the video id if nothing usable remains.
  - Extract `title`/`id` from yt-dlp's result info dict and pass the sanitized name explicitly as `FileResponse(..., filename=...)`, decoupled from whatever yt-dlp happens to name the file on disk.
  - Fix the real UUID-looking-filename bug: `response.blob()` in the frontend JS strips the `Content-Disposition` header, so `link.download = ''` makes the browser invent a blob-id-like name. Read the filename from the `Content-Disposition` response header in JS and set `link.download` to it explicitly (fallback to a generic name only if the header is somehow missing).
- **T-004 — Background job + live progress bar** (architecture change, builds on T-003):
  - Split `POST /download` into three endpoints:
    - `POST /download` — validates the URL, creates an in-memory job record (`job_id = uuid4()`, status `pending`), schedules the actual yt-dlp run on a background thread (`asyncio.get_running_loop().run_in_executor`), and returns `{"job_id": ...}` immediately (no file streaming here anymore).
    - `GET /progress/{job_id}` — returns the job's current `{status, percent, speed, downloaded_bytes, total_bytes, filename, error}` from the in-memory store. `status` is one of `downloading`, `finished`, `error`.
    - `GET /download/{job_id}/file` — once `status == finished`, streams the prepared file via `FileResponse` (reusing T-003's sanitized filename) with `BackgroundTask` cleanup of the job's temp dir and job record; returns 409/404 if called before completion or after cleanup.
  - Progress capture: a yt-dlp `progress_hooks` callback updates the job's dict on each tick using `downloaded_bytes`, `total_bytes` (or `total_bytes_estimate`), `speed`, and computed percent; a `postprocessor_hooks` callback (or the `finished` hook state) marks the job `finished` once the final file is ready.
  - Job store: a plain in-memory `dict[str, JobState]` guarded by a `threading.Lock` (single-process app, matches "no persistent storage" constraint). Entries are removed after the file is served, and a periodic/best-effort cleanup drops stale jobs (e.g. no progress endpoint hit within N minutes) so a browser tab closed mid-download doesn't leak a temp dir forever.
  - Frontend JS rework:
    - `downloadFile(mode)` now: POSTs to `/download` to obtain `job_id`, shows the progress bar area, then polls `GET /progress/{job_id}` every 500ms.
    - On each poll: update a `<progress>` bar (or styled div) with percent, and a text line with speed (human-readable, e.g. `1.2 MB/s`) and `downloaded / total` size (human-readable bytes).
    - On `status == finished`: stop polling, fetch `GET /download/{job_id}/file`, read filename from `Content-Disposition` (per T-003's fix), trigger the save via the existing blob+anchor pattern, then hide the progress bar and clear status text.
    - On `status == error`: stop polling, show the error message in the existing `#status` div, hide the progress bar.
  - HTML/CSS: add a progress bar element (native `<progress>` plus a small text line for speed/size) directly below the two buttons, hidden by default, shown only while a job is active.

## Acceptance Criteria

(from `ROADMAP.md` Priority 2)

- Pasting a playlist URL (or a watch URL with a `list=` param) and clicking MP3 or Video downloads only that one video, never the rest of the playlist.
- Starting a download shows a progress bar below the buttons that updates with percentage, speed, and downloaded/total size while the server is processing.
- When the download finishes, the file is automatically delivered to the browser (save dialog) without an extra click.
- The saved file's name matches the video's title (emoji/icons/symbols removed, normal characters kept) instead of a UUID or other opaque id, and still ends in the correct extension.
- If the download fails server-side, the UI shows a clear error in place of the progress bar (no server crash, no stuck spinner).

## Implementation Phases

### Phase 1 — T-003: Single video from playlists + correct filenames

**Files to change:**
- `app.py`
- `README.md` (document: playlist links only download the referenced video; filenames follow the video title with icons/emoji stripped)

**Changes in `app.py`:**
1. Add `ydl_opts["noplaylist"] = True` for both `mp3` and `video` modes.
2. Add a module-level helper:
   ```python
   import re

   _FILENAME_ALLOWED = re.compile(r"[^\w\s.,'()\-]", re.UNICODE)
   _WHITESPACE = re.compile(r"\s+")

   def sanitize_filename(title: str, fallback_id: str) -> str:
       cleaned = _FILENAME_ALLOWED.sub("", title or "")
       cleaned = _WHITESPACE.sub(" ", cleaned).strip()
       return cleaned or fallback_id
   ```
   (`\w` under `re.UNICODE`, the Python 3 default, matches any-language letters/digits/underscore, so non-Latin titles are preserved; emoji/symbols/pictographs fall outside `\w`, whitespace, and the explicit punctuation set and are dropped.)
3. In `fetch_media`, call `downloader.extract_info(url, download=True)` (instead of `.download([url])`) to get the info dict back; read `info.get("title")` and `info.get("id")`, compute `display_name = sanitize_filename(title, id) + Path(media_path).suffix`.
4. Pass `filename=display_name` to the existing `FileResponse(...)` call instead of `media_path.name`.
5. In the frontend `<script>`, change the success branch of `downloadFile`:
   ```js
   const disposition = response.headers.get('content-disposition') || '';
   const match = /filename\*?=(?:UTF-8''|")?([^\";]+)/i.exec(disposition);
   const filename = match ? decodeURIComponent(match[1].replace(/"/g, '')) : 'download';
   ...
   link.download = filename;
   ```
   Keep the rest of the blob/anchor logic unchanged.

**Validation:**
- `python -m py_compile app.py`
- Unit-style check (via a small mocked test or manual `python -c`) that `sanitize_filename("Song 🔥 Title ✨ (Live)", "abc123") == "Song  Title  (Live)"` collapsed to single spaces → `"Song Title (Live)"`, and `sanitize_filename("🔥🔥🔥", "abc123") == "abc123"`.
- Manual/live check: paste a `watch?v=...&list=...` URL and confirm only the single video downloads; confirm the saved filename in the browser matches the (emoji-stripped) video title, not a UUID.

### Phase 2 — T-004: Background job + live progress bar

**Files to change:**
- `app.py`
- `README.md` (document the new `/download`, `/progress/{job_id}`, `/download/{job_id}/file` flow and the progress bar UI; note job state is in-memory/non-persistent)

**Backend changes in `app.py`:**
1. Add a `JobState` structure (dataclass or `TypedDict`) with fields: `status` (`"downloading" | "finished" | "error"`), `percent`, `speed`, `downloaded_bytes`, `total_bytes`, `temp_dir`, `file_path`, `filename`, `error_message`, `mode`, `created_at`.
2. Add a module-level `_jobs: dict[str, JobState]` and `_jobs_lock = threading.Lock()`.
3. `POST /download` (`url`, `mode` form fields, same validation as today):
   - Validate URL as today (raise `HTTPException(400)` on invalid/empty).
   - Create `job_id = str(uuid4())`, create the temp dir, seed a `JobState(status="downloading", ...)` in `_jobs` under the lock.
   - Define a `progress_hook(d)` closure that, under the lock, updates the job's `percent`/`speed`/`downloaded_bytes`/`total_bytes` from `d["status"]`, `d.get("downloaded_bytes")`, `d.get("total_bytes") or d.get("total_bytes_estimate")`, `d.get("speed")`; on `d["status"] == "error"` marks the job `error`.
   - Build `ydl_opts` same as T-003 (`noplaylist: True`, plus `progress_hooks: [progress_hook]`).
   - Define `run_job()` that calls `extract_info(url, download=True)`, computes `display_name` via `sanitize_filename` (from T-003), locates the produced file, and updates the job to `status="finished"` with `file_path`/`filename` set; wraps in try/except to set `status="error"`/`error_message` on failure, and cleans up the temp dir on error (not on success — success cleanup happens after the file is served).
   - Schedule `run_job` via `asyncio.get_running_loop().run_in_executor(None, run_job)` **without awaiting it** (fire-and-forget task kept referenced, e.g. appended to a module-level `set` of pending futures to avoid GC warnings).
   - Return `JSONResponse({"job_id": job_id})`.
4. `GET /progress/{job_id}`:
   - Look up the job under the lock; 404 (JSON error) if unknown.
   - Return `{"status", "percent", "speed", "downloaded_bytes", "total_bytes", "error"}` (omit file path from the response).
5. `GET /download/{job_id}/file`:
   - Look up the job; 404 if unknown, 409 (JSON error) if not yet `finished`.
   - Return `FileResponse(job.file_path, filename=job.filename, background=BackgroundTask(cleanup_job, job_id))` where `cleanup_job` removes the temp dir and pops the job from `_jobs` under the lock.
6. Best-effort stale-job sweep: a small helper (called opportunistically, e.g. at the top of `POST /download`) that drops any job older than a fixed TTL (e.g. 30 minutes) whose temp dir hasn't been claimed, cleaning up its temp dir — keeps the in-memory store from growing unbounded if a browser tab is closed mid-download. No background scheduler/thread is introduced solely for this; it piggybacks on request handling to keep the app dependency-free.

**Frontend changes (inline `HTML`/JS in `app.py`):**
1. Add markup below the buttons:
   ```html
   <div id="progress-wrap" hidden>
     <progress id="progress-bar" max="100" value="0"></progress>
     <div id="progress-text"></div>
   </div>
   ```
2. Rework `downloadFile(mode)`:
   - POST to `/download`, parse `{job_id}` (existing error handling for non-2xx/JSON stays the same shape as today).
   - Show `#progress-wrap`, start `setInterval` polling `GET /progress/{job_id}` every 500ms.
   - On each tick: update `#progress-bar.value` from `percent`; update `#progress-text` with a human-readable line, e.g. `formatSpeed(speed) + ' · ' + formatBytes(downloaded_bytes) + ' / ' + formatBytes(total_bytes)`.
   - Add small `formatBytes(n)` / `formatSpeed(n)` helpers (KB/MB/GB, `/s` suffix), handling `null`/unknown total gracefully (e.g. show downloaded size only if total is unknown).
   - On `status === 'finished'`: `clearInterval`, fetch `GET /download/{job_id}/file`, apply the same `Content-Disposition`-based filename logic from T-003, trigger the blob/anchor save, then hide `#progress-wrap` and reset text/bar.
   - On `status === 'error'`: `clearInterval`, hide `#progress-wrap`, show `error` (or a generic message) in `#status`.
   - Guard against overlapping downloads from double-clicks (e.g. disable buttons while a job is active, re-enable on finish/error).

**Validation:**
- `python -m py_compile app.py`
- Manual/live check via `docker run` (or local uvicorn): start an MP3 download, observe the progress bar advancing with percent/speed/size, confirm the file auto-downloads on completion with the correct sanitized title-based filename; repeat for Video mode.
- Manual check: trigger a server-side failure path (e.g. malformed-but-http-prefixed URL) and confirm the progress bar disappears and the existing `#status` error text appears instead, with no server crash.
- Manual check: confirm a playlist URL still only fetches the single video (retained from T-003) end-to-end through the new job flow.

## Validation (both phases)

```bash
python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test
```

Live download checks (MP3, Video, playlist-link-single-video, progress bar behavior, filename correctness, error path) are performed manually/by the reviewer against a real YouTube URL, consistent with how T-001/T-002 were validated — no new automated test framework is introduced.