# Plan

Status: **ready_for_implement**

Goal: implement Priority 4 (player-client fallback for PO Token / 429 failures) and Priority 5 (720p video download option) of `ROADMAP.md`.

## Scope (Priority 4) — T-006: Prefer non-web player clients

- Problem: after T-005, some downloads now fail via a different path: `HTTP Error 429: Too Many Requests` fetching the webpage, then `Unable to fetch GVS PO Token for web client: Missing required Visitor Data`, ending in `This video is not available`. The `web` client increasingly requires a PO Token yt-dlp can't generate, and is more exposed to IP-based rate limiting.
- Fix: add `extractor_args: {"youtube": {"player_client": ["android", "ios", "tv", "web"]}}` (or the current yt-dlp-recommended ordering) to the shared `ydl_opts` in `app.py` (same dict built at line ~317, alongside `js_runtimes`/`remote_components`), so yt-dlp tries clients that don't strictly require a PO Token first, and only falls back to `web` last.
- This applies uniformly to both `mp3` and `video` (and the new `video720` from Priority 5) modes since it's set on the shared `ydl_opts` before the per-mode `format`/`postprocessors` update.
- No new error-handling paths are needed: if every client fails, the existing `run_job` try/except already marks the job `status="error"` and the UI already surfaces that.

### Files to change (T-006)

- `app.py` — add the `extractor_args` `player_client` ordering to the shared `ydl_opts` dict.
- `README.md` — document that the app tries multiple YouTube player clients (`android`, `ios`, `tv`, then `web`) to avoid PO-Token/rate-limit failures on the `web` client, and that this is a best-effort mitigation that may need retuning as YouTube changes its anti-bot behavior.

### Validation (T-006)

- `python -m py_compile app.py`
- `docker build -t yt-dl .` — must exit 0.
- Live check: re-run a download of a video URL that previously failed with `Missing required Visitor Data` / `This video is not available` (e.g. the reported `LV-NXucnyrc` case) through the running container, and confirm it now succeeds.
- Inspect `docker logs` during that download and confirm an alternate client (`android`/`ios`/`tv`) is used and the job completes without the PO-Token warning blocking it.
- Regression spot-check: re-run one plain MP3 download and one plain Video download (non-playlist) and confirm they still succeed with progress bar + correct filename, unaffected by the client-ordering change.

## Scope (Priority 5) — T-007: Add a "Download Video 720p" option

- Add a third mode, `video720`, alongside the existing `mp3`/`video` modes, both server-side and in the UI.
- Server-side (`app.py`):
  - Extend the `mode` type from `Literal["mp3", "video"]` to `Literal["mp3", "video", "video720"]` everywhere it's declared: the `/download` route's `Form(...)` parameter (line ~280) and `JobState.mode` (line ~216).
  - In the `if mode == "mp3": ... else: ...` branch (line ~325 onward) that sets `format`/`merge_output_format`, add a branch for `video720`:
    ```python
    elif mode == "video720":
        ydl_opts.update(
            {
                "format": "bestvideo[height<=720]+bestaudio/best[height<=720]",
                "merge_output_format": "mp4",
            }
        )
    else:  # "video"
        ydl_opts.update({"format": "bestvideo+bestaudio/best", "merge_output_format": "mp4"})
    ```
    This caps the video stream at 720p while still falling back to the best available quality below 720p if that's all that exists (yt-dlp's format selector already does this — no extra fallback logic needed).
  - No changes needed to `sanitize_filename`, the job store, `progress_hook`, or the `/progress`/`/download/{job_id}/file` endpoints — they're mode-agnostic already.
- Frontend (inline HTML/JS in `app.py`):
  - Add a third button next to the existing two (line ~49): `<button type="button" onclick="downloadFile('video720')">Download Video 720p</button>`.
  - `downloadFile(mode)` already takes `mode` as a parameter and POSTs it through unchanged — confirm no mode-specific branching exists in the JS that would need a new case (per the current implementation, the JS is generic over `mode`, so this should be a markup-only change).

### Files to change (T-007)

- `app.py` — extend the `mode` `Literal` type (route param + `JobState`), add the `video720` format-selection branch, add the third HTML button.
- `README.md` — document the new **Download Video 720p** button: caps video quality at 720p, falls back to the best available quality if the source is below 720p, and behaves like the existing Video button otherwise (progress bar, auto-delivery, sanitized filename).

### Validation (T-007)

- `python -m py_compile app.py`
- `docker build -t yt-dl .` — must exit 0.
- Live check: click **Download Video 720p** on a video known to have >720p streams available; inspect the delivered file's video resolution (e.g. via `ffprobe`) and confirm it is ≤720p.
- Live check: click **Download Video 720p** on a video whose best quality is below 720p (or force via a low-quality test video) and confirm the download still succeeds with no error.
- Regression spot-check: confirm **Download MP3** and **Download Video** (unrestricted) still behave exactly as before — correct format/quality, progress bar, filename, playlist-single-video behavior.

## Acceptance Criteria

(from `ROADMAP.md` Priority 4 and Priority 5 — see those sections for full text)

- Priority 4: a previously PO-Token/429-failing video now downloads via an alternate player client; existing behavior (playlist/progress/filename/Deno) unaffected; failures still surface a clean UI error with no crash.
- Priority 5: a third "Download Video 720p" button exists; it caps video quality at 720p with graceful fallback when the source is lower quality; progress bar, auto-delivery, filename sanitization, and existing MP3/Video buttons are unaffected.

## Implementation Order

1. T-006 first (player-client fallback) — it's an independent, small `ydl_opts` change that other live-download validation (including T-007's) benefits from being in place first, since it reduces the chance of unrelated PO-Token failures muddying 720p testing.
2. T-007 second (720p mode) — builds on the same `ydl_opts` dict shape.

## Validation (both tasks, combined)

```bash
python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download Video 720p"
docker stop yt-dl-test
```

Live download checks (PO-Token/429 recovery, 720p resolution cap + fallback, MP3/Video regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced.

---

# Previous Plan (Priority 3 — completed, kept for reference)

Goal: implement Priority 3 of `ROADMAP.md` — fix downloads failing with a false `This video is not available` error caused by yt-dlp having no JavaScript runtime available in the container.

## Scope (Priority 3)

- **T-005 — Install Deno as yt-dlp's JS runtime**:
  - Add Deno (a single static binary, yt-dlp's documented lightweight JS runtime) to the `Dockerfile` so it's on `PATH` at container run time.
  - Point yt-dlp at it explicitly via `ydl_opts` (rather than relying on autodetection) so behavior doesn't silently regress if `deno` ever isn't found on `PATH`.
  - Allow yt-dlp to retrieve its official GitHub-hosted EJS challenge-solver scripts when current YouTube extraction requires them.
  - Verify server logs stop showing `WARNING: ... No supported JavaScript runtime could be found` during a normal download, and that a video URL exhibiting the `This video is not available` failure now succeeds.
  - Document the new build-time/runtime dependency and its purpose in `README.md`.

## Acceptance Criteria

(from `ROADMAP.md` Priority 3)

- `docker build` still completes without errors after adding Deno.
- Re-running a download of a video URL that previously failed with `This video is not available` (due to the missing JS runtime) succeeds and delivers the file.
- Server logs no longer show `No supported JavaScript runtime could be found` during a normal MP3 or Video download.
- Existing Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected.

## Implementation Phases

### Phase 1 — T-005: Add Deno and wire it into yt-dlp

**Files to change:**
- `Dockerfile`
- `app.py`
- `README.md` (document the Deno dependency and why it's needed)

**Changes in `Dockerfile`:**
1. Install Deno in the build stage. Simplest reliable path for a `python:3.12-slim` (Debian) base without adding curl/unzip as extra layers if avoidable — use the official install script, which only needs `curl` and `unzip`:
   ```dockerfile
   RUN apt-get update \
       && apt-get install -y --no-install-recommends ffmpeg curl unzip \
       && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \
       && rm -rf /var/lib/apt/lists/*
   ```
   This places the `deno` binary at `/usr/local/bin/deno`, already on `PATH` for subsequent `RUN`/`CMD` layers and for the app at runtime.
2. Keep the rest of the Dockerfile (`WORKDIR`, `COPY requirements.txt`, `pip install`, `COPY app.py`, `EXPOSE`, `CMD`) unchanged.

**Changes in `app.py`:**
1. In the shared `ydl_opts` construction (around line 317, alongside `outtmpl`/`noplaylist`/`progress_hooks`), explicitly point yt-dlp at the installed runtime instead of relying purely on `PATH` autodetection, so a missing/misconfigured binary fails loudly rather than silently degrading:
   ```python
   ydl_opts: dict[str, object] = {
       "outtmpl": output_template,
       "noplaylist": True,
       "progress_hooks": [progress_hook],
       "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}},
       "remote_components": ["ejs:github"],
   }
   ```
   (Exact option name/shape to be confirmed against the installed yt-dlp version's `--js-runtimes` support at implementation time — yt-dlp exposes this as a CLI flag `--js-runtimes RUNTIME[:PATH]`; the implementer should check `yt_dlp.YoutubeDL` / `yt_dlp.options` for the corresponding `ydl_opts` key in the pinned `yt-dlp` version and use that key, falling back to relying on autodetection via `PATH` only if no explicit option key exists in that version, in which case the Dockerfile's `PATH` install alone satisfies the requirement.)
2. Add `remote_components: ["ejs:github"]` to allow the EJS scripts needed by current yt-dlp releases to be fetched from the official yt-dlp GitHub repository.
3. Add `import shutil` if not already imported and used only for this lookup.

**Validation:**
- `python -m py_compile app.py`
- `docker build -t yt-dl .` — must exit 0.
- `docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl`, then `docker exec yt-dl-test deno --version` to confirm the binary is present and runnable in the final image.
- Live check: paste the video URL that previously failed (`https://www.youtube.com/watch?v=7MrdyaSlOfI`, without playlist params) through the running container for MP3 or Video mode, and confirm it downloads successfully instead of failing with `This video is not available`.
- Inspect `docker logs yt-dl-test` during that download and confirm the `No supported JavaScript runtime could be found` warning is no longer present.
- Regression spot-check: re-run one playlist-link download (`&list=...`) and confirm only the single video downloads, progress bar still updates, and the delivered filename still matches the sanitized video title (Priority 1/2 behavior unaffected).
- `docker stop yt-dl-test`

**Documentation update (`README.md`):**
- Add a short note under "Getting Started" / near the Build section stating that the image installs Deno and permits official GitHub-hosted EJS challenge-solver retrieval so yt-dlp can perform JS-based signature/PO-token deciphering required by current YouTube extraction, and that GitHub access is needed for downloads requiring those scripts.

## Validation (Priority 3)

```bash
python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
docker exec yt-dl-test deno --version
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test
```

Live download checks (previously-failing video URL now succeeds, log warning gone, playlist-link/progress-bar/filename regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced.

---

# Previous Plan (Priority 2 — completed, kept for reference)

Goal: implement Priority 2 of `ROADMAP.md` — single-video-only downloads from playlist links, correctly-named output files, and a live progress bar in the browser UI.

## Scope

- **T-003 — Single video from playlist links + correct filenames** (no architecture change; still one synchronous `POST /download` request/response):
  - Pass yt-dlp's `noplaylist: True` so a playlist URL or a `watch?v=...&list=...` URL only ever downloads the one referenced video.
  - Add a `sanitize_filename(title, fallback_id)` helper: strip characters outside letters (any language)/digits/spaces/`- _ . ( ) , '`, collapse resulting whitespace, trim, and fall back to the video id if nothing usable remains.
  - Extract `title`/`id` from yt-dlp's result info dict and pass the sanitized name explicitly as `FileResponse(..., filename=...)`, decoupled from whatever yt-dlp happens to name the file on disk.
  - Fix the real UUID-looking-filename bug: `response.blob()` in the frontend JS strips the `Content-Disposition` header, so `link.download = ''` makes the browser invent a blob-id-like name. Read the filename from the `Content-Disposition` response header in JS and set `link.download` to it explicitly (fallback to a generic name only if the header is somehow missing).
- **T-004 — Background job + live progress bar** (architecture change, builds on T-003):
  - Split `POST /download` into three endpoints:
    - `POST /download` — validates the URL, creates an in-memory job record (`job_id = uuid4()`, status `pending`), schedules the actual yt-dlp run on a background thread (`asyncio.get_running_loop().run_in_executor`), and returns `{"job_id": ...}` immediately (no file streaming here anymore).
    - `GET /progress/{job_id}` — returns the job's current `{status, percent, speed, downloaded_bytes, total_bytes, filename, error}` from the in-memory store. `status` is one of `downloading`, `finished`, `error`.
    - `GET /download/{job_id}/file` — once `status == finished`, streams the prepared file via `FileResponse` (reusing T-003's sanitized filename) with `BackgroundTask` cleanup of the job's temp dir and job record; returns 409/404 if called before completion or after cleanup.
  - Progress capture: a yt-dlp `progress_hooks` callback updates the job's dict on each tick using `downloaded_bytes`, `total_bytes` (or `total_bytes_estimate`), `speed`, and computed percent; a `postprocessor_hooks` callback (or the `finished` hook state) marks the job `finished` once the final file is ready.
  - Job store: a plain in-memory `dict[str, JobState]` guarded by a `threading.Lock` (single-process app, matches "no persistent storage" constraint). Entries are removed after the file is served, and a periodic/best-effort cleanup drops stale jobs (e.g. no progress endpoint hit within N minutes) so a browser tab closed mid-download doesn't leak a temp dir forever.
  - Frontend JS rework:
    - `downloadFile(mode)` now: POSTs to `/download` to obtain `job_id`, shows the progress bar area, then polls `GET /progress/{job_id}` every 500ms.
    - On each poll: update a `<progress>` bar (or styled div) with percent, and a text line with speed (human-readable, e.g. `1.2 MB/s`) and `downloaded / total` size (human-readable bytes).
    - On `status == finished`: stop polling, fetch `GET /download/{job_id}/file`, read filename from `Content-Disposition` (per T-003's fix), trigger the save via the existing blob+anchor pattern, then hide the progress bar and clear status text.
    - On `status == error`: stop polling, show the error message in the existing `#status` div, hide the progress bar.
  - HTML/CSS: add a progress bar element (native `<progress>` plus a small text line for speed/size) directly below the two buttons, hidden by default, shown only while a job is active.

## Acceptance Criteria

(from `ROADMAP.md` Priority 2)

- Pasting a playlist URL (or a watch URL with a `list=` param) and clicking MP3 or Video downloads only that one video, never the rest of the playlist.
- Starting a download shows a progress bar below the buttons that updates with percentage, speed, and downloaded/total size while the server is processing.
- When the download finishes, the file is automatically delivered to the browser (save dialog) without an extra click.
- The saved file's name matches the video's title (emoji/icons/symbols removed, normal characters kept) instead of a UUID or other opaque id, and still ends in the correct extension.
- If the download fails server-side, the UI shows a clear error in place of the progress bar (no server crash, no stuck spinner).

## Implementation Phases

### Phase 1 — T-003: Single video from playlists + correct filenames

**Files to change:**
- `app.py`
- `README.md` (document: playlist links only download the referenced video; filenames follow the video title with icons/emoji stripped)

**Changes in `app.py`:**
1. Add `ydl_opts["noplaylist"] = True` for both `mp3` and `video` modes.
2. Add a module-level helper:
   ```python
   import re

   _FILENAME_ALLOWED = re.compile(r"[^\w\s.,'()\-]", re.UNICODE)
   _WHITESPACE = re.compile(r"\s+")

   def sanitize_filename(title: str, fallback_id: str) -> str:
       cleaned = _FILENAME_ALLOWED.sub("", title or "")
       cleaned = _WHITESPACE.sub(" ", cleaned).strip()
       return cleaned or fallback_id
   ```
   (`\w` under `re.UNICODE`, the Python 3 default, matches any-language letters/digits/underscore, so non-Latin titles are preserved; emoji/symbols/pictographs fall outside `\w`, whitespace, and the explicit punctuation set and are dropped.)
3. In `fetch_media`, call `downloader.extract_info(url, download=True)` (instead of `.download([url])`) to get the info dict back; read `info.get("title")` and `info.get("id")`, compute `display_name = sanitize_filename(title, id) + Path(media_path).suffix`.
4. Pass `filename=display_name` to the existing `FileResponse(...)` call instead of `media_path.name`.
5. In the frontend `<script>`, change the success branch of `downloadFile`:
   ```js
   const disposition = response.headers.get('content-disposition') || '';
   const match = /filename\*?=(?:UTF-8''|")?([^\";]+)/i.exec(disposition);
   const filename = match ? decodeURIComponent(match[1].replace(/"/g, '')) : 'download';
   ...
   link.download = filename;
   ```
   Keep the rest of the blob/anchor logic unchanged.

**Validation:**
- `python -m py_compile app.py`
- Unit-style check (via a small mocked test or manual `python -c`) that `sanitize_filename("Song 🔥 Title ✨ (Live)", "abc123") == "Song  Title  (Live)"` collapsed to single spaces → `"Song Title (Live)"`, and `sanitize_filename("🔥🔥🔥", "abc123") == "abc123"`.
- Manual/live check: paste a `watch?v=...&list=...` URL and confirm only the single video downloads; confirm the saved filename in the browser matches the (emoji-stripped) video title, not a UUID.

### Phase 2 — T-004: Background job + live progress bar

**Files to change:**
- `app.py`
- `README.md` (document the new `/download`, `/progress/{job_id}`, `/download/{job_id}/file` flow and the progress bar UI; note job state is in-memory/non-persistent)

**Backend changes in `app.py`:**
1. Add a `JobState` structure (dataclass or `TypedDict`) with fields: `status` (`"downloading" | "finished" | "error"`), `percent`, `speed`, `downloaded_bytes`, `total_bytes`, `temp_dir`, `file_path`, `filename`, `error_message`, `mode`, `created_at`.
2. Add a module-level `_jobs: dict[str, JobState]` and `_jobs_lock = threading.Lock()`.
3. `POST /download` (`url`, `mode` form fields, same validation as today):
   - Validate URL as today (raise `HTTPException(400)` on invalid/empty).
   - Create `job_id = str(uuid4())`, create the temp dir, seed a `JobState(status="downloading", ...)` in `_jobs` under the lock.
   - Define a `progress_hook(d)` closure that, under the lock, updates the job's `percent`/`speed`/`downloaded_bytes`/`total_bytes` from `d["status"]`, `d.get("downloaded_bytes")`, `d.get("total_bytes") or d.get("total_bytes_estimate")`, `d.get("speed")`; on `d["status"] == "error"` marks the job `error`.
   - Build `ydl_opts` same as T-003 (`noplaylist: True`, plus `progress_hooks: [progress_hook]`).
   - Define `run_job()` that calls `extract_info(url, download=True)`, computes `display_name` via `sanitize_filename` (from T-003), locates the produced file, and updates the job to `status="finished"` with `file_path`/`filename` set; wraps in try/except to set `status="error"`/`error_message` on failure, and cleans up the temp dir on error (not on success — success cleanup happens after the file is served).
   - Schedule `run_job` via `asyncio.get_running_loop().run_in_executor(None, run_job)` **without awaiting it** (fire-and-forget task kept referenced, e.g. appended to a module-level `set` of pending futures to avoid GC warnings).
   - Return `JSONResponse({"job_id": job_id})`.
4. `GET /progress/{job_id}`:
   - Look up the job under the lock; 404 (JSON error) if unknown.
   - Return `{"status", "percent", "speed", "downloaded_bytes", "total_bytes", "error"}` (omit file path from the response).
5. `GET /download/{job_id}/file`:
   - Look up the job; 404 if unknown, 409 (JSON error) if not yet `finished`.
   - Return `FileResponse(job.file_path, filename=job.filename, background=BackgroundTask(cleanup_job, job_id))` where `cleanup_job` removes the temp dir and pops the job from `_jobs` under the lock.
6. Best-effort stale-job sweep: a small helper (called opportunistically, e.g. at the top of `POST /download`) that drops any job older than a fixed TTL (e.g. 30 minutes) whose temp dir hasn't been claimed, cleaning up its temp dir — keeps the in-memory store from growing unbounded if a browser tab is closed mid-download. No background scheduler/thread is introduced solely for this; it piggybacks on request handling to keep the app dependency-free.

**Frontend changes (inline `HTML`/JS in `app.py`):**
1. Add markup below the buttons:
   ```html
   <div id="progress-wrap" hidden>
     <progress id="progress-bar" max="100" value="0"></progress>
     <div id="progress-text"></div>
   </div>
   ```
2. Rework `downloadFile(mode)`:
   - POST to `/download`, parse `{job_id}` (existing error handling for non-2xx/JSON stays the same shape as today).
   - Show `#progress-wrap`, start `setInterval` polling `GET /progress/{job_id}` every 500ms.
   - On each tick: update `#progress-bar.value` from `percent`; update `#progress-text` with a human-readable line, e.g. `formatSpeed(speed) + ' · ' + formatBytes(downloaded_bytes) + ' / ' + formatBytes(total_bytes)`.
   - Add small `formatBytes(n)` / `formatSpeed(n)` helpers (KB/MB/GB, `/s` suffix), handling `null`/unknown total gracefully (e.g. show downloaded size only if total is unknown).
   - On `status === 'finished'`: `clearInterval`, fetch `GET /download/{job_id}/file`, apply the same `Content-Disposition`-based filename logic from T-003, trigger the blob/anchor save, then hide `#progress-wrap` and reset text/bar.
   - On `status === 'error'`: `clearInterval`, hide `#progress-wrap`, show `error` (or a generic message) in `#status`.
   - Guard against overlapping downloads from double-clicks (e.g. disable buttons while a job is active, re-enable on finish/error).

**Validation:**
- `python -m py_compile app.py`
- Manual/live check via `docker run` (or local uvicorn): start an MP3 download, observe the progress bar advancing with percent/speed/size, confirm the file auto-downloads on completion with the correct sanitized title-based filename; repeat for Video mode.
- Manual check: trigger a server-side failure path (e.g. malformed-but-http-prefixed URL) and confirm the progress bar disappears and the existing `#status` error text appears instead, with no server crash.
- Manual check: confirm a playlist URL still only fetches the single video (retained from T-003) end-to-end through the new job flow.

## Validation (both phases)

```bash
python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test
```

Live download checks (MP3, Video, playlist-link-single-video, progress bar behavior, filename correctness, error path) are performed manually/by the reviewer against a real YouTube URL, consistent with how T-001/T-002 were validated — no new automated test framework is introduced.