diff --git a/.ai/HANDOFF.md b/.ai/HANDOFF.md index e05525e..545e0bf 100644 --- a/.ai/HANDOFF.md +++ b/.ai/HANDOFF.md @@ -229,3 +229,64 @@ Append-only role handoff log. Each role adds one entry when its step is complete | Next Role | implement | --- + +### T-006 — plan — 2026-09-11T13:33:14Z + +| Field | Value | +|-------|-------| +| Agent | claude | +| Summary | Planned Priority 4 fix for PO-Token/429 download failures: add `extractor_args.youtube.player_client` ordering (android/ios/tv before web) to the shared `ydl_opts` in `app.py` so extraction avoids the web client's PO Token requirement first. | +| Files Changed | ROADMAP.md, .ai/PLAN.md, .ai/TASKS.md | +| Next Role | implement | + +--- + +### T-007 — plan — 2026-09-11T13:33:14Z + +| Field | Value | +|-------|-------| +| Agent | claude | +| Summary | Planned Priority 5 feature: add a third "Download Video 720p" mode/button, capped at 720p via `bestvideo[height<=720]+bestaudio/best[height<=720]` with graceful fallback to lower quality, reusing existing progress/filename/playlist behavior. | +| Files Changed | ROADMAP.md, .ai/PLAN.md, .ai/TASKS.md | +| Next Role | implement | + +--- + +### T-006 — implement — 2026-09-11T13:36:04Z + +| Field | Value | +|-------|-------| +| Agent | codex | +| Summary | Configured yt-dlp to prefer Android, iOS, and TV player clients before web, avoiding the web client's PO-token failure path. | +| Files Changed | app.py, README.md, ROADMAP.md, .ai/PLAN.md, .ai/TASKS.md, .ai/HANDOFF.md | +| Validation | `python3 -m py_compile app.py`, `docker build -t yt-dl .`, and a live MP3 job for `https://www.youtube.com/watch?v=LV-NXucnyrc` passed; alternate clients were used after a web 429 and the file was delivered. | +| Commit | Pending reviewer approval | +| Next Role | review | + +--- + +### T-006 — implement — 2026-09-11T13:42:04Z + +| Field | Value | +|-------|-------| +| Agent | codex | +| Summary | Squashed the reviewed alternate YouTube player-client fallback into the task commit. | +| Files Changed | app.py, README.md, ROADMAP.md, .ai/PLAN.md, .ai/REVIEW.md, .ai/TASKS.md, .ai/HANDOFF.md | +| Validation | Reviewer verification passed; `git diff --check` passed. | +| Commit | Created by this handoff's task commit | +| Next Role | none | + +--- + +### T-006 — review — 2026-09-11T13:41:31Z + +| Field | Value | +|-------|-------| +| Agent | claude | +| Summary | Reviewed and verified the player-client ordering fix: rebuilt the Docker image, confirmed `player_client` is genuinely consumed by the installed yt-dlp extractor, and drove live downloads (previously-failing `LV-NXucnyrc`, plain MP3/Video, playlist-link, error-path) through the container — no PO-Token-blocking failure, alternate clients used after a web 429, all regressions passed. | +| Files Changed | .ai/TASKS.md, .ai/REVIEW.md, .ai/HANDOFF.md | +| Verdict | PASS | +| Blocking Findings | none | +| Next Role | implement | + +--- diff --git a/.ai/PLAN.md b/.ai/PLAN.md index 7283d22..8e261e8 100644 --- a/.ai/PLAN.md +++ b/.ai/PLAN.md @@ -2,6 +2,93 @@ Status: **ready_for_implement** +Goal: implement Priority 4 (player-client fallback for PO Token / 429 failures) and Priority 5 (720p video download option) of `ROADMAP.md`. + +## Scope (Priority 4) — T-006: Prefer non-web player clients + +- Problem: after T-005, some downloads now fail via a different path: `HTTP Error 429: Too Many Requests` fetching the webpage, then `Unable to fetch GVS PO Token for web client: Missing required Visitor Data`, ending in `This video is not available`. The `web` client increasingly requires a PO Token yt-dlp can't generate, and is more exposed to IP-based rate limiting. +- Fix: add `extractor_args: {"youtube": {"player_client": ["android", "ios", "tv", "web"]}}` (or the current yt-dlp-recommended ordering) to the shared `ydl_opts` in `app.py` (same dict built at line ~317, alongside `js_runtimes`/`remote_components`), so yt-dlp tries clients that don't strictly require a PO Token first, and only falls back to `web` last. +- This applies uniformly to both `mp3` and `video` (and the new `video720` from Priority 5) modes since it's set on the shared `ydl_opts` before the per-mode `format`/`postprocessors` update. +- No new error-handling paths are needed: if every client fails, the existing `run_job` try/except already marks the job `status="error"` and the UI already surfaces that. + +### Files to change (T-006) + +- `app.py` — add the `extractor_args` `player_client` ordering to the shared `ydl_opts` dict. +- `README.md` — document that the app tries multiple YouTube player clients (`android`, `ios`, `tv`, then `web`) to avoid PO-Token/rate-limit failures on the `web` client, and that this is a best-effort mitigation that may need retuning as YouTube changes its anti-bot behavior. + +### Validation (T-006) + +- `python -m py_compile app.py` +- `docker build -t yt-dl .` — must exit 0. +- Live check: re-run a download of a video URL that previously failed with `Missing required Visitor Data` / `This video is not available` (e.g. the reported `LV-NXucnyrc` case) through the running container, and confirm it now succeeds. +- Inspect `docker logs` during that download and confirm an alternate client (`android`/`ios`/`tv`) is used and the job completes without the PO-Token warning blocking it. +- Regression spot-check: re-run one plain MP3 download and one plain Video download (non-playlist) and confirm they still succeed with progress bar + correct filename, unaffected by the client-ordering change. + +## Scope (Priority 5) — T-007: Add a "Download Video 720p" option + +- Add a third mode, `video720`, alongside the existing `mp3`/`video` modes, both server-side and in the UI. +- Server-side (`app.py`): + - Extend the `mode` type from `Literal["mp3", "video"]` to `Literal["mp3", "video", "video720"]` everywhere it's declared: the `/download` route's `Form(...)` parameter (line ~280) and `JobState.mode` (line ~216). + - In the `if mode == "mp3": ... else: ...` branch (line ~325 onward) that sets `format`/`merge_output_format`, add a branch for `video720`: + ```python + elif mode == "video720": + ydl_opts.update( + { + "format": "bestvideo[height<=720]+bestaudio/best[height<=720]", + "merge_output_format": "mp4", + } + ) + else: # "video" + ydl_opts.update({"format": "bestvideo+bestaudio/best", "merge_output_format": "mp4"}) + ``` + This caps the video stream at 720p while still falling back to the best available quality below 720p if that's all that exists (yt-dlp's format selector already does this — no extra fallback logic needed). + - No changes needed to `sanitize_filename`, the job store, `progress_hook`, or the `/progress`/`/download/{job_id}/file` endpoints — they're mode-agnostic already. +- Frontend (inline HTML/JS in `app.py`): + - Add a third button next to the existing two (line ~49): ``. + - `downloadFile(mode)` already takes `mode` as a parameter and POSTs it through unchanged — confirm no mode-specific branching exists in the JS that would need a new case (per the current implementation, the JS is generic over `mode`, so this should be a markup-only change). + +### Files to change (T-007) + +- `app.py` — extend the `mode` `Literal` type (route param + `JobState`), add the `video720` format-selection branch, add the third HTML button. +- `README.md` — document the new **Download Video 720p** button: caps video quality at 720p, falls back to the best available quality if the source is below 720p, and behaves like the existing Video button otherwise (progress bar, auto-delivery, sanitized filename). + +### Validation (T-007) + +- `python -m py_compile app.py` +- `docker build -t yt-dl .` — must exit 0. +- Live check: click **Download Video 720p** on a video known to have >720p streams available; inspect the delivered file's video resolution (e.g. via `ffprobe`) and confirm it is ≤720p. +- Live check: click **Download Video 720p** on a video whose best quality is below 720p (or force via a low-quality test video) and confirm the download still succeeds with no error. +- Regression spot-check: confirm **Download MP3** and **Download Video** (unrestricted) still behave exactly as before — correct format/quality, progress bar, filename, playlist-single-video behavior. + +## Acceptance Criteria + +(from `ROADMAP.md` Priority 4 and Priority 5 — see those sections for full text) + +- Priority 4: a previously PO-Token/429-failing video now downloads via an alternate player client; existing behavior (playlist/progress/filename/Deno) unaffected; failures still surface a clean UI error with no crash. +- Priority 5: a third "Download Video 720p" button exists; it caps video quality at 720p with graceful fallback when the source is lower quality; progress bar, auto-delivery, filename sanitization, and existing MP3/Video buttons are unaffected. + +## Implementation Order + +1. T-006 first (player-client fallback) — it's an independent, small `ydl_opts` change that other live-download validation (including T-007's) benefits from being in place first, since it reduces the chance of unrelated PO-Token failures muddying 720p testing. +2. T-007 second (720p mode) — builds on the same `ydl_opts` dict shape. + +## Validation (both tasks, combined) + +```bash +python -m py_compile app.py +docker build -t yt-dl . +docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl +sleep 3 +curl -sf http://localhost:8080/ | grep -q "Download Video 720p" +docker stop yt-dl-test +``` + +Live download checks (PO-Token/429 recovery, 720p resolution cap + fallback, MP3/Video regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced. + +--- + +# Previous Plan (Priority 3 — completed, kept for reference) + Goal: implement Priority 3 of `ROADMAP.md` — fix downloads failing with a false `This video is not available` error caused by yt-dlp having no JavaScript runtime available in the container. ## Scope (Priority 3) diff --git a/.ai/REVIEW.md b/.ai/REVIEW.md index 04d701e..a68f84c 100644 --- a/.ai/REVIEW.md +++ b/.ai/REVIEW.md @@ -193,3 +193,40 @@ No blocker, major, minor, or nit findings. **Risks:** - The fix depends on outbound network access to `github.com` (for the EJS solver script) and `deno.land` (Dockerfile install-time only); if GitHub is unreachable from the container at runtime, the EJS-dependent extraction path could fail again — this is already called out in `README.md` as a requirement, so no action needed. - `js_runtimes`/`remote_components` are relatively new, evolving yt-dlp options (confirmed only against the currently pinned `requirements.txt` range, `yt-dlp>=2024.4.9`, as resolved to `2026.08.19` in the built image); a future yt-dlp release changing this option's shape would need to be caught by re-running this same build+live-download validation, not by `py_compile` alone. + +--- + +## T-006 — `app.py` (`extractor_args.youtube.player_client` ordering in shared `ydl_opts`) + `README.md` update + +**Verdict:** PASS + +### Findings + +No blocker, major, minor, or nit findings. + +### Verification + +**Steps performed:** +- Re-read `.ai/PLAN.md` (Priority 4 / T-006 spec) and diffed it against `app.py`, `README.md`, `ROADMAP.md`. The single-line addition — `"extractor_args": {"youtube": {"player_client": ["android", "ios", "tv", "web"]}}` in the shared `ydl_opts` dict, applied uniformly before the per-mode `format` update — matches the plan exactly. `README.md` documents the client-ordering mitigation and its best-effort nature; `ROADMAP.md` Priority 4 accurately describes the bug, fix, and acceptance criteria. +- `python3 -m py_compile app.py` — succeeded. +- `docker build -t yt-dl .` — succeeded (cached, layers unchanged since T-005). +- Verified the `extractor_args`/`player_client` key is genuinely consumed by the installed yt-dlp (`2026.08.19`) inside the built image, not silently ignored: `grep player_client` in `yt_dlp/extractor/youtube/_video.py` shows `self._configuration_arg('player_client')` is read directly by the extractor, and constructing a real `yt_dlp.YoutubeDL(...)` with the exact `ydl_opts` `app.py` builds raised no warnings/errors. +- Ran the container and drove live downloads through the real job flow (`POST /download` → poll `GET /progress/{job_id}` → `GET /download/{job_id}/file`): + - **Previously-failing video** (`LV-NXucnyrc`, mode `mp3`) — completed successfully. `docker logs` showed `WARNING: ... HTTP Error 429: Too Many Requests` on the initial webpage fetch (the originally-reported failure trigger), immediately followed by `Downloading android player API JSON`, `Downloading ios player API JSON`, `Downloading tv client config`/`tv player API JSON` — confirming the alternate clients were tried and used. No `Missing required Visitor Data` or `This video is not available` appeared; the job reached `status: "finished"` and the file was delivered via `Content-Disposition` with a correctly sanitized title-based filename (`Baby's 1st Space Adventure ....mp3`), verified as a valid MP3 with `file`. + - **Regression: plain MP3** (`jNQXAC9IVRw`) — unaffected, unchanged from prior task behavior. + - **Regression: plain Video** (`jNQXAC9IVRw`, mode `video`) — completed, delivered a valid MP4 (`file` confirms ISO Media MP4) with filename `Me at the zoo.mp4`. + - **Regression: playlist-link** (`jNQXAC9IVRw&list=RDMMjNQXAC9IVRw`, mode `mp3`) — `docker logs` showed `Downloading just the video jNQXAC9IVRw because of --no-playlist`, confirming T-003's single-video behavior is unaffected by the client-ordering change. + - **Error path**: posted a non-existent URL (`http://example.com/not-a-video`) — job transitioned to `status: "error"` with a clear `error` message (`HTTP Error 404: Not Found`), no server crash (`GET /` still returned 200 immediately after), confirming the existing error-handling path is unaffected. + - Progress bar percent/speed/downloaded/total advanced correctly for all live jobs. + - Stopped and removed the test container, deleted downloaded test files. + +**Findings from verification:** All acceptance criteria for T-006 hold: +- A video URL previously failing with `Missing required Visitor Data` / `This video is not available` now downloads successfully via an alternate player client (confirmed live with `LV-NXucnyrc`). +- Server logs show the alternate player clients (`android`/`ios`/`tv`) being used, and no PO-Token-blocking failure occurred. +- Existing playlist-single-video/progress-bar/filename/Deno behavior (T-003/T-004/T-005) is unaffected — all regression checks passed. +- Failures still surface a clean UI error with no crash (verified with the error-path check). +- `python -m py_compile app.py` passes. + +**Risks:** +- This is an explicitly best-effort mitigation (as documented in `README.md` and `ROADMAP.md`): YouTube's anti-bot posture changes over time, and the `android`/`ios`/`tv` clients could themselves become restricted later, requiring the client list to be retuned. No action needed now; this is a known, accepted risk per the plan's scope. +- The live test observed `WARNING: ... Some android/ios client https formats have been skipped ... SABR-only streaming experiment` — a yt-dlp/YouTube-side warning unrelated to this change (it did not block the download, format selection fell through to a working format) — noted for awareness, not a regression introduced by T-006. diff --git a/.ai/TASKS.md b/.ai/TASKS.md index 6968789..0503701 100644 --- a/.ai/TASKS.md +++ b/.ai/TASKS.md @@ -25,3 +25,5 @@ Command expectations: | T-003 | `app.py` (`noplaylist` option, `sanitize_filename` helper, title-based `FileResponse` filename, frontend `Content-Disposition`-based `link.download` fix) + `README.md` update | done | Pasting a playlist/`list=` URL downloads only the referenced video; saved filename matches the video title with emoji/icons stripped (normal chars kept) instead of a UUID, with correct extension; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile`, unit-checked `sanitize_filename`, and ran a live end-to-end playlist-link MP3 download + a plain video download through the built Docker image against real YouTube URLs — all passed; see `.ai/REVIEW.md` | none | | T-004 | `app.py` (job-based `POST /download` + `GET /progress/{job_id}` + `GET /download/{job_id}/file`, in-memory job store, yt-dlp `progress_hooks`, frontend polling + progress bar UI) + `README.md` update | done | Progress bar appears below the buttons on download start and updates with percent/speed/downloaded-total size; file auto-delivers to the browser on completion with the correct sanitized filename; server-side failure shows an error in place of the bar with no crash; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile` and drove the full job lifecycle live through the built Docker image against real YouTube URLs (progress polling, auto-delivered file, 404/409 semantics, server-side error path with no crash) — all passed; see `.ai/REVIEW.md` | none | | T-005 | `Dockerfile` (install Deno JS runtime) + `app.py` (point yt-dlp `ydl_opts` at the installed Deno binary and permit official EJS solver retrieval) + `README.md` update | done | `docker build` exits 0; `docker exec deno --version` succeeds; a video URL previously failing with `This video is not available` now downloads successfully; server logs no longer show `No supported JavaScript runtime could be found` during a download; playlist-single-video/progress-bar/filename behavior from T-003/T-004 is unaffected; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile`, `docker build`, verified the `js_runtimes`/`remote_components` `ydl_opts` shape against the installed yt-dlp source, and drove live MP3 downloads (previously-failing URL + playlist-link regression) through the built Docker image — no missing-runtime warning, EJS solver fetched and used, single-video/progress/filename behavior intact; see `.ai/REVIEW.md` | none | +| T-006 | `app.py` (`extractor_args.youtube.player_client` ordering in shared `ydl_opts`) + `README.md` update | done | A video URL previously failing with `Missing required Visitor Data` / `This video is not available` now downloads successfully via an alternate player client; existing playlist-single-video/progress-bar/filename/Deno behavior unaffected; failures still surface a clean UI error with no crash; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile`, `docker build`, confirmed the `player_client` key is read by the installed yt-dlp extractor, and drove live downloads (previously-failing `LV-NXucnyrc`, plain MP3/Video, playlist-link, error-path) through the built Docker image — all passed; see `.ai/REVIEW.md` | none | +| T-007 | `app.py` (`video720` mode: `Literal` type extension, `bestvideo[height<=720]+bestaudio/best[height<=720]` format branch, third HTML button) + `README.md` update | ready_for_implement | UI shows a third "Download Video 720p" button; downloads are capped at 720p with graceful fallback to lower quality when the source is below 720p; progress bar, auto-delivery, and sanitized filename behavior match existing modes; existing MP3/Video buttons unchanged; `python -m py_compile app.py` passes | pending | implement | diff --git a/README.md b/README.md index b447576..7d97569 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,11 @@ extraction, avoiding false "This video is not available" failures for some videos. The container must be able to reach GitHub during downloads that need those scripts. +To reduce YouTube PO-token and rate-limit failures, downloads try the Android, +iOS, and TV player clients before the web client. This is a best-effort +mitigation; YouTube's anti-bot requirements change and the client order may +need retuning in the future. + ### Run ```bash diff --git a/ROADMAP.md b/ROADMAP.md index e17b293..163ee62 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -79,3 +79,48 @@ Objective: fix downloads failing due to yt-dlp's missing JavaScript runtime. - Fixing videos that are genuinely unavailable (age-restricted with no workaround, region-blocked, deleted, private) — this priority only addresses the missing-JS-runtime-induced false negative. - General yt-dlp version upgrade policy beyond what's needed to pair with the new JS runtime. + +## Priority 4 + +Objective: fix downloads failing due to YouTube's PO Token requirement and IP-based rate limiting on the `web` player client. + +- Bug: after Priority 3's fix, some downloads now fail differently: `WARNING: [youtube] LV-...: Unable to download webpage: HTTP Error 429: Too Many Requests`, followed by `WARNING: [youtube] Unable to fetch GVS PO Token for web client: Missing required Visitor Data`, ending in `ERROR: [youtube] ...: This video is not available`. The `web` client increasingly requires a Proof-of-Origin (PO) Token yt-dlp cannot generate on its own, and is also more exposed to YouTube's IP-based rate limiting. +- Fix: configure yt-dlp's `extractor_args` (`player_client`) to prefer player clients that historically don't require a PO Token as strictly (e.g. `android`, `ios`, `tv`), falling back to `web` only if those fail, instead of relying on the `web` client first. +- This is a best-effort mitigation, not a permanent fix: YouTube's anti-bot requirements change over time, so the chosen client list may need retuning later if YouTube tightens restrictions on the alternate clients too. + +## Acceptance Criteria (Priority 4) + +- Re-running a download of a video URL that previously failed with `Missing required Visitor Data` / `This video is not available` under the `web` client now succeeds using an alternate player client. +- Server logs show the alternate player client(s) being used and no longer show `Unable to fetch GVS PO Token for web client` blocking the download. +- Existing Priority 1/2/3 behavior (single-video-from-playlist, progress bar, filename sanitization, Deno JS runtime) is unaffected. +- If all configured player clients fail for a given video, the existing error-handling path still surfaces a clear error in the UI with no server crash (no new failure mode introduced). + +## Out of Scope + +- Running a separate PO-Token-provider service/container. +- Retry/backoff handling for transient 429s (may be revisited later if switching clients doesn't sufficiently resolve rate limiting). +- Guaranteeing every video downloads successfully — YouTube's anti-bot measures are outside this app's control. + +## Priority 5 + +Objective: add a capped-720p video download option alongside the existing best-quality video download. + +- Add a third button, **Download Video 720p**, next to the existing **Download MP3** and **Download Video** buttons. +- **Download Video** keeps today's behavior unchanged: best available video+audio quality, no cap. +- **Download Video 720p** downloads the best available video+audio quality capped at 720p — i.e. yt-dlp format selection equivalent to `bestvideo[height<=720]+bestaudio/best[height<=720]`, merged the same way as today's video mode (`mp4` container via `merge_output_format`). +- If the video's best available quality is below 720p (e.g. only 480p exists), the 720p button downloads the best quality actually available — no error, no upscaling. +- All existing behavior applies unchanged to the new mode: single-video-from-playlist (`noplaylist`), background job + live progress bar, sanitized title-based filename with the correct extension, and error handling on failure. + +## Acceptance Criteria (Priority 5) + +- The UI shows three buttons: Download MP3, Download Video, Download Video 720p. +- Clicking **Download Video 720p** on a video that has streams above 720p delivers a file whose video stream is at most 720p (not the unrestricted best quality). +- Clicking **Download Video 720p** on a video whose best quality is below 720p still succeeds and delivers that lower-quality file (no error). +- The 720p mode shows the same live progress bar (percent/speed/downloaded-total size) and the same auto-delivery-on-completion behavior as the existing Video and MP3 modes. +- The delivered filename follows the same sanitized-title convention as the existing modes, with the correct video extension. +- Existing **Download MP3** and **Download Video** behavior is unchanged. + +## Out of Scope + +- Additional quality tiers beyond 720p (e.g. 480p, 1080p buttons) — only the existing best-quality option and the new 720p-capped option are in scope. +- A quality-selection dropdown or other UI beyond a third fixed button. diff --git a/app.py b/app.py index e766cb1..9f193d7 100644 --- a/app.py +++ b/app.py @@ -321,6 +321,8 @@ async def download( "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}}, # Current YouTube extraction may need yt-dlp's official EJS solver scripts. "remote_components": ["ejs:github"], + # Prefer clients that can avoid the web client's PO-token requirement. + "extractor_args": {"youtube": {"player_client": ["android", "ios", "tv", "web"]}}, } if mode == "mp3": ydl_opts.update(