From 5a89def7d0407e24affbc4781e9cc20709aa4e79 Mon Sep 17 00:00:00 2001 From: Domagoj Zecevic Date: Fri, 11 Sep 2026 15:26:54 +0200 Subject: [PATCH] fix(downloads): restore YouTube downloads with Deno --- .ai/HANDOFF.md | 50 +++++++++++++++++++++++++++++ .ai/PLAN.md | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++ .ai/REVIEW.md | 42 +++++++++++++++++++++++++ .ai/TASKS.md | 1 + Dockerfile | 3 +- README.md | 8 +++++ ROADMAP.md | 21 +++++++++++++ app.py | 3 ++ 8 files changed, 212 insertions(+), 1 deletion(-) diff --git a/.ai/HANDOFF.md b/.ai/HANDOFF.md index 38912cf..e05525e 100644 --- a/.ai/HANDOFF.md +++ b/.ai/HANDOFF.md @@ -179,3 +179,53 @@ Append-only role handoff log. Each role adds one entry when its step is complete | Next Role | none | --- + +### T-005 — plan — 2026-09-11T12:45:39Z + +| Field | Value | +|-------|-------| +| Agent | claude | +| Summary | Planned Priority 3 fix for downloads failing with a false "This video is not available" error caused by yt-dlp having no JS runtime in the container: install Deno in the Dockerfile and point yt-dlp's `ydl_opts` at it explicitly, plus a README note on the new dependency. | +| Files Changed | ROADMAP.md, .ai/PLAN.md, .ai/TASKS.md | +| Next Role | implement | + +--- + +### T-005 — implement — 2026-09-11T12:53:11Z + +| Field | Value | +|-------|-------| +| Agent | codex | +| Summary | Installed Deno, explicitly configured yt-dlp to use it, and enabled official GitHub EJS challenge-solver retrieval so the previously failing YouTube URL downloads successfully. | +| Files Changed | Dockerfile, app.py, README.md, ROADMAP.md, .ai/PLAN.md, .ai/TASKS.md, .ai/HANDOFF.md | +| Validation | `python3 -m py_compile app.py`, `docker build -t yt-dl .`, Deno/EJS configuration checks, and a live MP3 job for `https://www.youtube.com/watch?v=7MrdyaSlOfI` passed; no missing-runtime or skipped-EJS warning appeared. | +| Commit | Pending reviewer approval | +| Next Role | review | + +--- + +### T-005 — implement — 2026-09-11T13:26:25Z + +| Field | Value | +|-------|-------| +| Agent | codex | +| Summary | Squashed the reviewed Deno and EJS solver configuration into the task commit. | +| Files Changed | Dockerfile, app.py, README.md, ROADMAP.md, .ai/PLAN.md, .ai/REVIEW.md, .ai/TASKS.md, .ai/HANDOFF.md | +| Validation | Reviewer verification passed; `git diff --check` passed. | +| Commit | Created by this handoff's task commit | +| Next Role | none | + +--- + +### T-005 — review — 2026-09-11T13:25:24Z + +| Field | Value | +|-------|-------| +| Agent | claude | +| Summary | Reviewed and verified the Deno JS runtime fix: rebuilt the Docker image, confirmed the `js_runtimes`/`remote_components` `ydl_opts` shape against the installed yt-dlp source, and drove live MP3 downloads (previously-failing URL + a playlist-link regression check) through the container with no missing-runtime warning and the EJS solver actively used. | +| Files Changed | .ai/TASKS.md, .ai/REVIEW.md, .ai/HANDOFF.md | +| Verdict | PASS | +| Blocking Findings | none | +| Next Role | implement | + +--- diff --git a/.ai/PLAN.md b/.ai/PLAN.md index 726ea3b..7283d22 100644 --- a/.ai/PLAN.md +++ b/.ai/PLAN.md @@ -2,6 +2,91 @@ Status: **ready_for_implement** +Goal: implement Priority 3 of `ROADMAP.md` — fix downloads failing with a false `This video is not available` error caused by yt-dlp having no JavaScript runtime available in the container. + +## Scope (Priority 3) + +- **T-005 — Install Deno as yt-dlp's JS runtime**: + - Add Deno (a single static binary, yt-dlp's documented lightweight JS runtime) to the `Dockerfile` so it's on `PATH` at container run time. + - Point yt-dlp at it explicitly via `ydl_opts` (rather than relying on autodetection) so behavior doesn't silently regress if `deno` ever isn't found on `PATH`. + - Allow yt-dlp to retrieve its official GitHub-hosted EJS challenge-solver scripts when current YouTube extraction requires them. + - Verify server logs stop showing `WARNING: ... No supported JavaScript runtime could be found` during a normal download, and that a video URL exhibiting the `This video is not available` failure now succeeds. + - Document the new build-time/runtime dependency and its purpose in `README.md`. + +## Acceptance Criteria + +(from `ROADMAP.md` Priority 3) + +- `docker build` still completes without errors after adding Deno. +- Re-running a download of a video URL that previously failed with `This video is not available` (due to the missing JS runtime) succeeds and delivers the file. +- Server logs no longer show `No supported JavaScript runtime could be found` during a normal MP3 or Video download. +- Existing Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected. + +## Implementation Phases + +### Phase 1 — T-005: Add Deno and wire it into yt-dlp + +**Files to change:** +- `Dockerfile` +- `app.py` +- `README.md` (document the Deno dependency and why it's needed) + +**Changes in `Dockerfile`:** +1. Install Deno in the build stage. Simplest reliable path for a `python:3.12-slim` (Debian) base without adding curl/unzip as extra layers if avoidable — use the official install script, which only needs `curl` and `unzip`: + ```dockerfile + RUN apt-get update \ + && apt-get install -y --no-install-recommends ffmpeg curl unzip \ + && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \ + && rm -rf /var/lib/apt/lists/* + ``` + This places the `deno` binary at `/usr/local/bin/deno`, already on `PATH` for subsequent `RUN`/`CMD` layers and for the app at runtime. +2. Keep the rest of the Dockerfile (`WORKDIR`, `COPY requirements.txt`, `pip install`, `COPY app.py`, `EXPOSE`, `CMD`) unchanged. + +**Changes in `app.py`:** +1. In the shared `ydl_opts` construction (around line 317, alongside `outtmpl`/`noplaylist`/`progress_hooks`), explicitly point yt-dlp at the installed runtime instead of relying purely on `PATH` autodetection, so a missing/misconfigured binary fails loudly rather than silently degrading: + ```python + ydl_opts: dict[str, object] = { + "outtmpl": output_template, + "noplaylist": True, + "progress_hooks": [progress_hook], + "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}}, + "remote_components": ["ejs:github"], + } + ``` + (Exact option name/shape to be confirmed against the installed yt-dlp version's `--js-runtimes` support at implementation time — yt-dlp exposes this as a CLI flag `--js-runtimes RUNTIME[:PATH]`; the implementer should check `yt_dlp.YoutubeDL` / `yt_dlp.options` for the corresponding `ydl_opts` key in the pinned `yt-dlp` version and use that key, falling back to relying on autodetection via `PATH` only if no explicit option key exists in that version, in which case the Dockerfile's `PATH` install alone satisfies the requirement.) +2. Add `remote_components: ["ejs:github"]` to allow the EJS scripts needed by current yt-dlp releases to be fetched from the official yt-dlp GitHub repository. +3. Add `import shutil` if not already imported and used only for this lookup. + +**Validation:** +- `python -m py_compile app.py` +- `docker build -t yt-dl .` — must exit 0. +- `docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl`, then `docker exec yt-dl-test deno --version` to confirm the binary is present and runnable in the final image. +- Live check: paste the video URL that previously failed (`https://www.youtube.com/watch?v=7MrdyaSlOfI`, without playlist params) through the running container for MP3 or Video mode, and confirm it downloads successfully instead of failing with `This video is not available`. +- Inspect `docker logs yt-dl-test` during that download and confirm the `No supported JavaScript runtime could be found` warning is no longer present. +- Regression spot-check: re-run one playlist-link download (`&list=...`) and confirm only the single video downloads, progress bar still updates, and the delivered filename still matches the sanitized video title (Priority 1/2 behavior unaffected). +- `docker stop yt-dl-test` + +**Documentation update (`README.md`):** +- Add a short note under "Getting Started" / near the Build section stating that the image installs Deno and permits official GitHub-hosted EJS challenge-solver retrieval so yt-dlp can perform JS-based signature/PO-token deciphering required by current YouTube extraction, and that GitHub access is needed for downloads requiring those scripts. + +## Validation (Priority 3) + +```bash +python -m py_compile app.py +docker build -t yt-dl . +docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl +sleep 3 +docker exec yt-dl-test deno --version +curl -sf http://localhost:8080/ | grep -q "Download MP3" +docker stop yt-dl-test +``` + +Live download checks (previously-failing video URL now succeeds, log warning gone, playlist-link/progress-bar/filename regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced. + +--- + +# Previous Plan (Priority 2 — completed, kept for reference) + Goal: implement Priority 2 of `ROADMAP.md` — single-video-only downloads from playlist links, correctly-named output files, and a live progress bar in the browser UI. ## Scope diff --git a/.ai/REVIEW.md b/.ai/REVIEW.md index 26defdb..04d701e 100644 --- a/.ai/REVIEW.md +++ b/.ai/REVIEW.md @@ -151,3 +151,45 @@ No blocker or major findings. **Risks:** - Progress percent can remain at 0% for formats where yt-dlp cannot report a total size (noted above, minor, not required). - The in-memory job store means an app restart mid-download loses all job state/progress for any in-flight browser sessions; this matches the explicitly stated "no persistent storage" design constraint and is now documented in `README.md`. + +--- + +## T-005 — `Dockerfile` (install Deno JS runtime) + `app.py` (point yt-dlp `ydl_opts` at the installed Deno binary and permit official EJS solver retrieval) + `README.md` update + +**Verdict:** PASS + +### Findings + +No blocker, major, minor, or nit findings. + +### Verification + +**Steps performed:** +- Re-read `.ai/PLAN.md` (Phase 1 spec for T-005) and diffed it against `Dockerfile`, `app.py`, `README.md`, `ROADMAP.md`. The Dockerfile change (install `curl`/`unzip`, run the official Deno install script into `/usr/local`), the `ydl_opts` additions (`js_runtimes`, `remote_components`), and the README note all match the plan. +- `python3 -m py_compile app.py` — succeeded. +- `docker build -t yt-dl .` — succeeded (cached from the implementer's build, layers unchanged). +- Confirmed `shutil` is already imported at the top of `app.py` (used elsewhere for `rmtree`), so no missing-import risk from the new `shutil.which("deno")` call. +- **Verified the `js_runtimes`/`remote_components` `ydl_opts` shape against the actual installed yt-dlp version inside the built image** (yt-dlp `2026.08.19`), since the plan flagged this as needing confirmation at implementation time: + - Inspected `yt_dlp/YoutubeDL.py` inside the container: `self.params['js_runtimes']` is validated by `_clean_js_runtimes`, which requires exactly `dict[str, dict|None]` (`{"deno": {"path": ...}}`) — matches `app.py`'s shape precisely, including that `config.get('path')` is what's read internally. + - Inspected `yt_dlp/options.py`: `dest='js_runtimes'` and `dest='remote_components'` confirm these are the correct `ydl_opts` keys for the installed version. + - Constructed a `yt_dlp.YoutubeDL(...)` instance inside the container with the exact `ydl_opts` dict `app.py` builds — no warnings/errors, confirming the options are accepted and not silently ignored. +- Ran the container (`docker run --rm -d -p 8081:8080 ...`) and: + - `docker exec ... deno --version` → `deno 2.9.6` — binary present and runnable, on `PATH`. + - Live MP3 download of the previously-failing URL (`https://www.youtube.com/watch?v=7MrdyaSlOfI`) via the real job flow (`POST /download` → poll `GET /progress/{job_id}` → `GET /download/{job_id}/file`) — completed successfully, file delivered with the correct sanitized title-based filename via `Content-Disposition`. + - `docker logs` during that download showed `[youtube] [jsc:deno] Solving JS challenges using deno` and `[youtube] [jsc:deno] Downloading challenge solver lib script from https://github.com/yt-dlp/ejs/releases/...` — confirms Deno and the GitHub-hosted EJS solver are actively used, not just installed-but-unused. + - Full container log for both live downloads showed **no** `No supported JavaScript runtime could be found` warning anywhere. + - Regression check: re-ran a playlist-context URL (`watch?v=7MrdyaSlOfI&list=RDMM7MrdyaSlOfI`) through the same job flow — logs show only the single video's extraction/download (no playlist enumeration), progress percent/speed/downloaded/total advanced correctly through completion, and the delivered filename matched the sanitized video title — confirms T-003/T-004 behavior is unaffected. + - Stopped and removed the test container. +- Reviewed the `README.md` addition: accurately explains the Deno runtime dependency, the GitHub-hosted EJS remote component, why they're needed (JS-based signature/PO-token deciphering), and the GitHub reachability requirement. +- Reviewed the `ROADMAP.md` Priority 3 entry: accurately reflects the bug, root cause, and fix, consistent with what was implemented and verified. + +**Findings from verification:** All acceptance criteria for T-005 hold: +- `docker build` completes without errors after adding Deno. +- The previously-failing video URL now downloads successfully end-to-end. +- Server logs no longer show the missing-JS-runtime warning during a normal download. +- Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected. +- `python -m py_compile app.py` passes. + +**Risks:** +- The fix depends on outbound network access to `github.com` (for the EJS solver script) and `deno.land` (Dockerfile install-time only); if GitHub is unreachable from the container at runtime, the EJS-dependent extraction path could fail again — this is already called out in `README.md` as a requirement, so no action needed. +- `js_runtimes`/`remote_components` are relatively new, evolving yt-dlp options (confirmed only against the currently pinned `requirements.txt` range, `yt-dlp>=2024.4.9`, as resolved to `2026.08.19` in the built image); a future yt-dlp release changing this option's shape would need to be caught by re-running this same build+live-download validation, not by `py_compile` alone. diff --git a/.ai/TASKS.md b/.ai/TASKS.md index 2e90808..6968789 100644 --- a/.ai/TASKS.md +++ b/.ai/TASKS.md @@ -24,3 +24,4 @@ Command expectations: | T-002 | Dockerfile + README.md (build/run docs) | done | `docker build -t yt-dl .` exits 0; `docker run --rm -p 8080:8080 yt-dl` starts server; `curl http://localhost:8080/` returns HTML with download buttons | Reviewer re-ran `docker build` + container smoke test, plus a live end-to-end MP3 and video download against a real YouTube URL through the container — all passed; see `.ai/REVIEW.md` | none | | T-003 | `app.py` (`noplaylist` option, `sanitize_filename` helper, title-based `FileResponse` filename, frontend `Content-Disposition`-based `link.download` fix) + `README.md` update | done | Pasting a playlist/`list=` URL downloads only the referenced video; saved filename matches the video title with emoji/icons stripped (normal chars kept) instead of a UUID, with correct extension; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile`, unit-checked `sanitize_filename`, and ran a live end-to-end playlist-link MP3 download + a plain video download through the built Docker image against real YouTube URLs — all passed; see `.ai/REVIEW.md` | none | | T-004 | `app.py` (job-based `POST /download` + `GET /progress/{job_id}` + `GET /download/{job_id}/file`, in-memory job store, yt-dlp `progress_hooks`, frontend polling + progress bar UI) + `README.md` update | done | Progress bar appears below the buttons on download start and updates with percent/speed/downloaded-total size; file auto-delivers to the browser on completion with the correct sanitized filename; server-side failure shows an error in place of the bar with no crash; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile` and drove the full job lifecycle live through the built Docker image against real YouTube URLs (progress polling, auto-delivered file, 404/409 semantics, server-side error path with no crash) — all passed; see `.ai/REVIEW.md` | none | +| T-005 | `Dockerfile` (install Deno JS runtime) + `app.py` (point yt-dlp `ydl_opts` at the installed Deno binary and permit official EJS solver retrieval) + `README.md` update | done | `docker build` exits 0; `docker exec deno --version` succeeds; a video URL previously failing with `This video is not available` now downloads successfully; server logs no longer show `No supported JavaScript runtime could be found` during a download; playlist-single-video/progress-bar/filename behavior from T-003/T-004 is unaffected; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile`, `docker build`, verified the `js_runtimes`/`remote_components` `ydl_opts` shape against the installed yt-dlp source, and drove live MP3 downloads (previously-failing URL + playlist-link regression) through the built Docker image — no missing-runtime warning, EJS solver fetched and used, single-video/progress/filename behavior intact; see `.ai/REVIEW.md` | none | diff --git a/Dockerfile b/Dockerfile index 02d40af..fa793c0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,8 @@ FROM python:3.12-slim RUN apt-get update \ - && apt-get install -y --no-install-recommends ffmpeg \ + && apt-get install -y --no-install-recommends ffmpeg curl unzip \ + && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \ && rm -rf /var/lib/apt/lists/* WORKDIR /app diff --git a/README.md b/README.md index 3c48a75..b447576 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,14 @@ A small, self-hosted YouTube downloader with a browser interface. Paste a video docker build -t yt-dl . ``` +The image installs Deno as yt-dlp's JavaScript runtime. The application also +allows yt-dlp to fetch its EJS challenge-solver scripts from the official +yt-dlp GitHub repository when required. Together, these let yt-dlp perform the +JavaScript-based signature and PO-token deciphering required by current YouTube +extraction, avoiding false "This video is not available" failures for some +videos. The container must be able to reach GitHub during downloads that need +those scripts. + ### Run ```bash diff --git a/ROADMAP.md b/ROADMAP.md index c5f4572..e17b293 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -58,3 +58,24 @@ Objective: single-video-only downloads from playlist links, plus live download p - Playlist batch downloads (downloading every video in a playlist in one action). - User accounts / history. - Persisting job/progress state across server restarts. + +## Priority 3 + +Objective: fix downloads failing due to yt-dlp's missing JavaScript runtime. + +- Bug: some videos fail with `ERROR: [youtube] : This video is not available`, preceded by `WARNING: [youtube] No supported JavaScript runtime could be found. Only deno is enabled by default...`. yt-dlp falls back to an alternate player client (e.g. `visionos`) when it can't run JS-based signature/PO-token deciphering, and that fallback path incorrectly reports some videos as unavailable. +- Root cause: the Docker image installs no JS runtime, so yt-dlp can't perform the JS-dependent extraction steps modern YouTube extraction increasingly requires. +- Fix: install Deno (yt-dlp's documented lightweight JS runtime) in the `Dockerfile`, configure yt-dlp (via `ydl_opts` and/or a `--js-runtimes` equivalent) to use it, and allow retrieval of the official GitHub-hosted EJS challenge-solver scripts required by current yt-dlp releases, so normal extraction succeeds without falling back to a degraded client. +- Confirm the `WARNING: No supported JavaScript runtime could be found` message no longer appears in server logs during a download. + +## Acceptance Criteria (Priority 3) + +- `docker build` still completes without errors after adding Deno. +- Re-running a download of a video URL that previously failed with `This video is not available` (due to the missing JS runtime) succeeds and delivers the file. +- Server logs no longer show `No supported JavaScript runtime could be found` during a normal MP3 or Video download. +- Existing Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected. + +## Out of Scope + +- Fixing videos that are genuinely unavailable (age-restricted with no workaround, region-blocked, deleted, private) — this priority only addresses the missing-JS-runtime-induced false negative. +- General yt-dlp version upgrade policy beyond what's needed to pair with the new JS runtime. diff --git a/app.py b/app.py index 1d46e8f..e766cb1 100644 --- a/app.py +++ b/app.py @@ -318,6 +318,9 @@ async def download( "outtmpl": output_template, "noplaylist": True, "progress_hooks": [progress_hook], + "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}}, + # Current YouTube extraction may need yt-dlp's official EJS solver scripts. + "remote_components": ["ejs:github"], } if mode == "mp3": ydl_opts.update(