youtubedownloader/.ai/PLAN.md

Plan

Status: ready_for_implement

Goal: implement Priority 4 (player-client fallback for PO Token / 429 failures) and Priority 5 (720p video download option) of ROADMAP.md.

Scope (Priority 4) — T-006: Prefer non-web player clients

• Problem: after T-005, some downloads now fail via a different path: HTTP Error 429: Too Many Requests fetching the webpage, then Unable to fetch GVS PO Token for web client: Missing required Visitor Data, ending in This video is not available. The web client increasingly requires a PO Token yt-dlp can't generate, and is more exposed to IP-based rate limiting.
• Fix: add extractor_args: {"youtube": {"player_client": ["android", "ios", "tv", "web"]}} (or the current yt-dlp-recommended ordering) to the shared ydl_opts in app.py (same dict built at line ~317, alongside js_runtimes/remote_components), so yt-dlp tries clients that don't strictly require a PO Token first, and only falls back to web last.
• This applies uniformly to both mp3 and video (and the new video720 from Priority 5) modes since it's set on the shared ydl_opts before the per-mode format/postprocessors update.
• No new error-handling paths are needed: if every client fails, the existing run_job try/except already marks the job status="error" and the UI already surfaces that.

Files to change (T-006)

• app.py — add the extractor_args player_client ordering to the shared ydl_opts dict.
• README.md — document that the app tries multiple YouTube player clients (android, ios, tv, then web) to avoid PO-Token/rate-limit failures on the web client, and that this is a best-effort mitigation that may need retuning as YouTube changes its anti-bot behavior.

Validation (T-006)

• python -m py_compile app.py
• docker build -t yt-dl . — must exit 0.
• Live check: re-run a download of a video URL that previously failed with Missing required Visitor Data / This video is not available (e.g. the reported LV-NXucnyrc case) through the running container, and confirm it now succeeds.
• Inspect docker logs during that download and confirm an alternate client (android/ios/tv) is used and the job completes without the PO-Token warning blocking it.
• Regression spot-check: re-run one plain MP3 download and one plain Video download (non-playlist) and confirm they still succeed with progress bar + correct filename, unaffected by the client-ordering change.

Scope (Priority 5) — T-007: Add a "Download Video 720p" option

• Add a third mode, video720, alongside the existing mp3/video modes, both server-side and in the UI.
• Server-side (app.py):
  • Extend the mode type from Literal["mp3", "video"] to Literal["mp3", "video", "video720"] everywhere it's declared: the /download route's Form(...) parameter (line ~280) and JobState.mode (line ~216).
  • In the if mode == "mp3": ... else: ... branch (line ~325 onward) that sets format/merge_output_format, add a branch for video720:

    elif mode == "video720":
        ydl_opts.update(
            {
                "format": "bestvideo[height<=720]+bestaudio/best[height<=720]",
                "merge_output_format": "mp4",
            }
        )
    else:  # "video"
        ydl_opts.update({"format": "bestvideo+bestaudio/best", "merge_output_format": "mp4"})
    

    This caps the video stream at 720p while still falling back to the best available quality below 720p if that's all that exists (yt-dlp's format selector already does this — no extra fallback logic needed).

  • No changes needed to sanitize_filename, the job store, progress_hook, or the /progress//download/{job_id}/file endpoints — they're mode-agnostic already.
• Frontend (inline HTML/JS in app.py):
  • Add a third button next to the existing two (line ~49): <button type="button" onclick="downloadFile('video720')">Download Video 720p</button>.
  • downloadFile(mode) already takes mode as a parameter and POSTs it through unchanged — confirm no mode-specific branching exists in the JS that would need a new case (per the current implementation, the JS is generic over mode, so this should be a markup-only change).

Files to change (T-007)

• app.py — extend the mode Literal type (route param + JobState), add the video720 format-selection branch, add the third HTML button.
• README.md — document the new Download Video 720p button: caps video quality at 720p, falls back to the best available quality if the source is below 720p, and behaves like the existing Video button otherwise (progress bar, auto-delivery, sanitized filename).

Validation (T-007)

• python -m py_compile app.py
• docker build -t yt-dl . — must exit 0.
• Live check: click Download Video 720p on a video known to have >720p streams available; inspect the delivered file's video resolution (e.g. via ffprobe) and confirm it is ≤720p.
• Live check: click Download Video 720p on a video whose best quality is below 720p (or force via a low-quality test video) and confirm the download still succeeds with no error.
• Regression spot-check: confirm Download MP3 and Download Video (unrestricted) still behave exactly as before — correct format/quality, progress bar, filename, playlist-single-video behavior.

Acceptance Criteria

(from ROADMAP.md Priority 4 and Priority 5 — see those sections for full text)

• Priority 4: a previously PO-Token/429-failing video now downloads via an alternate player client; existing behavior (playlist/progress/filename/Deno) unaffected; failures still surface a clean UI error with no crash.
• Priority 5: a third "Download Video 720p" button exists; it caps video quality at 720p with graceful fallback when the source is lower quality; progress bar, auto-delivery, filename sanitization, and existing MP3/Video buttons are unaffected.

Implementation Order

1. T-006 first (player-client fallback) — it's an independent, small ydl_opts change that other live-download validation (including T-007's) benefits from being in place first, since it reduces the chance of unrelated PO-Token failures muddying 720p testing.
2. T-007 second (720p mode) — builds on the same ydl_opts dict shape.

Validation (both tasks, combined)

python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download Video 720p"
docker stop yt-dl-test

Live download checks (PO-Token/429 recovery, 720p resolution cap + fallback, MP3/Video regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced.

---

Previous Plan (Priority 3 — completed, kept for reference)

Goal: implement Priority 3 of ROADMAP.md — fix downloads failing with a false This video is not available error caused by yt-dlp having no JavaScript runtime available in the container.

Scope (Priority 3)

• T-005 — Install Deno as yt-dlp's JS runtime:
  • Add Deno (a single static binary, yt-dlp's documented lightweight JS runtime) to the Dockerfile so it's on PATH at container run time.
  • Point yt-dlp at it explicitly via ydl_opts (rather than relying on autodetection) so behavior doesn't silently regress if deno ever isn't found on PATH.
  • Allow yt-dlp to retrieve its official GitHub-hosted EJS challenge-solver scripts when current YouTube extraction requires them.
  • Verify server logs stop showing WARNING: ... No supported JavaScript runtime could be found during a normal download, and that a video URL exhibiting the This video is not available failure now succeeds.
  • Document the new build-time/runtime dependency and its purpose in README.md.

Acceptance Criteria

(from ROADMAP.md Priority 3)

• docker build still completes without errors after adding Deno.
• Re-running a download of a video URL that previously failed with This video is not available (due to the missing JS runtime) succeeds and delivers the file.
• Server logs no longer show No supported JavaScript runtime could be found during a normal MP3 or Video download.
• Existing Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected.

Implementation Phases

Phase 1 — T-005: Add Deno and wire it into yt-dlp

Files to change:

• Dockerfile
• app.py
• README.md (document the Deno dependency and why it's needed)

Changes in Dockerfile:

1. Install Deno in the build stage. Simplest reliable path for a python:3.12-slim (Debian) base without adding curl/unzip as extra layers if avoidable — use the official install script, which only needs curl and unzip:

   RUN apt-get update \
       && apt-get install -y --no-install-recommends ffmpeg curl unzip \
       && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \
       && rm -rf /var/lib/apt/lists/*
   

   This places the deno binary at /usr/local/bin/deno, already on PATH for subsequent RUN/CMD layers and for the app at runtime.

2. Keep the rest of the Dockerfile (WORKDIR, COPY requirements.txt, pip install, COPY app.py, EXPOSE, CMD) unchanged.

Changes in app.py:

1. In the shared ydl_opts construction (around line 317, alongside outtmpl/noplaylist/progress_hooks), explicitly point yt-dlp at the installed runtime instead of relying purely on PATH autodetection, so a missing/misconfigured binary fails loudly rather than silently degrading:

   ydl_opts: dict[str, object] = {
       "outtmpl": output_template,
       "noplaylist": True,
       "progress_hooks": [progress_hook],
       "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}},
       "remote_components": ["ejs:github"],
   }
   

   (Exact option name/shape to be confirmed against the installed yt-dlp version's --js-runtimes support at implementation time — yt-dlp exposes this as a CLI flag --js-runtimes RUNTIME[:PATH]; the implementer should check yt_dlp.YoutubeDL / yt_dlp.options for the corresponding ydl_opts key in the pinned yt-dlp version and use that key, falling back to relying on autodetection via PATH only if no explicit option key exists in that version, in which case the Dockerfile's PATH install alone satisfies the requirement.)

2. Add remote_components: ["ejs:github"] to allow the EJS scripts needed by current yt-dlp releases to be fetched from the official yt-dlp GitHub repository.
3. Add import shutil if not already imported and used only for this lookup.

Validation:

• python -m py_compile app.py
• docker build -t yt-dl . — must exit 0.
• docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl, then docker exec yt-dl-test deno --version to confirm the binary is present and runnable in the final image.
• Live check: paste the video URL that previously failed (https://www.youtube.com/watch?v=7MrdyaSlOfI, without playlist params) through the running container for MP3 or Video mode, and confirm it downloads successfully instead of failing with This video is not available.
• Inspect docker logs yt-dl-test during that download and confirm the No supported JavaScript runtime could be found warning is no longer present.
• Regression spot-check: re-run one playlist-link download (&list=...) and confirm only the single video downloads, progress bar still updates, and the delivered filename still matches the sanitized video title (Priority 1/2 behavior unaffected).
• docker stop yt-dl-test

Documentation update (README.md):

• Add a short note under "Getting Started" / near the Build section stating that the image installs Deno and permits official GitHub-hosted EJS challenge-solver retrieval so yt-dlp can perform JS-based signature/PO-token deciphering required by current YouTube extraction, and that GitHub access is needed for downloads requiring those scripts.

Validation (Priority 3)

python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
docker exec yt-dl-test deno --version
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test

Live download checks (previously-failing video URL now succeeds, log warning gone, playlist-link/progress-bar/filename regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced.

---

Previous Plan (Priority 2 — completed, kept for reference)

Goal: implement Priority 2 of ROADMAP.md — single-video-only downloads from playlist links, correctly-named output files, and a live progress bar in the browser UI.

Scope

• T-003 — Single video from playlist links + correct filenames (no architecture change; still one synchronous POST /download request/response):
  • Pass yt-dlp's noplaylist: True so a playlist URL or a watch?v=...&list=... URL only ever downloads the one referenced video.
  • Add a sanitize_filename(title, fallback_id) helper: strip characters outside letters (any language)/digits/spaces/- _ . ( ) , ', collapse resulting whitespace, trim, and fall back to the video id if nothing usable remains.
  • Extract title/id from yt-dlp's result info dict and pass the sanitized name explicitly as FileResponse(..., filename=...), decoupled from whatever yt-dlp happens to name the file on disk.
  • Fix the real UUID-looking-filename bug: response.blob() in the frontend JS strips the Content-Disposition header, so link.download = '' makes the browser invent a blob-id-like name. Read the filename from the Content-Disposition response header in JS and set link.download to it explicitly (fallback to a generic name only if the header is somehow missing).
• T-004 — Background job + live progress bar (architecture change, builds on T-003):
  • Split POST /download into three endpoints:
    • POST /download — validates the URL, creates an in-memory job record (job_id = uuid4(), status pending), schedules the actual yt-dlp run on a background thread (asyncio.get_running_loop().run_in_executor), and returns {"job_id": ...} immediately (no file streaming here anymore).
    • GET /progress/{job_id} — returns the job's current {status, percent, speed, downloaded_bytes, total_bytes, filename, error} from the in-memory store. status is one of downloading, finished, error.
    • GET /download/{job_id}/file — once status == finished, streams the prepared file via FileResponse (reusing T-003's sanitized filename) with BackgroundTask cleanup of the job's temp dir and job record; returns 409/404 if called before completion or after cleanup.
  • Progress capture: a yt-dlp progress_hooks callback updates the job's dict on each tick using downloaded_bytes, total_bytes (or total_bytes_estimate), speed, and computed percent; a postprocessor_hooks callback (or the finished hook state) marks the job finished once the final file is ready.
  • Job store: a plain in-memory dict[str, JobState] guarded by a threading.Lock (single-process app, matches "no persistent storage" constraint). Entries are removed after the file is served, and a periodic/best-effort cleanup drops stale jobs (e.g. no progress endpoint hit within N minutes) so a browser tab closed mid-download doesn't leak a temp dir forever.
  • Frontend JS rework:
    • downloadFile(mode) now: POSTs to /download to obtain job_id, shows the progress bar area, then polls GET /progress/{job_id} every 500ms.
    • On each poll: update a <progress> bar (or styled div) with percent, and a text line with speed (human-readable, e.g. 1.2 MB/s) and downloaded / total size (human-readable bytes).
    • On status == finished: stop polling, fetch GET /download/{job_id}/file, read filename from Content-Disposition (per T-003's fix), trigger the save via the existing blob+anchor pattern, then hide the progress bar and clear status text.
    • On status == error: stop polling, show the error message in the existing #status div, hide the progress bar.
  • HTML/CSS: add a progress bar element (native <progress> plus a small text line for speed/size) directly below the two buttons, hidden by default, shown only while a job is active.

Acceptance Criteria

(from ROADMAP.md Priority 2)

• Pasting a playlist URL (or a watch URL with a list= param) and clicking MP3 or Video downloads only that one video, never the rest of the playlist.
• Starting a download shows a progress bar below the buttons that updates with percentage, speed, and downloaded/total size while the server is processing.
• When the download finishes, the file is automatically delivered to the browser (save dialog) without an extra click.
• The saved file's name matches the video's title (emoji/icons/symbols removed, normal characters kept) instead of a UUID or other opaque id, and still ends in the correct extension.
• If the download fails server-side, the UI shows a clear error in place of the progress bar (no server crash, no stuck spinner).

Implementation Phases

Phase 1 — T-003: Single video from playlists + correct filenames

Files to change:

• app.py
• README.md (document: playlist links only download the referenced video; filenames follow the video title with icons/emoji stripped)

Changes in app.py:

1. Add ydl_opts["noplaylist"] = True for both mp3 and video modes.
2. Add a module-level helper:

   import re
   
   _FILENAME_ALLOWED = re.compile(r"[^\w\s.,'()\-]", re.UNICODE)
   _WHITESPACE = re.compile(r"\s+")
   
   def sanitize_filename(title: str, fallback_id: str) -> str:
       cleaned = _FILENAME_ALLOWED.sub("", title or "")
       cleaned = _WHITESPACE.sub(" ", cleaned).strip()
       return cleaned or fallback_id
   

   (\w under re.UNICODE, the Python 3 default, matches any-language letters/digits/underscore, so non-Latin titles are preserved; emoji/symbols/pictographs fall outside \w, whitespace, and the explicit punctuation set and are dropped.)

3. In fetch_media, call downloader.extract_info(url, download=True) (instead of .download([url])) to get the info dict back; read info.get("title") and info.get("id"), compute display_name = sanitize_filename(title, id) + Path(media_path).suffix.
4. Pass filename=display_name to the existing FileResponse(...) call instead of media_path.name.
5. In the frontend <script>, change the success branch of downloadFile:

   const disposition = response.headers.get('content-disposition') || '';
   const match = /filename\*?=(?:UTF-8''|")?([^\";]+)/i.exec(disposition);
   const filename = match ? decodeURIComponent(match[1].replace(/"/g, '')) : 'download';
   ...
   link.download = filename;
   

   Keep the rest of the blob/anchor logic unchanged.

Validation:

• python -m py_compile app.py
• Unit-style check (via a small mocked test or manual python -c) that sanitize_filename("Song 🔥 Title ✨ (Live)", "abc123") == "Song Title (Live)" collapsed to single spaces → "Song Title (Live)", and sanitize_filename("🔥🔥🔥", "abc123") == "abc123".
• Manual/live check: paste a watch?v=...&list=... URL and confirm only the single video downloads; confirm the saved filename in the browser matches the (emoji-stripped) video title, not a UUID.

Phase 2 — T-004: Background job + live progress bar

Files to change:

• app.py
• README.md (document the new /download, /progress/{job_id}, /download/{job_id}/file flow and the progress bar UI; note job state is in-memory/non-persistent)

Backend changes in app.py:

1. Add a JobState structure (dataclass or TypedDict) with fields: status ("downloading" | "finished" | "error"), percent, speed, downloaded_bytes, total_bytes, temp_dir, file_path, filename, error_message, mode, created_at.
2. Add a module-level _jobs: dict[str, JobState] and _jobs_lock = threading.Lock().
3. POST /download (url, mode form fields, same validation as today):
   • Validate URL as today (raise HTTPException(400) on invalid/empty).
   • Create job_id = str(uuid4()), create the temp dir, seed a JobState(status="downloading", ...) in _jobs under the lock.
   • Define a progress_hook(d) closure that, under the lock, updates the job's percent/speed/downloaded_bytes/total_bytes from d["status"], d.get("downloaded_bytes"), d.get("total_bytes") or d.get("total_bytes_estimate"), d.get("speed"); on d["status"] == "error" marks the job error.
   • Build ydl_opts same as T-003 (noplaylist: True, plus progress_hooks: [progress_hook]).
   • Define run_job() that calls extract_info(url, download=True), computes display_name via sanitize_filename (from T-003), locates the produced file, and updates the job to status="finished" with file_path/filename set; wraps in try/except to set status="error"/error_message on failure, and cleans up the temp dir on error (not on success — success cleanup happens after the file is served).
   • Schedule run_job via asyncio.get_running_loop().run_in_executor(None, run_job) without awaiting it (fire-and-forget task kept referenced, e.g. appended to a module-level set of pending futures to avoid GC warnings).
   • Return JSONResponse({"job_id": job_id}).
4. GET /progress/{job_id}:
   • Look up the job under the lock; 404 (JSON error) if unknown.
   • Return {"status", "percent", "speed", "downloaded_bytes", "total_bytes", "error"} (omit file path from the response).
5. GET /download/{job_id}/file:
   • Look up the job; 404 if unknown, 409 (JSON error) if not yet finished.
   • Return FileResponse(job.file_path, filename=job.filename, background=BackgroundTask(cleanup_job, job_id)) where cleanup_job removes the temp dir and pops the job from _jobs under the lock.
6. Best-effort stale-job sweep: a small helper (called opportunistically, e.g. at the top of POST /download) that drops any job older than a fixed TTL (e.g. 30 minutes) whose temp dir hasn't been claimed, cleaning up its temp dir — keeps the in-memory store from growing unbounded if a browser tab is closed mid-download. No background scheduler/thread is introduced solely for this; it piggybacks on request handling to keep the app dependency-free.

Frontend changes (inline HTML/JS in app.py):

1. Add markup below the buttons:

   <div id="progress-wrap" hidden>
     <progress id="progress-bar" max="100" value="0"></progress>
     <div id="progress-text"></div>
   </div>
   

2. Rework downloadFile(mode):
   • POST to /download, parse {job_id} (existing error handling for non-2xx/JSON stays the same shape as today).
   • Show #progress-wrap, start setInterval polling GET /progress/{job_id} every 500ms.
   • On each tick: update #progress-bar.value from percent; update #progress-text with a human-readable line, e.g. formatSpeed(speed) + ' · ' + formatBytes(downloaded_bytes) + ' / ' + formatBytes(total_bytes).
   • Add small formatBytes(n) / formatSpeed(n) helpers (KB/MB/GB, /s suffix), handling null/unknown total gracefully (e.g. show downloaded size only if total is unknown).
   • On status === 'finished': clearInterval, fetch GET /download/{job_id}/file, apply the same Content-Disposition-based filename logic from T-003, trigger the blob/anchor save, then hide #progress-wrap and reset text/bar.
   • On status === 'error': clearInterval, hide #progress-wrap, show error (or a generic message) in #status.
   • Guard against overlapping downloads from double-clicks (e.g. disable buttons while a job is active, re-enable on finish/error).

Validation:

• python -m py_compile app.py
• Manual/live check via docker run (or local uvicorn): start an MP3 download, observe the progress bar advancing with percent/speed/size, confirm the file auto-downloads on completion with the correct sanitized title-based filename; repeat for Video mode.
• Manual check: trigger a server-side failure path (e.g. malformed-but-http-prefixed URL) and confirm the progress bar disappears and the existing #status error text appears instead, with no server crash.
• Manual check: confirm a playlist URL still only fetches the single video (retained from T-003) end-to-end through the new job flow.

Validation (both phases)

python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test

Live download checks (MP3, Video, playlist-link-single-video, progress bar behavior, filename correctness, error path) are performed manually/by the reviewer against a real YouTube URL, consistent with how T-001/T-002 were validated — no new automated test framework is introduced.