youtubedownloader/.ai/PLAN.md

Plan

Status: ready_for_implement

Goal: implement Priority 3 of ROADMAP.md — fix downloads failing with a false This video is not available error caused by yt-dlp having no JavaScript runtime available in the container.

Scope (Priority 3)

• T-005 — Install Deno as yt-dlp's JS runtime:
  • Add Deno (a single static binary, yt-dlp's documented lightweight JS runtime) to the Dockerfile so it's on PATH at container run time.
  • Point yt-dlp at it explicitly via ydl_opts (rather than relying on autodetection) so behavior doesn't silently regress if deno ever isn't found on PATH.
  • Allow yt-dlp to retrieve its official GitHub-hosted EJS challenge-solver scripts when current YouTube extraction requires them.
  • Verify server logs stop showing WARNING: ... No supported JavaScript runtime could be found during a normal download, and that a video URL exhibiting the This video is not available failure now succeeds.
  • Document the new build-time/runtime dependency and its purpose in README.md.

Acceptance Criteria

(from ROADMAP.md Priority 3)

• docker build still completes without errors after adding Deno.
• Re-running a download of a video URL that previously failed with This video is not available (due to the missing JS runtime) succeeds and delivers the file.
• Server logs no longer show No supported JavaScript runtime could be found during a normal MP3 or Video download.
• Existing Priority 1/2 behavior (single-video-from-playlist, progress bar, filename sanitization) is unaffected.

Implementation Phases

Phase 1 — T-005: Add Deno and wire it into yt-dlp

Files to change:

• Dockerfile
• app.py
• README.md (document the Deno dependency and why it's needed)

Changes in Dockerfile:

1. Install Deno in the build stage. Simplest reliable path for a python:3.12-slim (Debian) base without adding curl/unzip as extra layers if avoidable — use the official install script, which only needs curl and unzip:

   RUN apt-get update \
       && apt-get install -y --no-install-recommends ffmpeg curl unzip \
       && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \
       && rm -rf /var/lib/apt/lists/*
   

   This places the deno binary at /usr/local/bin/deno, already on PATH for subsequent RUN/CMD layers and for the app at runtime.

2. Keep the rest of the Dockerfile (WORKDIR, COPY requirements.txt, pip install, COPY app.py, EXPOSE, CMD) unchanged.

Changes in app.py:

1. In the shared ydl_opts construction (around line 317, alongside outtmpl/noplaylist/progress_hooks), explicitly point yt-dlp at the installed runtime instead of relying purely on PATH autodetection, so a missing/misconfigured binary fails loudly rather than silently degrading:

   ydl_opts: dict[str, object] = {
       "outtmpl": output_template,
       "noplaylist": True,
       "progress_hooks": [progress_hook],
       "js_runtimes": {"deno": {"path": shutil.which("deno") or "deno"}},
       "remote_components": ["ejs:github"],
   }
   

   (Exact option name/shape to be confirmed against the installed yt-dlp version's --js-runtimes support at implementation time — yt-dlp exposes this as a CLI flag --js-runtimes RUNTIME[:PATH]; the implementer should check yt_dlp.YoutubeDL / yt_dlp.options for the corresponding ydl_opts key in the pinned yt-dlp version and use that key, falling back to relying on autodetection via PATH only if no explicit option key exists in that version, in which case the Dockerfile's PATH install alone satisfies the requirement.)

2. Add remote_components: ["ejs:github"] to allow the EJS scripts needed by current yt-dlp releases to be fetched from the official yt-dlp GitHub repository.
3. Add import shutil if not already imported and used only for this lookup.

Validation:

• python -m py_compile app.py
• docker build -t yt-dl . — must exit 0.
• docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl, then docker exec yt-dl-test deno --version to confirm the binary is present and runnable in the final image.
• Live check: paste the video URL that previously failed (https://www.youtube.com/watch?v=7MrdyaSlOfI, without playlist params) through the running container for MP3 or Video mode, and confirm it downloads successfully instead of failing with This video is not available.
• Inspect docker logs yt-dl-test during that download and confirm the No supported JavaScript runtime could be found warning is no longer present.
• Regression spot-check: re-run one playlist-link download (&list=...) and confirm only the single video downloads, progress bar still updates, and the delivered filename still matches the sanitized video title (Priority 1/2 behavior unaffected).
• docker stop yt-dl-test

Documentation update (README.md):

• Add a short note under "Getting Started" / near the Build section stating that the image installs Deno and permits official GitHub-hosted EJS challenge-solver retrieval so yt-dlp can perform JS-based signature/PO-token deciphering required by current YouTube extraction, and that GitHub access is needed for downloads requiring those scripts.

Validation (Priority 3)

python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
docker exec yt-dl-test deno --version
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test

Live download checks (previously-failing video URL now succeeds, log warning gone, playlist-link/progress-bar/filename regressions) are performed manually/by the reviewer against real YouTube URLs, consistent with how prior tasks were validated — no new automated test framework is introduced.

---

Previous Plan (Priority 2 — completed, kept for reference)

Goal: implement Priority 2 of ROADMAP.md — single-video-only downloads from playlist links, correctly-named output files, and a live progress bar in the browser UI.

Scope

• T-003 — Single video from playlist links + correct filenames (no architecture change; still one synchronous POST /download request/response):
  • Pass yt-dlp's noplaylist: True so a playlist URL or a watch?v=...&list=... URL only ever downloads the one referenced video.
  • Add a sanitize_filename(title, fallback_id) helper: strip characters outside letters (any language)/digits/spaces/- _ . ( ) , ', collapse resulting whitespace, trim, and fall back to the video id if nothing usable remains.
  • Extract title/id from yt-dlp's result info dict and pass the sanitized name explicitly as FileResponse(..., filename=...), decoupled from whatever yt-dlp happens to name the file on disk.
  • Fix the real UUID-looking-filename bug: response.blob() in the frontend JS strips the Content-Disposition header, so link.download = '' makes the browser invent a blob-id-like name. Read the filename from the Content-Disposition response header in JS and set link.download to it explicitly (fallback to a generic name only if the header is somehow missing).
• T-004 — Background job + live progress bar (architecture change, builds on T-003):
  • Split POST /download into three endpoints:
    • POST /download — validates the URL, creates an in-memory job record (job_id = uuid4(), status pending), schedules the actual yt-dlp run on a background thread (asyncio.get_running_loop().run_in_executor), and returns {"job_id": ...} immediately (no file streaming here anymore).
    • GET /progress/{job_id} — returns the job's current {status, percent, speed, downloaded_bytes, total_bytes, filename, error} from the in-memory store. status is one of downloading, finished, error.
    • GET /download/{job_id}/file — once status == finished, streams the prepared file via FileResponse (reusing T-003's sanitized filename) with BackgroundTask cleanup of the job's temp dir and job record; returns 409/404 if called before completion or after cleanup.
  • Progress capture: a yt-dlp progress_hooks callback updates the job's dict on each tick using downloaded_bytes, total_bytes (or total_bytes_estimate), speed, and computed percent; a postprocessor_hooks callback (or the finished hook state) marks the job finished once the final file is ready.
  • Job store: a plain in-memory dict[str, JobState] guarded by a threading.Lock (single-process app, matches "no persistent storage" constraint). Entries are removed after the file is served, and a periodic/best-effort cleanup drops stale jobs (e.g. no progress endpoint hit within N minutes) so a browser tab closed mid-download doesn't leak a temp dir forever.
  • Frontend JS rework:
    • downloadFile(mode) now: POSTs to /download to obtain job_id, shows the progress bar area, then polls GET /progress/{job_id} every 500ms.
    • On each poll: update a <progress> bar (or styled div) with percent, and a text line with speed (human-readable, e.g. 1.2 MB/s) and downloaded / total size (human-readable bytes).
    • On status == finished: stop polling, fetch GET /download/{job_id}/file, read filename from Content-Disposition (per T-003's fix), trigger the save via the existing blob+anchor pattern, then hide the progress bar and clear status text.
    • On status == error: stop polling, show the error message in the existing #status div, hide the progress bar.
  • HTML/CSS: add a progress bar element (native <progress> plus a small text line for speed/size) directly below the two buttons, hidden by default, shown only while a job is active.

Acceptance Criteria

(from ROADMAP.md Priority 2)

• Pasting a playlist URL (or a watch URL with a list= param) and clicking MP3 or Video downloads only that one video, never the rest of the playlist.
• Starting a download shows a progress bar below the buttons that updates with percentage, speed, and downloaded/total size while the server is processing.
• When the download finishes, the file is automatically delivered to the browser (save dialog) without an extra click.
• The saved file's name matches the video's title (emoji/icons/symbols removed, normal characters kept) instead of a UUID or other opaque id, and still ends in the correct extension.
• If the download fails server-side, the UI shows a clear error in place of the progress bar (no server crash, no stuck spinner).

Implementation Phases

Phase 1 — T-003: Single video from playlists + correct filenames

Files to change:

• app.py
• README.md (document: playlist links only download the referenced video; filenames follow the video title with icons/emoji stripped)

Changes in app.py:

1. Add ydl_opts["noplaylist"] = True for both mp3 and video modes.
2. Add a module-level helper:

   import re
   
   _FILENAME_ALLOWED = re.compile(r"[^\w\s.,'()\-]", re.UNICODE)
   _WHITESPACE = re.compile(r"\s+")
   
   def sanitize_filename(title: str, fallback_id: str) -> str:
       cleaned = _FILENAME_ALLOWED.sub("", title or "")
       cleaned = _WHITESPACE.sub(" ", cleaned).strip()
       return cleaned or fallback_id
   

   (\w under re.UNICODE, the Python 3 default, matches any-language letters/digits/underscore, so non-Latin titles are preserved; emoji/symbols/pictographs fall outside \w, whitespace, and the explicit punctuation set and are dropped.)

3. In fetch_media, call downloader.extract_info(url, download=True) (instead of .download([url])) to get the info dict back; read info.get("title") and info.get("id"), compute display_name = sanitize_filename(title, id) + Path(media_path).suffix.
4. Pass filename=display_name to the existing FileResponse(...) call instead of media_path.name.
5. In the frontend <script>, change the success branch of downloadFile:

   const disposition = response.headers.get('content-disposition') || '';
   const match = /filename\*?=(?:UTF-8''|")?([^\";]+)/i.exec(disposition);
   const filename = match ? decodeURIComponent(match[1].replace(/"/g, '')) : 'download';
   ...
   link.download = filename;
   

   Keep the rest of the blob/anchor logic unchanged.

Validation:

• python -m py_compile app.py
• Unit-style check (via a small mocked test or manual python -c) that sanitize_filename("Song 🔥 Title ✨ (Live)", "abc123") == "Song Title (Live)" collapsed to single spaces → "Song Title (Live)", and sanitize_filename("🔥🔥🔥", "abc123") == "abc123".
• Manual/live check: paste a watch?v=...&list=... URL and confirm only the single video downloads; confirm the saved filename in the browser matches the (emoji-stripped) video title, not a UUID.

Phase 2 — T-004: Background job + live progress bar

Files to change:

• app.py
• README.md (document the new /download, /progress/{job_id}, /download/{job_id}/file flow and the progress bar UI; note job state is in-memory/non-persistent)

Backend changes in app.py:

1. Add a JobState structure (dataclass or TypedDict) with fields: status ("downloading" | "finished" | "error"), percent, speed, downloaded_bytes, total_bytes, temp_dir, file_path, filename, error_message, mode, created_at.
2. Add a module-level _jobs: dict[str, JobState] and _jobs_lock = threading.Lock().
3. POST /download (url, mode form fields, same validation as today):
   • Validate URL as today (raise HTTPException(400) on invalid/empty).
   • Create job_id = str(uuid4()), create the temp dir, seed a JobState(status="downloading", ...) in _jobs under the lock.
   • Define a progress_hook(d) closure that, under the lock, updates the job's percent/speed/downloaded_bytes/total_bytes from d["status"], d.get("downloaded_bytes"), d.get("total_bytes") or d.get("total_bytes_estimate"), d.get("speed"); on d["status"] == "error" marks the job error.
   • Build ydl_opts same as T-003 (noplaylist: True, plus progress_hooks: [progress_hook]).
   • Define run_job() that calls extract_info(url, download=True), computes display_name via sanitize_filename (from T-003), locates the produced file, and updates the job to status="finished" with file_path/filename set; wraps in try/except to set status="error"/error_message on failure, and cleans up the temp dir on error (not on success — success cleanup happens after the file is served).
   • Schedule run_job via asyncio.get_running_loop().run_in_executor(None, run_job) without awaiting it (fire-and-forget task kept referenced, e.g. appended to a module-level set of pending futures to avoid GC warnings).
   • Return JSONResponse({"job_id": job_id}).
4. GET /progress/{job_id}:
   • Look up the job under the lock; 404 (JSON error) if unknown.
   • Return {"status", "percent", "speed", "downloaded_bytes", "total_bytes", "error"} (omit file path from the response).
5. GET /download/{job_id}/file:
   • Look up the job; 404 if unknown, 409 (JSON error) if not yet finished.
   • Return FileResponse(job.file_path, filename=job.filename, background=BackgroundTask(cleanup_job, job_id)) where cleanup_job removes the temp dir and pops the job from _jobs under the lock.
6. Best-effort stale-job sweep: a small helper (called opportunistically, e.g. at the top of POST /download) that drops any job older than a fixed TTL (e.g. 30 minutes) whose temp dir hasn't been claimed, cleaning up its temp dir — keeps the in-memory store from growing unbounded if a browser tab is closed mid-download. No background scheduler/thread is introduced solely for this; it piggybacks on request handling to keep the app dependency-free.

Frontend changes (inline HTML/JS in app.py):

1. Add markup below the buttons:

   <div id="progress-wrap" hidden>
     <progress id="progress-bar" max="100" value="0"></progress>
     <div id="progress-text"></div>
   </div>
   

2. Rework downloadFile(mode):
   • POST to /download, parse {job_id} (existing error handling for non-2xx/JSON stays the same shape as today).
   • Show #progress-wrap, start setInterval polling GET /progress/{job_id} every 500ms.
   • On each tick: update #progress-bar.value from percent; update #progress-text with a human-readable line, e.g. formatSpeed(speed) + ' · ' + formatBytes(downloaded_bytes) + ' / ' + formatBytes(total_bytes).
   • Add small formatBytes(n) / formatSpeed(n) helpers (KB/MB/GB, /s suffix), handling null/unknown total gracefully (e.g. show downloaded size only if total is unknown).
   • On status === 'finished': clearInterval, fetch GET /download/{job_id}/file, apply the same Content-Disposition-based filename logic from T-003, trigger the blob/anchor save, then hide #progress-wrap and reset text/bar.
   • On status === 'error': clearInterval, hide #progress-wrap, show error (or a generic message) in #status.
   • Guard against overlapping downloads from double-clicks (e.g. disable buttons while a job is active, re-enable on finish/error).

Validation:

• python -m py_compile app.py
• Manual/live check via docker run (or local uvicorn): start an MP3 download, observe the progress bar advancing with percent/speed/size, confirm the file auto-downloads on completion with the correct sanitized title-based filename; repeat for Video mode.
• Manual check: trigger a server-side failure path (e.g. malformed-but-http-prefixed URL) and confirm the progress bar disappears and the existing #status error text appears instead, with no server crash.
• Manual check: confirm a playlist URL still only fetches the single video (retained from T-003) end-to-end through the new job flow.

Validation (both phases)

python -m py_compile app.py
docker build -t yt-dl .
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test

Live download checks (MP3, Video, playlist-link-single-video, progress bar behavior, filename correctness, error path) are performed manually/by the reviewer against a real YouTube URL, consistent with how T-001/T-002 were validated — no new automated test framework is introduced.