Compare commits

...

2 Commits

Author SHA1 Message Date
Domagoj Zecevic 275b622243 feat(downloads): show live download progress 2 weeks ago
Domagoj Zecevic 2b06f1cf05 feat(downloads): keep playlist links to one video 2 weeks ago
  1. 89
      .ai/HANDOFF.md
  2. 203
      .ai/PLAN.md
  3. 79
      .ai/REVIEW.md
  4. 2
      .ai/TASKS.md
  5. 6
      README.md
  6. 37
      ROADMAP.md
  7. 300
      app.py

89
.ai/HANDOFF.md

@ -2,6 +2,17 @@
Append-only role handoff log. Each role adds one entry when its step is complete.
### T-003,T-004 — plan — 2026-09-07T10:14:33Z
| Field | Value |
|-------|-------|
| Agent | claude |
| Summary | Wrote Priority 2 implementation plan: T-003 (noplaylist option, title-based sanitized filenames, fix for the UUID-looking download name caused by `response.blob()` dropping `Content-Disposition`) and T-004 (job-based `/download` + `/progress/{job_id}` + `/download/{job_id}/file` with an in-memory job store and yt-dlp `progress_hooks`, plus a polling-driven progress bar UI); both tasks moved to ready_for_implement |
| Files Changed | ROADMAP.md, .ai/PLAN.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Next Role | implement |
---
### T-001,T-002 — plan — 2026-09-05T00:00:00Z
| Field | Value |
@ -90,3 +101,81 @@ Append-only role handoff log. Each role adds one entry when its step is complete
| Next Role | none |
---
### T-003 — implement — 2026-09-07T11:27:56Z
| Field | Value |
|-------|-------|
| Agent | codex |
| Summary | Limited playlist-context links to their referenced video and delivered sanitized title-based download filenames. |
| Files Changed | app.py, README.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Validation | `python3 -m py_compile app.py` passed (`python` is unavailable); mocked container route check for `noplaylist=True` and `Song Title (Live).mp3` passed; `docker build -t yt-dl .`, container homepage smoke test, and `git diff --check` passed |
| Commit | Pending reviewer approval |
| Next Role | review |
---
### T-003 — review — 2026-09-07T12:45:00Z
| Field | Value |
|-------|-------|
| Agent | claude |
| Summary | Reviewed `noplaylist`/filename-sanitization changes against `.ai/PLAN.md`; re-ran `py_compile`, unit-checked `sanitize_filename`, and ran a live end-to-end playlist-link MP3 download plus a plain video download through the built Docker image against real YouTube URLs — all passed. |
| Files Changed | .ai/REVIEW.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Verdict | PASS |
| Blocking Findings | none |
| Next Role | implement |
---
### T-003 — implement — 2026-09-07T12:48:26Z
| Field | Value |
|-------|-------|
| Agent | codex |
| Summary | Squashed the reviewed playlist-safe, title-based download changes into the task commit. |
| Files Changed | app.py, README.md, ROADMAP.md, .ai/PLAN.md, .ai/REVIEW.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Validation | Reviewer verification passed; `git diff --check` passed |
| Commit | Created by this handoff's task commit |
| Next Role | none |
---
### T-004 — implement — 2026-09-07T12:56:52Z
| Field | Value |
|-------|-------|
| Agent | codex |
| Summary | Added in-memory background download jobs, live polling progress, automatic file delivery, and cleanup. |
| Files Changed | app.py, README.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Validation | `python3 -m py_compile app.py` passed (`python` is unavailable); mocked container job lifecycle check for progress, delivery cleanup, and error state passed; `docker build -t yt-dl .`, container progress UI smoke test, and `git diff --check` passed |
| Commit | Pending reviewer approval |
| Next Role | review |
---
### T-004 — review — 2026-09-07T13:10:00Z
| Field | Value |
|-------|-------|
| Agent | claude |
| Summary | Reviewed the job-based download/progress/delivery split against `.ai/PLAN.md`; re-ran `py_compile` and drove the full job lifecycle live through the built Docker image against real YouTube URLs (progress polling, auto-delivered file, 404/409 semantics, server-side error path with no crash) — all passed with 3 non-required minor/nit notes. |
| Files Changed | .ai/REVIEW.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Verdict | PASS_WITH_NOTES |
| Blocking Findings | none |
| Next Role | implement |
---
### T-004 — implement — 2026-09-07T13:06:07Z
| Field | Value |
|-------|-------|
| Agent | codex |
| Summary | Squashed the reviewed background-job download progress experience into the task commit. |
| Files Changed | app.py, README.md, .ai/REVIEW.md, .ai/TASKS.md, .ai/HANDOFF.md |
| Validation | Reviewer verification passed; `git diff --check` passed |
| Commit | Created by this handoff's task commit |
| Next Role | none |
---

203
.ai/PLAN.md

@ -2,119 +2,136 @@
Status: **ready_for_implement**
Goal: implement a containerised YouTube downloader web app per `ROADMAP.md`.
Goal: implement Priority 2 of `ROADMAP.md` — single-video-only downloads from playlist links, correctly-named output files, and a live progress bar in the browser UI.
## Scope
- Single Docker container running a FastAPI server (Python 3.12-slim + ffmpeg + yt-dlp).
- Minimal browser UI: one URL text field + **Download MP3** and **Download Video** buttons.
- MP3 download: yt-dlp extracts best audio, ffmpeg re-encodes to MP3 at highest quality (`-q:a 0`), file streamed back to browser.
- Video download: yt-dlp fetches `bestvideo+bestaudio`, merges to MP4 via ffmpeg, file streamed back to browser.
- Invalid/empty URL returns a JSON error; the UI displays it without crashing.
- No persistent storage — temp dir per request, cleaned up after response.
- **T-003 — Single video from playlist links + correct filenames** (no architecture change; still one synchronous `POST /download` request/response):
- Pass yt-dlp's `noplaylist: True` so a playlist URL or a `watch?v=...&list=...` URL only ever downloads the one referenced video.
- Add a `sanitize_filename(title, fallback_id)` helper: strip characters outside letters (any language)/digits/spaces/`- _ . ( ) , '`, collapse resulting whitespace, trim, and fall back to the video id if nothing usable remains.
- Extract `title`/`id` from yt-dlp's result info dict and pass the sanitized name explicitly as `FileResponse(..., filename=...)`, decoupled from whatever yt-dlp happens to name the file on disk.
- Fix the real UUID-looking-filename bug: `response.blob()` in the frontend JS strips the `Content-Disposition` header, so `link.download = ''` makes the browser invent a blob-id-like name. Read the filename from the `Content-Disposition` response header in JS and set `link.download` to it explicitly (fallback to a generic name only if the header is somehow missing).
- **T-004 — Background job + live progress bar** (architecture change, builds on T-003):
- Split `POST /download` into three endpoints:
- `POST /download` — validates the URL, creates an in-memory job record (`job_id = uuid4()`, status `pending`), schedules the actual yt-dlp run on a background thread (`asyncio.get_running_loop().run_in_executor`), and returns `{"job_id": ...}` immediately (no file streaming here anymore).
- `GET /progress/{job_id}` — returns the job's current `{status, percent, speed, downloaded_bytes, total_bytes, filename, error}` from the in-memory store. `status` is one of `downloading`, `finished`, `error`.
- `GET /download/{job_id}/file` — once `status == finished`, streams the prepared file via `FileResponse` (reusing T-003's sanitized filename) with `BackgroundTask` cleanup of the job's temp dir and job record; returns 409/404 if called before completion or after cleanup.
- Progress capture: a yt-dlp `progress_hooks` callback updates the job's dict on each tick using `downloaded_bytes`, `total_bytes` (or `total_bytes_estimate`), `speed`, and computed percent; a `postprocessor_hooks` callback (or the `finished` hook state) marks the job `finished` once the final file is ready.
- Job store: a plain in-memory `dict[str, JobState]` guarded by a `threading.Lock` (single-process app, matches "no persistent storage" constraint). Entries are removed after the file is served, and a periodic/best-effort cleanup drops stale jobs (e.g. no progress endpoint hit within N minutes) so a browser tab closed mid-download doesn't leak a temp dir forever.
- Frontend JS rework:
- `downloadFile(mode)` now: POSTs to `/download` to obtain `job_id`, shows the progress bar area, then polls `GET /progress/{job_id}` every 500ms.
- On each poll: update a `<progress>` bar (or styled div) with percent, and a text line with speed (human-readable, e.g. `1.2 MB/s`) and `downloaded / total` size (human-readable bytes).
- On `status == finished`: stop polling, fetch `GET /download/{job_id}/file`, read filename from `Content-Disposition` (per T-003's fix), trigger the save via the existing blob+anchor pattern, then hide the progress bar and clear status text.
- On `status == error`: stop polling, show the error message in the existing `#status` div, hide the progress bar.
- HTML/CSS: add a progress bar element (native `<progress>` plus a small text line for speed/size) directly below the two buttons, hidden by default, shown only while a job is active.
## Acceptance Criteria
- `docker build -t yt-dl .` exits 0.
- `docker run --rm -p 8080:8080 yt-dl` starts the server and `curl http://localhost:8080/` returns 200.
- A valid YouTube URL + MP3 button delivers a `.mp3` file to the browser.
- A valid YouTube URL + Video button delivers a `.mp4` file to the browser.
- An empty URL or garbage URL returns an error visible in the UI.
(from `ROADMAP.md` Priority 2)
## Implementation Phases
### Phase 1 — T-001: FastAPI backend + HTML/JS frontend
**Files to create:**
- `app.py` — entire server + inline HTML
- `requirements.txt`
**`app.py` structure:**
1. **HTML constant** (`HTML`) — inline single-page UI:
- Text input for URL, placeholder "Paste YouTube URL…"
- Two buttons: `Download MP3` / `Download Video`
- Status `<div>` for errors/progress text
- JS `downloadFile(mode)`:
- Reads URL from input, validates non-empty client-side
- `fetch('/download', { method:'POST', body: FormData{url, mode} })`
- On success (content-type not application/json): creates object URL → `<a download>` click → revoke URL
- On error or JSON response: parse JSON and show `.message` in status div
2. **FastAPI app** (`app`):
- `GET /` — returns `HTMLResponse(HTML)`
- `POST /download``url: str = Form(...)`, `mode: Literal["mp3","video"] = Form(...)`
- Validate URL non-empty and starts with `http`; if not, raise `HTTPException(400)`
- Create `tempfile.TemporaryDirectory()`
- Build `ydl_opts`:
- MP3: `format='bestaudio/best'`, postprocessor `FFmpegExtractAudio` with `preferredcodec='mp3'`, `preferredquality='0'`
- Video: `format='bestvideo+bestaudio/best'`, `merge_output_format='mp4'`
- `outtmpl=tmpdir/%(title)s.%(ext)s`
- Run `yt_dlp.YoutubeDL(ydl_opts).download([url])` inside `asyncio.get_event_loop().run_in_executor(None, ...)` to avoid blocking the event loop
- Find the downloaded file in tmpdir (glob the single file present)
- Return `FileResponse(path, filename=..., background=BackgroundTask(cleanup_tmpdir))` — cleanup runs after response is sent
- Wrap yt-dlp call in try/except; on any exception raise `HTTPException(500, detail=str(e))`
3. **`requirements.txt`**:
```
fastapi>=0.111.0
uvicorn[standard]>=0.29.0
yt-dlp>=2024.4.9
python-multipart>=0.0.9
```
**Key decisions:**
- Inline HTML keeps the project a two-file app (plus Dockerfile) — no template engine, no static asset serving needed.
- `run_in_executor` keeps uvicorn's event loop unblocked while yt-dlp runs (which can take 30–120 s).
- `FileResponse` with `BackgroundTask` cleanup is the standard FastAPI pattern for temp file responses.
- yt-dlp is called via its Python API (not subprocess) to avoid shell escaping issues with URLs.
---
### Phase 2 — T-002: Dockerfile + README
**Files to create/update:**
- `Dockerfile`
- `README.md` (update with build/run instructions)
- Pasting a playlist URL (or a watch URL with a `list=` param) and clicking MP3 or Video downloads only that one video, never the rest of the playlist.
- Starting a download shows a progress bar below the buttons that updates with percentage, speed, and downloaded/total size while the server is processing.
- When the download finishes, the file is automatically delivered to the browser (save dialog) without an extra click.
- The saved file's name matches the video's title (emoji/icons/symbols removed, normal characters kept) instead of a UUID or other opaque id, and still ends in the correct extension.
- If the download fails server-side, the UI shows a clear error in place of the progress bar (no server crash, no stuck spinner).
**`Dockerfile`**:
```dockerfile
FROM python:3.12-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
## Implementation Phases
WORKDIR /app
### Phase 1 — T-003: Single video from playlists + correct filenames
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
**Files to change:**
- `app.py`
- `README.md` (document: playlist links only download the referenced video; filenames follow the video title with icons/emoji stripped)
COPY app.py .
**Changes in `app.py`:**
1. Add `ydl_opts["noplaylist"] = True` for both `mp3` and `video` modes.
2. Add a module-level helper:
```python
import re
EXPOSE 8080
_FILENAME_ALLOWED = re.compile(r"[^\w\s.,'()\-]", re.UNICODE)
_WHITESPACE = re.compile(r"\s+")
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]
def sanitize_filename(title: str, fallback_id: str) -> str:
cleaned = _FILENAME_ALLOWED.sub("", title or "")
cleaned = _WHITESPACE.sub(" ", cleaned).strip()
return cleaned or fallback_id
```
**`README.md`** must document:
- Prerequisites: Docker
- Build: `docker build -t yt-dl .`
- Run: `docker run --rm -p 8080:8080 yt-dl`
- Usage: open `http://localhost:8080` in a browser
- Note on video size / download time expectations
## Validation
(`\w` under `re.UNICODE`, the Python 3 default, matches any-language letters/digits/underscore, so non-Latin titles are preserved; emoji/symbols/pictographs fall outside `\w`, whitespace, and the explicit punctuation set and are dropped.)
3. In `fetch_media`, call `downloader.extract_info(url, download=True)` (instead of `.download([url])`) to get the info dict back; read `info.get("title")` and `info.get("id")`, compute `display_name = sanitize_filename(title, id) + Path(media_path).suffix`.
4. Pass `filename=display_name` to the existing `FileResponse(...)` call instead of `media_path.name`.
5. In the frontend `<script>`, change the success branch of `downloadFile`:
```js
const disposition = response.headers.get('content-disposition') || '';
const match = /filename\*?=(?:UTF-8''|")?([^\";]+)/i.exec(disposition);
const filename = match ? decodeURIComponent(match[1].replace(/"/g, '')) : 'download';
...
link.download = filename;
```
Keep the rest of the blob/anchor logic unchanged.
**Validation:**
- `python -m py_compile app.py`
- Unit-style check (via a small mocked test or manual `python -c`) that `sanitize_filename("Song 🔥 Title ✨ (Live)", "abc123") == "Song Title (Live)"` collapsed to single spaces → `"Song Title (Live)"`, and `sanitize_filename("🔥🔥🔥", "abc123") == "abc123"`.
- Manual/live check: paste a `watch?v=...&list=...` URL and confirm only the single video downloads; confirm the saved filename in the browser matches the (emoji-stripped) video title, not a UUID.
### Phase 2 — T-004: Background job + live progress bar
**Files to change:**
- `app.py`
- `README.md` (document the new `/download`, `/progress/{job_id}`, `/download/{job_id}/file` flow and the progress bar UI; note job state is in-memory/non-persistent)
**Backend changes in `app.py`:**
1. Add a `JobState` structure (dataclass or `TypedDict`) with fields: `status` (`"downloading" | "finished" | "error"`), `percent`, `speed`, `downloaded_bytes`, `total_bytes`, `temp_dir`, `file_path`, `filename`, `error_message`, `mode`, `created_at`.
2. Add a module-level `_jobs: dict[str, JobState]` and `_jobs_lock = threading.Lock()`.
3. `POST /download` (`url`, `mode` form fields, same validation as today):
- Validate URL as today (raise `HTTPException(400)` on invalid/empty).
- Create `job_id = str(uuid4())`, create the temp dir, seed a `JobState(status="downloading", ...)` in `_jobs` under the lock.
- Define a `progress_hook(d)` closure that, under the lock, updates the job's `percent`/`speed`/`downloaded_bytes`/`total_bytes` from `d["status"]`, `d.get("downloaded_bytes")`, `d.get("total_bytes") or d.get("total_bytes_estimate")`, `d.get("speed")`; on `d["status"] == "error"` marks the job `error`.
- Build `ydl_opts` same as T-003 (`noplaylist: True`, plus `progress_hooks: [progress_hook]`).
- Define `run_job()` that calls `extract_info(url, download=True)`, computes `display_name` via `sanitize_filename` (from T-003), locates the produced file, and updates the job to `status="finished"` with `file_path`/`filename` set; wraps in try/except to set `status="error"`/`error_message` on failure, and cleans up the temp dir on error (not on success — success cleanup happens after the file is served).
- Schedule `run_job` via `asyncio.get_running_loop().run_in_executor(None, run_job)` **without awaiting it** (fire-and-forget task kept referenced, e.g. appended to a module-level `set` of pending futures to avoid GC warnings).
- Return `JSONResponse({"job_id": job_id})`.
4. `GET /progress/{job_id}`:
- Look up the job under the lock; 404 (JSON error) if unknown.
- Return `{"status", "percent", "speed", "downloaded_bytes", "total_bytes", "error"}` (omit file path from the response).
5. `GET /download/{job_id}/file`:
- Look up the job; 404 if unknown, 409 (JSON error) if not yet `finished`.
- Return `FileResponse(job.file_path, filename=job.filename, background=BackgroundTask(cleanup_job, job_id))` where `cleanup_job` removes the temp dir and pops the job from `_jobs` under the lock.
6. Best-effort stale-job sweep: a small helper (called opportunistically, e.g. at the top of `POST /download`) that drops any job older than a fixed TTL (e.g. 30 minutes) whose temp dir hasn't been claimed, cleaning up its temp dir — keeps the in-memory store from growing unbounded if a browser tab is closed mid-download. No background scheduler/thread is introduced solely for this; it piggybacks on request handling to keep the app dependency-free.
**Frontend changes (inline `HTML`/JS in `app.py`):**
1. Add markup below the buttons:
```html
<div id="progress-wrap" hidden>
<progress id="progress-bar" max="100" value="0"></progress>
<div id="progress-text"></div>
</div>
```
2. Rework `downloadFile(mode)`:
- POST to `/download`, parse `{job_id}` (existing error handling for non-2xx/JSON stays the same shape as today).
- Show `#progress-wrap`, start `setInterval` polling `GET /progress/{job_id}` every 500ms.
- On each tick: update `#progress-bar.value` from `percent`; update `#progress-text` with a human-readable line, e.g. `formatSpeed(speed) + ' · ' + formatBytes(downloaded_bytes) + ' / ' + formatBytes(total_bytes)`.
- Add small `formatBytes(n)` / `formatSpeed(n)` helpers (KB/MB/GB, `/s` suffix), handling `null`/unknown total gracefully (e.g. show downloaded size only if total is unknown).
- On `status === 'finished'`: `clearInterval`, fetch `GET /download/{job_id}/file`, apply the same `Content-Disposition`-based filename logic from T-003, trigger the blob/anchor save, then hide `#progress-wrap` and reset text/bar.
- On `status === 'error'`: `clearInterval`, hide `#progress-wrap`, show `error` (or a generic message) in `#status`.
- Guard against overlapping downloads from double-clicks (e.g. disable buttons while a job is active, re-enable on finish/error).
**Validation:**
- `python -m py_compile app.py`
- Manual/live check via `docker run` (or local uvicorn): start an MP3 download, observe the progress bar advancing with percent/speed/size, confirm the file auto-downloads on completion with the correct sanitized title-based filename; repeat for Video mode.
- Manual check: trigger a server-side failure path (e.g. malformed-but-http-prefixed URL) and confirm the progress bar disappears and the existing `#status` error text appears instead, with no server crash.
- Manual check: confirm a playlist URL still only fetches the single video (retained from T-003) end-to-end through the new job flow.
## Validation (both phases)
```bash
# Build
python -m py_compile app.py
docker build -t yt-dl .
# Smoke test — server starts and home page returns 200
docker run --rm -d -p 8080:8080 --name yt-dl-test yt-dl
sleep 3
curl -sf http://localhost:8080/ | grep -q "Download MP3"
docker stop yt-dl-test
```
No unit-test framework is introduced; the app is small enough that the docker smoke test is the validation gate. The reviewer will perform a live download test.
Live download checks (MP3, Video, playlist-link-single-video, progress bar behavior, filename correctness, error path) are performed manually/by the reviewer against a real YouTube URL, consistent with how T-001/T-002 were validated — no new automated test framework is introduced.

79
.ai/REVIEW.md

@ -72,3 +72,82 @@ No blocker, major, or minor findings.
**Risks:**
- None outstanding. The live-download risk noted in the T-001 review has been verified and closed here.
---
## T-003 — `noplaylist` option, `sanitize_filename` helper, title-based filenames, frontend `Content-Disposition` fix
**Verdict:** PASS
### Findings
No blocker, major, minor, or nit findings.
### Verification
**Steps performed:**
- Re-read `.ai/PLAN.md` (Phase 1 spec for T-003) and diffed it against `app.py` / `README.md`; implementation matches the plan (`noplaylist: True` on both modes, `sanitize_filename` helper with the specified regex, `extract_info(..., download=True)` used to recover `title`/`id`, `filename=display_name` passed to `FileResponse`, frontend `Content-Disposition`-based `link.download` fix).
- `python3 -m py_compile app.py` — succeeded.
- Unit-checked `sanitize_filename` directly (scratch venv, no mocks):
- `sanitize_filename("Song 🔥 Title ✨ (Live)", "abc123")``"Song Title (Live)"`.
- `sanitize_filename("🔥🔥🔥", "abc123")``"abc123"` (fallback to id).
- `sanitize_filename("日本語 タイトル", "abc123")``"日本語 タイトル"` (non-Latin preserved).
- Verified Starlette's `FileResponse` encodes such filenames via `filename*=utf-8''...` (RFC 5987) when they contain non-ASCII/space/apostrophe characters, and confirmed the frontend regex (`/filename\*?=(?:UTF-8''|")?([^";]+)/i`) plus `decodeURIComponent` correctly recovers the original title from that header format.
- **Live end-to-end test**, built and ran the actual Docker image (`docker build -t yt-dl-review .`, `docker run -p 8081:8080`):
- `GET /` → 200, contains "Download MP3".
- `POST /download` with a playlist-context URL (`watch?v=dQw4w9WgXcQ&list=RDdQw4w9WgXcQ`), `mode=mp3` → 200; container logs show `Downloading just the video dQw4w9WgXcQ because of --no-playlist`, confirming only the referenced video is fetched, never the rest of the playlist/mix; response is a single valid MP3 (`file` confirms ID3 v2.4 MPEG audio) with `Content-Disposition: attachment; filename*=utf-8''Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster).mp3` — title-based, not a UUID.
- `mode=video` against a plain (non-playlist) URL → 200, valid MP4 (`file` confirms ISO Media MP4), filename `Me at the zoo.mp4` derived from the video title.
- `POST /download` with an invalid (non-`http`) URL → unchanged 400 JSON `{"message": ...}` behavior, confirming the existing validation path was not regressed.
- Cleaned up: stopped/removed the test container and scratch venv, deleted downloaded test files.
- Reviewed `README.md` changes; the new paragraph accurately describes the playlist-single-video behavior and the title-based/emoji-stripped filename rule, matching observed behavior.
**Findings from verification:** All acceptance criteria for T-003 hold:
- Pasting a playlist/`list=` URL downloads only the referenced video (confirmed via yt-dlp's own `--no-playlist` log line and a single file being produced).
- Saved filename matches the video title with emoji/icons stripped (normal chars, including non-Latin, kept) instead of a UUID, with the correct extension.
- `python -m py_compile app.py` passes.
**Risks:**
- None outstanding. `sanitize_filename` strips characters outside `\w`/whitespace/`- _ . ( ) , '`; this only affects the `Content-Disposition` header value used for the browser's save-as name, not the on-disk path used to read the file, so there is no path-traversal or injection concern from unsanitized titles.
---
## T-004 — Job-based `POST /download` + `GET /progress/{job_id}` + `GET /download/{job_id}/file`, in-memory job store, yt-dlp `progress_hooks`, frontend polling + progress bar UI
**Verdict:** PASS_WITH_NOTES
### Findings
1. **[minor]** `app.py:296-315` (`progress_hook`) — `job.percent` is only updated inside `if job.total_bytes:`, so when yt-dlp can't report a `total_bytes`/`total_bytes_estimate` (e.g. some DASH/live formats), the progress bar stays at 0% for the whole download even though `downloaded_bytes` is advancing and shown as text. Not required: the plan explicitly allows "downloaded size only if total is unknown" as the fallback UX, and this doesn't affect the common case (verified live: percent tracked correctly when total is known).
- Required fix: No.
2. **[minor]** `app.py:296-367``progress_hooks` only reports raw download percent; ffmpeg post-processing (mp3 extraction, video mux) after the download hits 100% is not reflected, so the bar can sit at 100% for a few extra seconds while the file is finalized before `status` flips to `finished`. Acceptable per plan (`postprocessor_hooks` was offered as one of two options, not mandatory), and verified it does not stall indefinitely or misreport.
- Required fix: No.
3. **[nit]** `app.py:356-362` — the friendlier `"The download could not be prepared."` message set by `progress_hook` on a hook-level `status == "error"` is generally overwritten by the broader `except Exception as exc: ... job.error_message = str(exc)` handler in `run_job`, since yt-dlp raises after invoking the hook. In practice this means the browser sees yt-dlp's raw exception text (confirmed in testing: `"ERROR: [generic] not-a-video: Unable to download webpage: HTTP Error 404..."`) rather than the friendlier hook message. Same class of note as the T-001 review (raw exception text surfaced to the browser); not a regression introduced by this task and not required to fix here.
- Required fix: No.
No blocker or major findings.
### Verification
**Steps performed:**
- Re-read `.ai/PLAN.md` (Phase 2 spec for T-004) and diffed it against `app.py` / `README.md`; the three-endpoint split, `JobState`/`_jobs`/`_jobs_lock`, `progress_hooks`-driven updates, `run_in_executor` fire-and-forget scheduling (`_pending_jobs` set to avoid GC warnings), stale-job TTL sweep piggybacked on `POST /download`, and the frontend polling/progress-bar/button-disable/error-handling all match the plan.
- `python3 -m py_compile app.py` — succeeded.
- Built the actual Docker image (`docker build -t yt-dl-review2 .`) and ran it, then drove the real HTTP flow end-to-end against real YouTube URLs:
- `POST /download` (playlist-context MP3 URL) → `{"job_id": ...}`; polling `GET /progress/{job_id}` every ~1s showed `status: "downloading"` with `percent`/`speed`/`downloaded_bytes`/`total_bytes` advancing correctly (0 → 61% → 100%), then `status: "finished"`.
- `GET /download/{job_id}/file` after `finished` → 200, correct MP3 bytes (`file` confirms valid MPEG audio), title-based `Content-Disposition` filename (reusing T-003's sanitization) — auto-delivered with no extra click needed on the client side.
- Job cleanup confirmed: `GET /progress/{job_id}` after the file was served → 404 `"Download job not found."`, confirming `BackgroundTask(cleanup_job, ...)` removed the job and temp dir after delivery.
- `GET /progress/<unknown>` and `GET /download/<unknown>/file` → both 404 as specified.
- Fetched `GET /download/{job_id}/file` on a still-`downloading` job → 409 `"The download is not ready yet."`, matching the plan's pre-completion semantics.
- **Error path**: posted a non-YouTube, non-existent URL (`http://example.com/not-a-video`) → job transitioned to `status: "error"` with a descriptive `error` message, no server crash (`GET /` still returned 200 immediately after), and `docker logs` showed no traceback — matches "server-side failure shows an error in place of the bar with no crash."
- Confirmed no temp-dir leak on the error path (`docker exec ... ls /tmp` showed the errored job's temp dir already removed); a separate job's temp dir that was intentionally never fetched was still present, consistent with the documented "removed after delivery, or after a timeout for abandoned jobs" behavior (not fetching a finished job's file is expected to leave it until TTL/next sweep).
- Cleaned up: stopped the test container, removed the `yt-dl-review2`/`yt-dl-review` test images, removed local scratch files.
- Reviewed `README.md` changes: accurately documents the job/progress/file endpoint flow, automatic delivery, and non-persistent in-memory job state.
**Findings from verification:** All acceptance criteria for T-004 hold:
- Progress bar appears on download start and updates with percent/speed/downloaded-total size (verified live with real percentages/speeds).
- File auto-delivers to the browser on completion with the correct sanitized filename (server-side `fetch` + `Content-Disposition` flow verified; same header-parsing logic as T-003, already confirmed to round-trip correctly in the browser).
- Server-side failure shows an error in place of the bar with no crash (verified: job marked `error`, server remained responsive).
- `python -m py_compile app.py` passes.
**Risks:**
- Progress percent can remain at 0% for formats where yt-dlp cannot report a total size (noted above, minor, not required).
- The in-memory job store means an app restart mid-download loses all job state/progress for any in-flight browser sessions; this matches the explicitly stated "no persistent storage" design constraint and is now documented in `README.md`.

2
.ai/TASKS.md

@ -22,3 +22,5 @@ Command expectations:
| --- | --- | --- | --- | --- | --- |
| T-001 | FastAPI backend (`app.py`) + inline HTML/JS frontend (`requirements.txt`) | done | `GET /` returns 200 with HTML containing "Download MP3"; `POST /download` with valid URL + mode streams a file; invalid URL returns 400 JSON with `.message` | Reviewer re-ran mocked FastAPI route checks (GET /, valid download, invalid/empty/missing-field errors, yt-dlp failure, unicode filename) — all passed; see `.ai/REVIEW.md` | none |
| T-002 | Dockerfile + README.md (build/run docs) | done | `docker build -t yt-dl .` exits 0; `docker run --rm -p 8080:8080 yt-dl` starts server; `curl http://localhost:8080/` returns HTML with download buttons | Reviewer re-ran `docker build` + container smoke test, plus a live end-to-end MP3 and video download against a real YouTube URL through the container — all passed; see `.ai/REVIEW.md` | none |
| T-003 | `app.py` (`noplaylist` option, `sanitize_filename` helper, title-based `FileResponse` filename, frontend `Content-Disposition`-based `link.download` fix) + `README.md` update | done | Pasting a playlist/`list=` URL downloads only the referenced video; saved filename matches the video title with emoji/icons stripped (normal chars kept) instead of a UUID, with correct extension; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile`, unit-checked `sanitize_filename`, and ran a live end-to-end playlist-link MP3 download + a plain video download through the built Docker image against real YouTube URLs — all passed; see `.ai/REVIEW.md` | none |
| T-004 | `app.py` (job-based `POST /download` + `GET /progress/{job_id}` + `GET /download/{job_id}/file`, in-memory job store, yt-dlp `progress_hooks`, frontend polling + progress bar UI) + `README.md` update | done | Progress bar appears below the buttons on download start and updates with percent/speed/downloaded-total size; file auto-delivers to the browser on completion with the correct sanitized filename; server-side failure shows an error in place of the bar with no crash; `python -m py_compile app.py` passes | Reviewer re-ran `py_compile` and drove the full job lifecycle live through the built Docker image against real YouTube URLs (progress polling, auto-delivered file, 404/409 semantics, server-side error path with no crash) — all passed; see `.ai/REVIEW.md` | none |

6
README.md

@ -24,7 +24,11 @@ docker run --rm -p 8080:8080 yt-dl
Open [http://localhost:8080](http://localhost:8080) in a browser, paste a YouTube URL, then select **Download MP3** or **Download Video**.
Downloads are prepared on the server before they are sent to your browser. Large videos and higher-quality formats can take several minutes, depending on the source video and your network connection.
Downloads run as temporary in-memory jobs, so a progress bar shows percentage, speed, and downloaded size while the server prepares the file. When it finishes, the file is delivered to your browser automatically. Large videos and higher-quality formats can take several minutes, depending on the source video and your network connection.
The browser starts a download with `POST /download`, polls `GET /progress/{job_id}`, and retrieves the completed file from `GET /download/{job_id}/file`. Job state and temporary files are removed after delivery (or after a timeout for abandoned jobs), and are not retained across server restarts.
If a pasted video link includes playlist context (such as a `list=` parameter), the app downloads only that referenced video. Downloaded files use the video title, with emoji and unsupported symbols removed while normal letters (including non-Latin characters), digits, and common punctuation are retained. An emoji-only title falls back to the video ID.
## AI Workflow

37
ROADMAP.md

@ -22,6 +22,39 @@ Objective: Containerised YouTube downloader with a minimal browser UI.
## Out of Scope
- Playlist batch downloads.
- Playlist batch downloads (only the single video referenced by the pasted link is ever downloaded — see Priority 2).
- User accounts / history.
- Progress bars / live download status (nice-to-have but not required for v1).
## Priority 2
Objective: single-video-only downloads from playlist links, plus live download progress in the UI.
- If the pasted URL is a playlist link or a video link that also carries a playlist context (e.g. `watch?v=...&list=...`), only the one referenced video is downloaded — the rest of the playlist is ignored. Implemented via yt-dlp's `noplaylist` option.
- Below the two download buttons, a progress bar area appears once a download starts, showing for the video currently being processed:
- percentage complete
- current download speed
- downloaded size / total size
- Progress transport: the server runs each download as a background job identified by a `job_id` and records live progress (from yt-dlp's `progress_hooks`) in server-side memory. The browser polls a `GET /progress/{job_id}` endpoint (~every 500ms) and updates the bar/text from the response.
- File delivery: when a job reaches 100%, the page automatically fetches `GET /download/{job_id}/file` and triggers the browser's normal save behavior — no extra click required, preserving today's one-click UX.
- Errors surfaced during the background job (invalid URL, yt-dlp failure) are reported through the same progress endpoint/UI status area used today, without a server crash.
- Still single-container, no external dependencies beyond Docker, no persistent storage — job state is in-memory only and is cleaned up once the file has been delivered (or after a reasonable failure/timeout window).
- The delivered filename is derived from the video's real title, not a random/opaque id (currently the browser sometimes ends up saving with a UUID-like name):
- Take the video title as reported by yt-dlp.
- Strip characters that aren't plain text — emoji, pictograms/icons, and other symbols outside normal letters/digits/punctuation — rather than replacing the whole title.
- Keep letters (any language), digits, spaces, and common punctuation (`- _ . ( ) , '`); collapse repeated whitespace left behind by stripped characters and trim the ends.
- If stripping leaves an empty name (e.g. an emoji-only title), fall back to the video id so a filename always exists.
- Keep the correct extension for the mode (`.mp3` for audio, `.mp4`/`.mkv` for video) after the cleaned title.
## Acceptance Criteria (Priority 2)
- Pasting a playlist URL (or a watch URL with a `list=` param) and clicking MP3 or Video downloads only that one video, never the rest of the playlist.
- Starting a download shows a progress bar below the buttons that updates with percentage, speed, and downloaded/total size while the server is processing.
- When the download finishes, the file is automatically delivered to the browser (save dialog) without an extra click.
- The saved file's name matches the video's title (with emoji/icons/symbols removed, normal characters kept) instead of a UUID or other opaque id, and still ends in the correct extension.
- If the download fails server-side, the UI shows a clear error in place of the progress bar (no server crash, no stuck spinner).
## Out of Scope
- Playlist batch downloads (downloading every video in a playlist in one action).
- User accounts / history.
- Persisting job/progress state across server restarts.

300
app.py

@ -1,10 +1,15 @@
"""FastAPI application serving a small YouTube download interface."""
import asyncio
import re
import shutil
import tempfile
import threading
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Literal
from uuid import uuid4
import yt_dlp
from fastapi import FastAPI, Form, HTTPException, Request
@ -13,6 +18,10 @@ from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
from starlette.background import BackgroundTask
_FILENAME_ALLOWED = re.compile(r"[^\w\s.,'()\-]", re.UNICODE)
_WHITESPACE = re.compile(r"\s+")
HTML = """<!doctype html>
<html lang="en">
<head>
@ -25,6 +34,9 @@ HTML = """<!doctype html>
main { max-width: 36rem; padding: 2rem; width: 100%; }
input { box-sizing: border-box; font-size: 1rem; padding: .8rem; width: 100%; }
button { cursor: pointer; font-size: 1rem; margin: 1rem .5rem 0 0; padding: .75rem 1rem; }
#progress-wrap { padding-top: 1rem; }
#progress-bar { width: 100%; }
#progress-text { min-height: 1.5rem; padding-top: .35rem; }
#status { color: #fca5a5; min-height: 1.5rem; padding-top: 1rem; }
</style>
</head>
@ -36,10 +48,118 @@ HTML = """<!doctype html>
<button type="button" onclick="downloadFile('mp3')">Download MP3</button>
<button type="button" onclick="downloadFile('video')">Download Video</button>
</div>
<div id="progress-wrap" hidden>
<progress id="progress-bar" max="100" value="0"></progress>
<div id="progress-text"></div>
</div>
<div id="status" role="status"></div>
</main>
<script>
let activeDownload = false;
function formatBytes(value) {
if (!Number.isFinite(value) || value < 0) return '';
const units = ['B', 'KB', 'MB', 'GB'];
let unit = 0;
let size = value;
while (size >= 1024 && unit < units.length - 1) {
size /= 1024;
unit += 1;
}
return `${size.toFixed(unit === 0 ? 0 : 1)} ${units[unit]}`;
}
function formatSpeed(value) {
const formatted = formatBytes(value);
return formatted ? `${formatted}/s` : '';
}
function filenameFromResponse(response) {
const disposition = response.headers.get('content-disposition') || '';
const match = /filename\\*?=(?:UTF-8''|")?([^";]+)/i.exec(disposition);
return match ? decodeURIComponent(match[1].replace(/"/g, '')) : 'download';
}
function setButtonsDisabled(disabled) {
document.querySelectorAll('button').forEach((button) => {
button.disabled = disabled;
});
}
function hideProgress() {
const progressWrap = document.getElementById('progress-wrap');
document.getElementById('progress-bar').value = 0;
document.getElementById('progress-text').textContent = '';
progressWrap.hidden = true;
}
async function saveDownload(jobId) {
const response = await fetch(`/download/${jobId}/file`);
if (!response.ok) {
const error = await response.json();
throw new Error(error.message || 'The download could not be delivered.');
}
const blob = await response.blob();
const objectUrl = URL.createObjectURL(blob);
const link = document.createElement('a');
link.href = objectUrl;
link.download = filenameFromResponse(response);
document.body.appendChild(link);
link.click();
link.remove();
setTimeout(() => URL.revokeObjectURL(objectUrl), 0);
}
function pollProgress(jobId) {
const status = document.getElementById('status');
const progressBar = document.getElementById('progress-bar');
const progressText = document.getElementById('progress-text');
let polling = false;
const interval = setInterval(async () => {
if (polling) return;
polling = true;
try {
const response = await fetch(`/progress/${jobId}`);
const progress = await response.json();
if (!response.ok) {
throw new Error(progress.message || 'The download progress could not be checked.');
}
progressBar.value = Number.isFinite(progress.percent) ? progress.percent : 0;
const downloaded = formatBytes(progress.downloaded_bytes);
const total = formatBytes(progress.total_bytes);
const sizeText = total ? `${downloaded} / ${total}` : downloaded;
progressText.textContent = [formatSpeed(progress.speed), sizeText].filter(Boolean).join(' · ');
if (progress.status === 'finished') {
clearInterval(interval);
await saveDownload(jobId);
hideProgress();
status.textContent = '';
activeDownload = false;
setButtonsDisabled(false);
} else if (progress.status === 'error') {
clearInterval(interval);
hideProgress();
status.textContent = progress.error || 'The download could not be prepared.';
activeDownload = false;
setButtonsDisabled(false);
}
} catch (error) {
clearInterval(interval);
hideProgress();
status.textContent = error.message || 'The download could not be prepared.';
activeDownload = false;
setButtonsDisabled(false);
} finally {
polling = false;
}
}, 500);
}
async function downloadFile(mode) {
if (activeDownload) return;
const url = document.getElementById('url').value.trim();
const status = document.getElementById('status');
if (!url) {
@ -47,30 +167,29 @@ HTML = """<!doctype html>
return;
}
activeDownload = true;
setButtonsDisabled(true);
status.textContent = 'Preparing your download…';
const data = new FormData();
data.append('url', url);
data.append('mode', mode);
try {
const response = await fetch('/download', { method: 'POST', body: data });
const contentType = response.headers.get('content-type') || '';
if (!response.ok || contentType.includes('application/json')) {
const error = await response.json();
const result = await response.json();
if (!response.ok) {
const error = result;
throw new Error(error.message || 'The download could not be prepared.');
}
const blob = await response.blob();
const objectUrl = URL.createObjectURL(blob);
const link = document.createElement('a');
link.href = objectUrl;
link.download = '';
document.body.appendChild(link);
link.click();
link.remove();
setTimeout(() => URL.revokeObjectURL(objectUrl), 0);
if (!result.job_id) {
throw new Error('The download could not be started.');
}
document.getElementById('progress-wrap').hidden = false;
status.textContent = '';
pollProgress(result.job_id);
} catch (error) {
status.textContent = error.message || 'The download could not be prepared.';
activeDownload = false;
setButtonsDisabled(false);
}
}
</script>
@ -81,11 +200,62 @@ HTML = """<!doctype html>
app = FastAPI()
@dataclass
class JobState:
"""Transient state for a browser-requested download."""
status: Literal["downloading", "finished", "error"]
percent: float = 0
speed: float | None = None
downloaded_bytes: int = 0
total_bytes: int | None = None
temp_dir: str = ""
file_path: str | None = None
filename: str | None = None
error_message: str | None = None
mode: Literal["mp3", "video"] = "mp3"
created_at: float = 0
_JOB_TTL_SECONDS = 30 * 60
_jobs: dict[str, JobState] = {}
_jobs_lock = threading.Lock()
_pending_jobs: set[asyncio.Future[None]] = set()
def sanitize_filename(title: str, fallback_id: str) -> str:
"""Return a safe, human-readable download filename stem."""
cleaned = _FILENAME_ALLOWED.sub("", title or "")
cleaned = _WHITESPACE.sub(" ", cleaned).strip()
return cleaned or fallback_id
def cleanup_tmpdir(path: str) -> None:
"""Remove a request's temporary download directory after streaming."""
shutil.rmtree(path, ignore_errors=True)
def cleanup_job(job_id: str) -> None:
"""Remove a delivered job and its temporary download directory."""
with _jobs_lock:
job = _jobs.pop(job_id, None)
if job:
cleanup_tmpdir(job.temp_dir)
def cleanup_stale_jobs() -> None:
"""Best-effort removal of abandoned download jobs."""
cutoff = time.monotonic() - _JOB_TTL_SECONDS
with _jobs_lock:
stale_jobs = [
_jobs.pop(job_id)
for job_id, job in list(_jobs.items())
if job.created_at < cutoff
]
for job in stale_jobs:
cleanup_tmpdir(job.temp_dir)
@app.exception_handler(HTTPException)
async def http_exception_handler(_: Request, exc: HTTPException) -> JSONResponse:
"""Expose errors in the format expected by the browser client."""
@ -108,14 +278,47 @@ async def home() -> str:
@app.post("/download")
async def download(
url: str = Form(...), mode: Literal["mp3", "video"] = Form(...)
) -> FileResponse:
"""Download and stream a requested audio or video file."""
) -> JSONResponse:
"""Start a download job and return its identifier immediately."""
if not url.strip() or not url.lower().startswith("http"):
raise HTTPException(status_code=400, detail="Please provide a valid URL starting with http.")
cleanup_stale_jobs()
job_id = str(uuid4())
temp_dir = tempfile.mkdtemp(prefix="youtube-download-")
output_template = str(Path(temp_dir) / "%(title)s.%(ext)s")
ydl_opts: dict[str, object] = {"outtmpl": output_template}
with _jobs_lock:
_jobs[job_id] = JobState(
status="downloading", temp_dir=temp_dir, mode=mode, created_at=time.monotonic()
)
def progress_hook(progress: dict[str, object]) -> None:
"""Copy yt-dlp progress updates into the job visible to the browser."""
with _jobs_lock:
job = _jobs.get(job_id)
if not job:
return
if progress.get("status") == "error":
job.status = "error"
job.error_message = "The download could not be prepared."
return
if progress.get("status") != "downloading":
return
downloaded = progress.get("downloaded_bytes")
total = progress.get("total_bytes") or progress.get("total_bytes_estimate")
speed = progress.get("speed")
job.downloaded_bytes = int(downloaded) if isinstance(downloaded, (int, float)) else 0
job.total_bytes = int(total) if isinstance(total, (int, float)) else None
job.speed = float(speed) if isinstance(speed, (int, float)) else None
if job.total_bytes:
job.percent = min(100, job.downloaded_bytes / job.total_bytes * 100)
ydl_opts: dict[str, object] = {
"outtmpl": output_template,
"noplaylist": True,
"progress_hooks": [progress_hook],
}
if mode == "mp3":
ydl_opts.update(
{
@ -132,22 +335,69 @@ async def download(
else:
ydl_opts.update({"format": "bestvideo+bestaudio/best", "merge_output_format": "mp4"})
def fetch_media() -> None:
with yt_dlp.YoutubeDL(ydl_opts) as downloader:
downloader.download([url])
def run_job() -> None:
try:
await asyncio.get_running_loop().run_in_executor(None, fetch_media)
with yt_dlp.YoutubeDL(ydl_opts) as downloader:
info = downloader.extract_info(url, download=True)
downloads = [path for path in Path(temp_dir).iterdir() if path.is_file()]
if len(downloads) != 1:
raise RuntimeError("The downloaded file could not be identified.")
media_path = downloads[0]
title = str(info.get("title") or "")
video_id = str(info.get("id") or "download")
display_name = sanitize_filename(title, video_id) + media_path.suffix
with _jobs_lock:
job = _jobs.get(job_id)
if job:
job.status = "finished"
job.percent = 100
job.file_path = str(media_path)
job.filename = display_name
except Exception as exc:
cleanup_tmpdir(temp_dir)
raise HTTPException(status_code=500, detail=str(exc)) from exc
with _jobs_lock:
job = _jobs.get(job_id)
if job:
job.status = "error"
job.error_message = str(exc)
future = asyncio.get_running_loop().run_in_executor(None, run_job)
_pending_jobs.add(future)
future.add_done_callback(_pending_jobs.discard)
return JSONResponse({"job_id": job_id})
@app.get("/progress/{job_id}")
async def progress(job_id: str) -> JSONResponse:
"""Return the browser-safe state of an active download job."""
with _jobs_lock:
job = _jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="Download job not found.")
payload = {
"status": job.status,
"percent": job.percent,
"speed": job.speed,
"downloaded_bytes": job.downloaded_bytes,
"total_bytes": job.total_bytes,
"error": job.error_message,
}
return JSONResponse(payload)
@app.get("/download/{job_id}/file")
async def download_file(job_id: str) -> FileResponse:
"""Stream a completed job's file, cleaning its state after delivery."""
with _jobs_lock:
job = _jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="Download job not found.")
if job.status != "finished" or not job.file_path or not job.filename:
raise HTTPException(status_code=409, detail="The download is not ready yet.")
file_path = job.file_path
filename = job.filename
return FileResponse(
media_path,
filename=media_path.name,
background=BackgroundTask(cleanup_tmpdir, temp_dir),
file_path,
filename=filename,
background=BackgroundTask(cleanup_job, job_id),
)

Loading…
Cancel
Save