Retry and Duplicate Semantics¶

This document clarifies VDL's idempotency model, duplicate detection, and retry strategies. It's important for understanding edge cases and designing robust workflows.

The Idempotency Model¶

VDL's core guarantee: if you ask for the same download twice, you get the same result.

Filesystem Is Source of Truth¶

The filesystem always wins. The optional state store (SQLite) is only a hint.

# Check 1: Filesystem check
if output_path.exists():
    return "skip"

# Check 2: State store check (interactive mode only)
if state_store.has_record(url):
    # But this is just a hint; verify disk
    if output_path.exists():
        return "skip"

Default Policy¶

When if_exists=SKIP (the default):

Completed media file exists → Skip
Only .part / temporary file exists → Resume or continue
Only sidecars exist (.vtt, .jpg, .json) → Continue (not a completed download)
Nothing exists → Download

Example: Interrupted Download¶

Initial download:
  config.output_dir/video.mp4.part  ← in-flight, 300 MB

Second attempt with if_exists=SKIP:
  // .part file detected, continues
  // Downloader resumes from byte 300M
  // Result: COMPLETED

Example: Partial Sidecar Only¶

First attempt failed after metadata fetch:
  config.output_dir/
    video.jpg      ← thumbnail (sidecar, not media)
    video.json     ← metadata (sidecar, not media)
    (video.mp4 never started)

Second attempt with if_exists=SKIP:
  // Only sidecars present, no media file
  // Duplicate check: "continue" (not a real duplicate)
  // Downloader starts fresh
  // Result: COMPLETED (or FAILED if network issue persists)

Duplicate Detection Rules¶

The DuplicateDetectionService examines:

Filename stem (before extension)
File extensions (media vs. non-media)
Browser/Finder-style copy suffixes (handles duplicates)

What Counts as a Completed Download¶

A file counts as completed media if: - It is a regular file (not a directory) - Its extension is not in the non-media list

Non-media extensions (do not count as completed): - .part, .ytdl, .tmp, .aria2 — in-flight files - .json, .vtt, .srt — metadata and captions - .jpg, .jpeg, .png, .webp, .gif — thumbnails - .description, .txt — text sidecars

Duplicate Stem Matching¶

When checking if a file is a duplicate, VDL ignores common copy suffixes:

Original download:
  episode.mp4

Duplicate matches (treated as the same):
  episode (1).mp4     ← Browser "copy" counter
  episode (2).mp4     ← Finder copy format
  episode copy.mp4    ← Finder copy format
  episode copy 2.mp4  ← Finder copy format

Not treated as duplicates (different content):
  episode 2.mp4       ← Real title, not a copy
  episode-v2.mp4      ← Different version

Why: Users copy files with Finder or browser. VDL recognizes that episode (1).mp4 is still the same download, not a new one.

Collision Policies¶

When a file already exists at the target path, if_exists determines the action:

Policy	Behavior	Use For
`SKIP`	Return `SKIPPED` without downloading	Archives, completed libraries
`RENAME`	Save as `title-2.mp4`, `title-3.mp4`, etc.	Incremental batch jobs
`OVERWRITE`	Replace the existing file	Updates, re-downloads
`FAIL`	Raise `VDLDownloadError`	Strict workflows, catch errors

Example: RENAME Strategy¶

result = await client.download(
    "https://example.com/video",
    output_dir="/videos",
    if_exists=IfExists.RENAME,
)

# First attempt:
#   /videos/video.mp4

# Second attempt (same URL):
#   /videos/video-2.mp4

# Third attempt:
#   /videos/video-3.mp4

Example: OVERWRITE Strategy¶

result = await client.download(
    "https://example.com/video",
    output_dir="/videos",
    if_exists=IfExists.OVERWRITE,
)

# Every attempt overwrites /videos/video.mp4
# Useful for re-downloading updated content

Retry Semantics¶

What Is "Retryable"?¶

A download is marked retryable if the failure is due to a transient issue:

Network timeouts
HTTP 429 (rate limit) with Retry-After
Authentication challenges (can retry with cookies)
Incomplete transfer (connection dropped mid-download)

Not retryable: - Invalid URL - Video not found (HTTP 404) - Completely unsupported site - Disk full (unless storage is freed)

Suggested Retry Strategies¶

VDL examines failures and suggests recovery:

Error	Suggested Strategy	Example
Subtitle fetch 429	Download without subtitles	`write_subtitles=False`
Bot detection	Retry with impersonation	Use `--impersonate firefox`
Auth challenge	Retry with cookies	Provide `.netrc` or cookies.txt
Network timeout	Retry with longer timeout	Reduce concurrent downloads

Example: Conditional Retry¶

from vdl.core.retry import retryable_failed_results, suggested_retry_strategy

results = await client.execute_plan(plan)

failed = retryable_failed_results(results)
if failed:
    strategy = suggested_retry_strategy(results)
    print(f"Suggested: {strategy}")

    if strategy == "download without subtitles":
        retry_results = await client.execute_plan(
            plan,
            write_subtitles=False,
        )

State Store Behavior (Interactive Mode)¶

When interactive mode is enabled with SQLiteStateStore:

What Gets Recorded¶

{
    "url": "https://example.com/video",
    "output_path": "/videos/video.mp4",
    "status": "completed",
    "started_at": "2026-06-12T10:30:00Z",
    "completed_at": "2026-06-12T10:35:42Z",
    "duration_seconds": 342,
    "filesize_bytes": 524288000,
    "error": None,
    "retryable": False,
    "metadata": {"title": "Example Video", ...}
}

When State Goes Stale¶

The DB record can diverge from reality if:

User deletes the file manually

DB: status=completed
Disk: file is gone
→ Next download: duplicate detection sees no file, downloads again

User moves/renames the file

DB: output_path=/videos/old-name.mp4
Disk: file is at /videos/new-name.mp4
→ Next download: sees no file at old path, downloads again

Multiple machines write to the same directory

Machine A: downloads video.mp4
Machine B: deletes video.mp4
Machine A DB: still says completed
→ DB is out of sync until next duplicate check

Reconciliation¶

To detect stale records, use client.reconciliation() (if available):

report = await client.reconciliation()

# Shows:
# - Records with missing files
# - Files on disk not in DB
# - Opportunities to resync

Concurrent Download Semantics¶

When downloading a playlist or batch, each item is independent:

results = await client.download_playlist(
    playlist_url,
    max_concurrent=4,  # Up to 4 downloads at once
)

Important: - Item 1 failure does not stop items 2–4 - Each item has independent duplicate detection - Cancellation stops all pending items - State records are written per-item as they complete

Race Condition Example¶

Initial state: /videos/ is empty

Two processes download simultaneously:
  Process A: GET /videos/video.mp4
  Process B: GET /videos/video.mp4

Process A checks: file doesn't exist → downloads
Process B checks: file doesn't exist (A not done yet) → downloads

Result: /videos/video-2.mp4 (RENAME policy)

To avoid this, use: - if_exists=SKIP (skip if any version exists) - Single-writer pattern (one process manages downloads) - Distributed locking (if multiple machines access same dir)

Edge Cases¶

Case 1: Empty Directory¶

Directory exists: /videos/
Contents: (empty)

Download attempt: continues, never skipped

Empty directories do not trigger duplicate detection.

Case 2: Directory Named Like a File¶

Attempted output: /videos/episode.mp4
On disk: /videos/episode.mp4/  (it's a directory!)

Duplicate check: FAIL
Downloader: Cannot write to /videos/episode.mp4 (is a directory)
Result: FAILED with "IsADirectoryError"

Case 3: Symlink to Missing Target¶

On disk: /videos/video.mp4 → /external/missing.mp4 (link target gone)

if_exists=SKIP: Check sees symlink exists → SKIPPED
if_exists=OVERWRITE: Overwrites symlink (doesn't follow it)

Case 4: Very Long Filename¶

Planned filename: really-long-title-that-exceeds-255-characters.mp4

Downloader: Truncates to filesystem max (usually 255 bytes)
Duplicate detection: Matches sanitized, truncated name

Case 5: Unicode Normalization¶

Requested: café.mp4
On disk: café.mp4  (different Unicode normalization)

On case-sensitive filesystems (Linux): Different files!
On case-insensitive filesystems (macOS): Same file

Recommendation: Use ASCII filenames or normalize in your application.

Best Practices¶

Practice 1: Validate Disk After Completion¶

result = await client.download(url)

if result.status == DownloadStatus.COMPLETED:
    # Verify the file actually exists
    assert Path(result.output_path).exists(), "File missing!"
    assert Path(result.output_path).stat().st_size > 0, "Empty file!"

Practice 2: Archive Completed Batches¶

After completing a large batch, move to a separate directory:

results = await client.execute_plan(plan)

# Move completed files to archive
for result in results:
    if result.status == DownloadStatus.COMPLETED:
        shutil.move(result.output_path, f"/archive/{result.output_path.name}")

Practice 3: Retry with Exponential Backoff¶

async def retry_with_backoff(client, url, max_retries=3):
    for attempt in range(max_retries):
        result = await client.download(url)
        if result.status == DownloadStatus.COMPLETED:
            return result

        if not is_retryable_result(result):
            raise Exception(f"Not retryable: {result.error}")

        delay = 2 ** attempt  # 1s, 2s, 4s
        print(f"Attempt {attempt + 1} failed, retrying in {delay}s...")
        await asyncio.sleep(delay)

    raise Exception(f"Failed after {max_retries} attempts")

Practice 4: Use State Store Hints, Not Decisions¶

# ✗ BAD: Trust the DB completely
record = await state_store.get(url)
if record and record.status == "completed":
    return "SKIPPED"

# ✓ GOOD: Use DB as a hint, verify disk
result = await client.download(url)  # Let VDL handle it
# VDL uses disk as source of truth