Skip to content

Discovery

Discovery crawls a page URL for downloadable media links using configurable site rules.

What is Discovery?

When a URL points to a gallery, index, or media collection (not a single video), discovery crawls the page to extract links.

Examples: - Artist portfolio with embedded videos - News site with article links - Social media profile with uploads - Photo gallery with downloadable media

Using Discovery

CLI

Use vdl crawl for headless page discovery and download:

vdl crawl "https://example.com/gallery"

To stop after the first few successful downloads:

vdl crawl "https://example.com/gallery" --first 3

Like vdl download, crawl supports runtime flags such as --output, --quality, --if-exists, cookies options, and --json:

vdl crawl "https://example.com/gallery" --first 3 --output ~/Videos --quality 720p
vdl crawl "https://example.com/gallery" --first 3 --json
vdl crawl "https://example.com/gallery" --crawl-pagination

Discovery is also available in interactive mode when you choose the crawl option.

Python

import asyncio
from vdl import CrawlerPolicy, VDLClient
from vdl.models import Quality

async def main():
    client = VDLClient()

    # Crawl a page and build a plan
    plan = await client.discover(
        "https://example.com/gallery",
        quality=Quality.P720,
        crawler_policy=CrawlerPolicy(min_filesize_bytes=7 * 1024 * 1024),
    )

    print(f"Found {plan.total} items")

    # Then execute to download all discovered links
    results = await client.execute_plan(
        plan,
        crawler_policy=CrawlerPolicy(min_filesize_bytes=7 * 1024 * 1024),
    )

    # Or try candidates in order until three downloads succeed
    first_three = await client.execute_plan(
        plan,
        crawler_policy=CrawlerPolicy(first_successful=3),
    )

    # Or follow numbered/navigation pages during discovery
    expanded = await client.discover(
        "https://example.com/gallery",
        crawler_policy=CrawlerPolicy(crawl_pagination_pages=True),
    )

asyncio.run(main())

CrawlerPolicy is the public crawl-specific control surface. It lets library callers adjust success-target execution and the crawl-only filesize filter without affecting direct downloads.

on_status Callback

Track discovery progress:

async def on_status(event_type: str, data: dict) -> None:
    if event_type == "started":
        print("Crawling started...")
    elif event_type == "crawling":
        print(f"Crawled {data.get('links_found')} links...")
    elif event_type == "completed":
        print(f"Done! Total items: {data.get('total')}")
    elif event_type == "failed":
        print(f"Error: {data.get('error')}")

plan = await client.discover(
    "URL",
    on_status=on_status
)

Site Rules (websites.yaml)

Discovery uses configurable rules to extract media links from specific sites.

Location

~/.config/vdl/websites.yaml

Or custom path:

vdl config set site_rules_path ~/.config/vdl/my-rules.yaml

Format

- domain: example.com
  name: "Example Gallery"
  video_patterns:
    - "//video/@href"        # XPath expressions
    - "//a[contains(@href, '.mp4')]/@href"
  pagination_patterns:
    - rel: next
      selector: "a.next-page"

Discovery Report Metadata

After discovery, the plan includes metadata about the crawl:

plan = await client.discover("URL")

discovery_info = plan.metadata.get("discovery", {})
print(f"Matched rule: {discovery_info.get('matched_rule')}")
print(f"Crawler method: {discovery_info.get('crawler_method')}")
print(f"Links found: {discovery_info.get('total_links_seen')}")
print(f"Fallback: {discovery_info.get('fallback_used')}")

Fallback Behavior

If no site rule matches, VDL falls back to a generic HTML crawler that: 1. Extracts all links with video/media extensions 2. Blacklists ads and tracking URLs 3. Handles pagination if patterns are detected

This works for most generic pages without custom rules.

Error Handling

Discovery can fail if: - The URL is invalid - The page cannot be fetched - No discovery resolver is configured - No links are found

Always check the plan's metadata and total before executing:

plan = await client.discover("URL")

if plan.total == 0:
    print("No items found")
elif "error" in plan.metadata:
    print(f"Discovery failed: {plan.metadata['error']}")
else:
    results = await client.execute_plan(plan)

Workflow Diagram

flowchart TD
    URL["discover(url)"]
    SiteRule{"Matching<br/>site rule?"}

    SiteRuleCrawl["SiteRuleDiscoveryResolver<br/>apply patterns"]
    GenericCrawl["Generic HTML crawler<br/>extract links"]

    Links["Extract all links"]
    Filter["Blacklist ads/tracking<br/>Apply pagination"]
    Plan["AcquisitionPlan<br/>PlanKind.DISCOVERY_CANDIDATES"]

    URL --> SiteRule
    SiteRule -- Yes --> SiteRuleCrawl --> Links
    SiteRule -- No --> GenericCrawl --> Links
    Links --> Filter
    Filter --> Plan

Examples

See demos/crawl_websites.py.

See Also