Extractors

Extractors

🌐 English · Deutsch

Extractors resolve a video host's embed URL into a direct stream link (usually HLS/MP4) that yt-dlp/FFmpeg can download.

Host status

Enabled (SUPPORTED_PROVIDERS in config.py):

Host	Module	Notes
VOE	voe.py	Multi-stage deobfuscation of the inline player JSON (ROT13, junk removal, double Base64, byte shift, reversal); falls back to a redirect URL in the page, captcha support. Default provider for most model classes
Vidmoly	vidmoly.py	Plain inline JS — the file: '…m3u8' URL is regexed out of the <script> tags. Requires referer https://vidmoly.biz; view URLs are rewritten to embed URLs first
Vidoza	vidoza.py	Plain inline JW-Player config, src: "…" regexed out, no deobfuscation
VeeV	veev.py	Needs a headless browser (patchright): the CDN URL only works with the session cookies + User-Agent it was issued to. download_from_veev() grabs URL/cookies/UA once, closes the browser and transfers over parallel HTTP range requests — so it bypasses the generic yt-dlp/ffmpeg pipeline. Models do not special-case it: models/common/common.py's _download_via_hoster() routes it by the resolved host, so every model family and every third-party module gets it automatically. Availability check via /api/veev/check
Vidara	vidara.py	API-based: POST <host>/api/stream with the filecode returns the m3u8. The API host is taken from the embed URL, so all of Vidara's rotating mirror domains work
Vidavaca	vidavaca.py	Vidara clone — same /api/stream scheme, extraction is delegated to the helpers in vidara.py

Present but disabled (commented out in SUPPORTED_PROVIDERS):

Host	Module	Notes
Doodstream	doodstream.py	Signed short-lived links: the pass_md5 URL + token are regexed out of the embed page, the final link is assembled from a random string, the token and a Unix expiry
Filemoon	filemoon.py	Two strategies: the modern Byse REST API (/api/videos/<code>, AES-256-GCM payload whose key is reassembled from key_parts), else legacy pages packed with the Dean Edwards eval(function(p,a,c,k,e,d)) obfuscator
LoadX	loadx.py	Stub — both functions raise NotImplementedError
Luluvdo	luluvdo.py	Stub — both functions raise NotImplementedError
Streamtape	streamtape.py	Stub — both functions raise NotImplementedError

hanime.py also lives in extractors/provider/ but is not a hoster and is not listed in SUPPORTED_PROVIDERS — see the next section.

hanime (special case — browser-based, no third-party hoster)

hanime.tv doesn't use an external video hoster; it serves its own AES-128 encrypted HLS. Since its Astro rewrite the site also signs every /api/v8 request (handshake via auth.hanime.tv + a per-request signature) and the player only loads the stream after the poster is clicked, so plain HTTP can't reach the data. The provider therefore splits its work:

• Browsing / search — the unsigned search.htv-services.com host, plain HTTP (no browser). This is why start-page cards load instantly.
• Detail + stream — a headless browser (patchright, same dependency as VeeV) in models/hanime_tv/browser.py. fetch_video(slug) reads metadata from the page's ld+json + DOM (title, poster, tags, censored, franchise episodes) and, when a stream is needed, clicks the play overlay and intercepts the …highwinds-cdn.com/….m3u8 request. Results are cached for 5 min with a per-slug lock so opening a title spawns only one browser.

The extractor extractors/provider/hanime.py exposes get_direct_link_from_hanime (returns the m3u8 via the scraper) and download_from_hanime.

The stream is AES-128 encrypted, so yt-dlp cannot download it directly — the segments have to be decrypted first. The download is therefore done by a single FFmpeg pass, which fetches the segments and decrypts AES-128 natively (-allowed_extensions ALL, since the segments are disguised as .html; crypto in the protocol whitelist; and a Referer so the sign.bin key can be fetched). The known manifest duration gives a real progress bar.

Auto-discovery

extractors/__init__.py imports all modules from extractors/provider/ and registers every function whose name starts with get_direct_link_from_, get_preview_image_link_from_ or get_subtitles_from_ in the provider_functions dict:

provider_functions["get_direct_link_from_voe"](embed_url)

Model code does not pick the extractor by the label the streaming site printed next to the embed — sites frequently label a link "Vidara" while the redirect lands on a voe.sx page. get_direct_link_for() resolves the redirect first and looks the netloc up in HOST_PROVIDER_MAP (exact or subdomain-suffix match); the site's own label is only the fallback.

At app startup, _get_working_providers() (web/app.py) probes each entry from SUPPORTED_PROVIDERS with an empty URL string: if the extractor raises NotImplementedError it counts as not implemented; any other exception means "implemented" (it correctly rejected the empty URL). Only functional providers appear in the UI (WORKING_PROVIDERS).

Adding a new extractor

1. Create the file: src/mediaforge/extractors/provider/<host>.py
2. Implement the function (mind the naming convention — <host> lowercase):

import logging
logger = logging.getLogger(__name__)

try:
    from ...config import GLOBAL_SESSION, is_source_unavailable
except ImportError:
    from mediaforge.config import GLOBAL_SESSION, is_source_unavailable


def get_direct_link_from_<host>(embed_url: str) -> str:
    if not embed_url:
        raise ValueError("Empty URL")
    resp = GLOBAL_SESSION.get(embed_url)
    if is_source_unavailable(resp.text, resp.status_code):
        raise ValueError("Source removed")
    # ... parse HTML/JS, extract the stream URL ...
    return direct_link

3. Enable the provider: add its name to SUPPORTED_PROVIDERS (config.py) — spelling must match func_name = f"get_direct_link_from_{p.lower()}".
4. Define headers: if the host needs special headers, add them to PROVIDER_HEADERS_D (download) and possibly PROVIDER_HEADERS_W (watch) in config.py — these are passed to yt-dlp/FFmpeg.
5. Optional: get_preview_image_link_from_<host>() for preview images and get_subtitles_from_<host>() for subtitle tracks ([{"url", "lang", "label"}], return [] on failure — subtitles must never break a download).
6. Optional: if the host also serves the link under other domains, add them to HOST_PROVIDER_MAP so mislabeled embeds still resolve.

Third-party hosters (register_hoster)

Steps 1–5 above ("Adding a new extractor") apply when you can edit MediaForge's own source tree. A third-party module under ~/.mediaforge/thirdparties/<name>/ cannot drop a file into extractors/provider/ — that directory is only auto-discovered from inside the source tree — so it calls register_hoster() (mediaforge/extractors/__init__.py) from its own register(app) instead:

from mediaforge.extractors import register_hoster

def register(app):
    register_thirdparty(item_id="my_hoster_mod", label="My Hoster", ...)
    register_hoster(
        "my_hoster_mod",
        name="MyHoster",
        get_direct_link=get_direct_link_from_myhoster,
        headers={"Referer": "https://myhoster.example/"},
        host_patterns=("myhoster.example",),
    )

This does the same three things steps 3–4 above do by hand -- adds the name to SUPPORTED_PROVIDERS, wires the function into provider_functions, and merges the headers into PROVIDER_HEADERS_D/PROVIDER_HEADERS_W -- plus merges host_patterns into HOST_PROVIDER_MAP and refreshes WORKING_PROVIDERS so the new hoster shows up live, no restart. See .examples/thirdparties/README.md's "Hosters (register_hoster)" section for the full contract and the uninstall/cleanup behaviour.

Helpers and constants from config.py

Function / constant	Purpose
GLOBAL_SESSION	Thread-safe niquests session with realistic headers and DoH DNS
MEDIAFORGE_TEMP_DIR	Shared scratch directory (<os-temp>/mediaforge) for work-in-progress files: raw downloads, ffmpeg passes, encode/upscale temp outputs. Write your intermediate file here and move it to its destination only once it is complete, so a slow or networked target never sees a partial file. Call os.makedirs(MEDIAFORGE_TEMP_DIR, exist_ok=True) first — a reboot or a tmp cleaner may have removed it.
MEDIAFORGE_CONFIG_DIR	Per-user config/data directory (~/.mediaforge)
is_source_unavailable(html, status)	Detects removed videos (404/410/451 + HTML markers) without an extra request
check_redirect_available(url)	Follows the streaming site's redirect and checks whether the host still has the video (curl_cffi with Chrome impersonation against Cloudflare)
resolve_redirect_url(url)	Returns the final host URL behind a redirect

Captcha integration

When an extractor detects a captcha page (is_captcha_page from playwright/captcha.py), it can call solve_captcha(...). If the call runs inside the queue worker, the queue_id is set via thread-local — which makes the solving interactive in the WebUI (live screenshot + click forwarding). Chromium runs with stealth patches, network ad blocking and protection against ad overlays.

Provider contract tests

A provider breaks in a very specific way: the site quietly changes its markup, the parser stops finding episodes, and Auto-Sync starts reporting "could not read the series page". Nothing in the codebase changed, so nothing in CI noticed — the first signal is an issue from somebody whose downloads stopped three days ago.

Two checks close that gap, and they are deliberately separate.

Offline (tests/test_provider_contracts.py, part of the normal suite). Parses the recorded fixtures in tests/contracts/ and asserts the parser still extracts what it used to — episode URLs still match the provider's episode_pattern, seasons and episodes are still found. This catches our regressions: a refactor that breaks episode extraction fails the pull request that introduced it. No network, so it runs everywhere.

Live (MEDIAFORGE_CONTRACT_LIVE=1, the provider-contracts.yaml workflow, daily). Fetches the real page and checks the shape of what comes back. This catches their changes. It is not in ci.yaml on purpose: a site being down would otherwise fail an unrelated pull request, and a check that fails for reasons the author cannot fix is one the team learns to ignore and then disables. It reports by commenting on a single open provider-contract issue rather than opening one a day.

Counts are compared with a floor, not for equality — a series gains episodes, and a check that fires because a new one aired is a check nobody trusts. What must not change is that there are episodes, seasons and a title.

Recording a fixture

python tests/contracts/record.py aniworld https://aniworld.to/anime/stream/<slug>

The recorder strips scripts and inline handlers, and refuses to save a page that carries session markers (logout, csrf, …) or one the parser found no episodes in — recording a broken parse would pin the broken state as the expected one. Nothing from an adult provider goes in there; the fixtures are committed to a public repository.

When the live check fails and the offline one passes, the site changed: fix the parser, re-record, and the offline test now guards the new shape.