huggingface_hub v1.32.0 ships a shared blobs cache: Xet-backed files are now deduplicated locally across repos (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the release notes and huggingface/huggingface_hub#4498.
Concretely, downloading two repos that share components now only pays the download cost once — e.g. Qwen/Qwen-Image and Qwen/Qwen-Image-Edit share ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB), Kwai-Kolors/Kolors / Kolors-Inpainting ~56% (>20GB).
This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).
Proposed follow-ups:
- Verify dedup actually kicks in for modular diffusers repos (Xet-enabled, identical blobs) and measure the gain.
- If confirmed, plan removal/simplification of the "download from original repo" workaround, bumping the
huggingface_hub floor as needed and keeping a fallback for older versions.
Requested by Sayak Paul - Slack thread - Agent trace
huggingface_hub v1.32.0ships a shared blobs cache: Xet-backed files are now deduplicated locally across repos (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the release notes and huggingface/huggingface_hub#4498.Concretely, downloading two repos that share components now only pays the download cost once — e.g.
Qwen/Qwen-ImageandQwen/Qwen-Image-Editshare ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB),Kwai-Kolors/Kolors/Kolors-Inpainting~56% (>20GB).This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).
Proposed follow-ups:
huggingface_hubfloor as needed and keeping a fallback for older versions.Requested by Sayak Paul - Slack thread - Agent trace