Add Echo modular pipeline by Gelercatty · Pull Request #14696 · huggingface/diffusers

Gelercatty · 2026-09-03T11:57:19Z

What does this PR do?

Adds official Diffusers Modular Pipeline support for Echo, a long-video model that generates synchronized video and audio with cross-shot memory.

This supersedes #13910 with a modular implementation that incorporates the earlier review feedback:

adds the standalone diffusers.modular_pipelines.echo package and the public EchoModularPipeline / EchoBlocks APIs
reuses existing lower-level Diffusers components without exposing LTX in Echo's public API
keeps checkpoint conversion in scripts/ instead of adding an original-checkpoint runtime pipeline
supports an optional first frame and up to seven paired image/audio memory slots
supports variable-length memory audio and crops audio longer than 9.62 seconds to its highest-response window
implements Echo's positive-only conditioning and stochastic 8-step DMD denoising workflow
adds mixed-precision-safe audio decoding, documentation, and tests using a public tiny fixture

Official resources:

Source implementation: https://github.com/jd-opensource/JoyAI-Echo/tree/echo-long-diffusers
Model weights: https://huggingface.co/Echo-Team/JoyAI-Echo
Tracking issue: [New Pipeline/Model] Add JoyAI-Echo multi-shot audio-video generation pipeline #13909

Closes #13909
Supersedes #13910

Testing

make style
make fix-copies
make quality
PYTHONPATH=src python -m pytest -q tests/modular_pipelines/echo/test_modular_pipeline_echo.py
- 31 passed, 6 skipped

AI-assisted self-review

Codex was used to help review, refactor, test, and prepare this contribution. The final diff was reviewed using the repository's model-integration and self-review guidance.

The final self-review found no blocking correctness issues. Echo has its own public package and naming throughout; existing lower-level Diffusers component types are reused only as implementation dependencies. Memory-slot count, short-audio handling, variable audio lengths, response-aware 9.62-second cropping, deterministic DMD re-noising, mixed-precision decoding, loading, save/load, workflow, and memory behavior are covered by the test suite.

Validation scope: the final pass used the public Echo-Team/tiny-echo-modular-pipe fixture on CPU. A full production-checkpoint conversion and end-to-end GPU inference run was not repeated in this final pass.

Before submitting

Who can review?

@asomoza @yiyixuxu

HuggingFaceDocBuilderDev · 2026-09-04T00:32:55Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu

thanks, I left some feedbacks

Gelercatty · 2026-09-07T07:27:33Z

@yiyixuxu Thanks for the review — all feedback is addressed in 64f423a and 679db3f.

removed the echo_ prefix from leaf source filenames
changed module-level helpers to explicit component arguments
reduced text-encoder outputs and derived batch size/dtype downstream
moved all assembled blocks to modular_blocks_echo.py
split VAE-only encoding from transformer-specific packing/RoPE preparation
removed VAE requirements from before-denoise blocks
flattened EchoBlocks to text_encoder, vae_encoder, denoise, decode
inlined the single-use _as_list helper
added a behavior test that runs the VAE encoder standalone without a transformer

Validation:

make fix-copies
Echo-scoped Ruff check and format check
forward/call docstring check
AI contribution check
pytest -q tests/modular_pipelines/echo/test_modular_pipeline_echo.py: 31 passed, 6 skipped

Self-review found no remaining blocking issue in this update. The repository-wide make quality is currently blocked by 52 pre-existing formatting differences outside the Echo files; all Echo-scoped quality checks pass.

Could you please take another look?

yiyixuxu

thanks!
i left some more comments

yiyixuxu · 2026-09-08T23:54:03Z

+        mel = torch.log(torch.clamp(mel_transform(waveform), min=1e-5)).permute(0, 2, 1).unsqueeze(0)
+
+    latents = audio_vae.encode(mel.to(audio_vae.dtype)).latent_dist.mode()
+    latents = _pack_audio_latents(latents)


is it possible to make the audio latent and video latent consistent? i.e. both normalized in encoder and packed in before_denoise? see a bit more write up here #14730

Done in 9d182ea. Both encoders now return normalized, unpacked VAE tensors. Audio statistics are reshaped over the channel/mel axes before normalization, and audio packing now lives in before_denoise.py, matching video. Added FP32/BF16 packed-statistics equivalence tests and standalone audio/video encoder coverage.

yiyixuxu · 2026-09-09T00:06:06Z

+            latent_height = block_state.height // components.vae_spatial_compression_ratio
+            latent_width = block_state.width // components.vae_spatial_compression_ratio


Suggested change

latent_height = block_state.height // components.vae_spatial_compression_ratio

latent_width = block_state.width // components.vae_spatial_compression_ratio

_, _, latent_height, latent_width = memory_video_latents[0].shape

Done in 9d182ea. The memory geometry is now derived from memory_video_latents[0].shape[-2:]. Echo's video VAE tensors are 5D (B, C, F, H, W), so this reads the spatial dimensions directly. Added a standalone preparation test with non-default rectangular latent geometry.

yiyixuxu · 2026-09-09T00:06:34Z

+            InputParam.template("height", default=512),
+            InputParam.template("width", default=704),


Suggested change

InputParam.template("height", default=512),

InputParam.template("width", default=704),

I think we can derive from latent shape

Done in 9d182ea. Removed height and width from EchoPrepareConditioningStep. It derives memory geometry from the VAE latent tensors and runs standalone with only the transformer loaded.

yiyixuxu · 2026-09-09T00:14:12Z

+    def __call__(self, components, block_state: BlockState, i: int, sigma: float):
+        batch_size = block_state.latents.shape[0]
+        transformer_dtype = components.transformer.dtype
+        video_context = self._expand_batch(block_state.connector_prompt_embeds, batch_size).to(transformer_dtype)


can we have a dedicated step to expand all the inputs? like https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/krea2/before_denoise.py#L46

Done in 9d182ea. Added a dedicated EchoInputsStep before latent preparation/the denoising loop. It expands both text branches, the attention mask, first-frame/memory tokens, and memory RoPE coordinates once. Removed the two old expansion helpers. Tests cover shared/per-prompt conditions, cached-input reuse without mutation, and two prompts with two videos each.

yiyixuxu · 2026-09-09T00:15:25Z

+
+
+# auto_docstring
+class EchoDenoiseLoopStep(LoopSequentialPipelineBlocks):


ohh this is typically inside denoise.py (even though it is a multi-block) - sorry if I've confused you earlier

Done in 9d182ea. Thanks for clarifying! Moved EchoDenoiseLoopStep and its loop block dictionary back into denoise.py. The top-level core-denoise sequence remains assembled in modular_blocks_echo.py.

yiyixuxu · 2026-09-09T00:16:27Z

+        latent_num_frames = (block_state.num_frames - 1) // components.vae_temporal_compression_ratio + 1
+        latent_height = block_state.height // components.vae_spatial_compression_ratio
+        latent_width = block_state.width // components.vae_spatial_compression_ratio
+        latents = _unpack_latents(


I think we can move the unpack to core denoise blocks (see #14730)

Done in 9d182ea. Added EchoUnpackLatentsStep at the end of the core-denoise group for both video and audio. The decoders now accept normalized VAE-form tensors, require no geometry inputs, and only denormalize/decode. Initial noise inputs use the same unpacked form. Added zero-sigma round-trip and standalone-decoder tests, and documented the latent boundary.

Gelercatty · 2026-09-15T10:19:18Z

@yiyixuxu Thanks for the follow-up review! All six suggestions are addressed in 9d182ea, with a reply on each thread.

Encoders now return normalized, unpacked VAE latents; packing happens before denoising, and unpacking happens at the end of the core-denoise group. Decoders only denormalize and decode.
Memory conditioning derives its spatial geometry directly from the latent tensors, without height/width inputs.
A dedicated input step expands both text branches and all conditioning tensors once before the denoising loop. The loop wrapper is back in denoise.py.
Updated documentation and regression coverage for standalone blocks, audio normalization, rectangular memory geometry, cached conditioning, multi-prompt/multi-video generation, and latent round trips.

Validation:

Echo CPU suite using the cached public tiny fixture: 35 passed, 12 skipped. The skips cover accelerator-only tests, workflow-map/default tests, and the inapplicable num_images_per_prompt interface test.
Four deterministic 8-step before/after comparisons (unconditioned, memory-conditioned, multi-prompt, and multi-video): video and audio outputs were bitwise identical, with maximum absolute difference 0, including nontrivial audio normalization statistics.
Formatting, copy consistency, quality checks, forward/call docstrings, AI-policy checks, scoped modular auto-docstrings, and git diff checks all passed.

Self-review is complete with no blocking issues found. These checks used the tiny CPU fixture; full production-checkpoint/GPU validation was not rerun for this revision.

Could you please take another look? Thanks!

yiyixuxu

thanks!

yiyixuxu · 2026-09-16T07:17:02Z

@sayakpaul can you do a review and help merge in?

stevhliu

thanks for adding docs!

stevhliu · 2026-09-16T14:58:25Z

+[Echo](https://github.com/jd-opensource/JoyAI-Echo) is a long-video generation model. It adds an optional clean first
+frame, ordered image/audio memory slots, and a stochastic few-step Distribution Matching Distillation (DMD) sampler.


Suggested change

[Echo](https://github.com/jd-opensource/JoyAI-Echo) is a long-video generation model. It adds an optional clean first

frame, ordered image/audio memory slots, and a stochastic few-step Distribution Matching Distillation (DMD) sampler.

[Echo](https://github.com/jd-opensource/JoyAI-Echo) is a long-video generation model. It supports an optional clean first

frame, ordered image/audio memory slots, and a stochastic few-step Distribution Matching Distillation (DMD) sampler.

stevhliu · 2026-09-16T14:59:01Z

+Echo is implemented as a Modular Pipeline so its text encoding, memory conditioning, stochastic DMD denoising, and
+decoding blocks can be run as a complete workflow or composed independently.


Suggested change

Echo is implemented as a Modular Pipeline so its text encoding, memory conditioning, stochastic DMD denoising, and

decoding blocks can be run as a complete workflow or composed independently.

Echo is implemented as a [`ModularPipeline`] so its text encoding, memory conditioning, stochastic DMD denoising, and

decoding blocks can be run as a complete workflow or composed independently.

stevhliu · 2026-09-16T15:00:07Z

+Modular workflows can cache the VAE encoder's normalized, unpacked tensors: video latents have shape
+`(batch, channels, frames, height, width)` and audio latents have shape `(batch, channels, time, mel_bins)`.


Suggested change

Modular workflows can cache the VAE encoder's normalized, unpacked tensors: video latents have shape

`(batch, channels, frames, height, width)` and audio latents have shape `(batch, channels, time, mel_bins)`.

Modular workflows can cache the VAE encoder's normalized, unpacked tensors. Video latents have shape

`(batch, channels, frames, height, width)` and audio latents have shape `(batch, channels, time, mel_bins)`.

stevhliu · 2026-09-16T15:01:34Z

+The core `denoise` block packs these tensors, expands conditioning for `num_videos_per_prompt`, and unpacks its
+denoised outputs back to the same VAE form. Pass initial `latents` and `audio_latents` in this unpacked form as well.
+Decoders accept normalized VAE tensors without separate geometry arguments and denormalize immediately before
+decoding. `output=["latents", "audio_latents"]` returns normalized VAE tensors; `output_type="latent"` with
+`output=["videos", "audio"]` returns the denormalized VAE tensors without decoding.


Suggested change

The core `denoise` block packs these tensors, expands conditioning for `num_videos_per_prompt`, and unpacks its

denoised outputs back to the same VAE form. Pass initial `latents` and `audio_latents` in this unpacked form as well.

Decoders accept normalized VAE tensors without separate geometry arguments and denormalize immediately before

decoding. `output=["latents", "audio_latents"]` returns normalized VAE tensors; `output_type="latent"` with

`output=["videos", "audio"]` returns the denormalized VAE tensors without decoding.

The core `denoise` block packs those tensors, expands conditioning for `num_videos_per_prompt`, and unpacks

denoised outputs back to the same VAE form. Pass initial `latents` and `audio_latents` in that unpacked form too.

Decoders take normalized VAE tensors and denormalize immediately before

decoding. `output=["latents", "audio_latents"]` returns normalized VAE tensors. With `output_type="latent"`and

`output=["videos", "audio"]`, you get the denormalized VAE tensors without decoding.

stevhliu · 2026-09-16T15:01:53Z

+)
+```
+
+The default DMD sigma schedule is the released eight-step schedule. It predicts x0 at every step and re-noises with


Suggested change

The default DMD sigma schedule is the released eight-step schedule. It predicts x0 at every step and re-noises with

The default DMD sigma schedule is the released eight-step schedule. It predicts `x0` at every step and re-noises with

Gelercatty · 2026-09-17T06:44:52Z

Thanks @yiyixuxu and @stevhliu for the reviews! I've applied all five documentation suggestions in 90d0c3d, with minor whitespace and line-wrapping fixes. This update is documentation-only; documentation formatting, TOC checks, repository guide checks, and git diff --check pass.

Add Echo modular pipeline

a256a7b

github-actions Bot added documentation Improvements or additions to documentation tests modular-pipelines utils size/L PR with diff > 200 LOC labels Sep 3, 2026

yiyixuxu added this to Diffusers Roadmap Sep 4, 2026

github-project-automation Bot moved this to In Progress in Diffusers Roadmap Sep 4, 2026

yiyixuxu reviewed Sep 4, 2026

View reviewed changes

Address Echo modular pipeline review feedback

64f423a

github-actions Bot added the fixes-issue label Sep 7, 2026

Inline Echo input list normalization

679db3f

yiyixuxu reviewed Sep 9, 2026

View reviewed changes

yiyixuxu added this to the Release 0.41.0 milestone Sep 14, 2026

Align Echo latent boundaries and batch expansion

9d182ea

Gelercatty requested a review from yiyixuxu September 15, 2026 10:19

yiyixuxu approved these changes Sep 16, 2026

View reviewed changes

yiyixuxu requested a review from sayakpaul September 16, 2026 07:15

yiyixuxu added the close-to-merge label Sep 16, 2026

yiyixuxu requested a review from stevhliu September 16, 2026 07:37

stevhliu approved these changes Sep 16, 2026

View reviewed changes

Polish Echo documentation per review

90d0c3d

		latent_height = block_state.height // components.vae_spatial_compression_ratio
		latent_width = block_state.width // components.vae_spatial_compression_ratio

	latent_height = block_state.height // components.vae_spatial_compression_ratio
	latent_width = block_state.width // components.vae_spatial_compression_ratio
	_, _, latent_height, latent_width = memory_video_latents[0].shape

		InputParam.template("height", default=512),
		InputParam.template("width", default=704),



		# auto_docstring
		class EchoDenoiseLoopStep(LoopSequentialPipelineBlocks):

		[Echo](https://github.com/jd-opensource/JoyAI-Echo) is a long-video generation model. It adds an optional clean first
		frame, ordered image/audio memory slots, and a stochastic few-step Distribution Matching Distillation (DMD) sampler.

		Echo is implemented as a Modular Pipeline so its text encoding, memory conditioning, stochastic DMD denoising, and
		decoding blocks can be run as a complete workflow or composed independently.

		Modular workflows can cache the VAE encoder's normalized, unpacked tensors: video latents have shape
		`(batch, channels, frames, height, width)` and audio latents have shape `(batch, channels, time, mel_bins)`.

	The default DMD sigma schedule is the released eight-step schedule. It predicts x0 at every step and re-noises with
	The default DMD sigma schedule is the released eight-step schedule. It predicts `x0` at every step and re-noises with

Conversation

Gelercatty commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Testing

AI-assisted self-review

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Sep 4, 2026

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gelercatty commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gelercatty commented Sep 15, 2026

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Sep 16, 2026

Uh oh!

stevhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gelercatty commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Gelercatty commented Sep 3, 2026 •

edited

Loading

Gelercatty commented Sep 7, 2026 •

edited

Loading