Skip to content

Add composable distillation loss hooks - #44

Draft
CharlelieLrt wants to merge 1 commit into
NVlabs:mainfrom
CharlelieLrt:codex/distillation-loss-hooks
Draft

CharlelieLrt wants to merge 1 commit into
NVlabs:mainfrom
CharlelieLrt:codex/distillation-loss-hooks

Conversation

@CharlelieLrt

@CharlelieLrt CharlelieLrt commented Oct 14, 2026 •

Copy link
Copy Markdown

Description

This PR adds optional loss hooks for users who integrate FastGen into an external
training loop. Shared reductions support masking and weighting for sCM, MeanFlow,
DSM, VSD, and adversarial losses. DMD2 and f-distill callers can select a training
phase explicitly, and ratio helpers accept caller-owned bins and a process group:

losses, outputs = model.single_train_step(batch, iteration, phase="auxiliary")
h, bins = f_divergence_weights(
    scores, t, weighting_fn, bins=bins,
    logit_weight=score_weight, statistics_group=data_group,
)

Existing calls keep their defaults, and the external loop continues to own
optimizers and checkpoints. TRAINING_INTEGRATION.md describes the masking
contracts and extension points. PhysicsNeMo can then delegate shared loss math
to FastGen instead of maintaining private compatibility reductions.

Dependencies

  • Depends on

    • This PR starts from FastGen main and has no prerequisite PR.
  • Depends on this PR

    • PhysicsNeMo #2100 replaces local compatibility code with the proposed FastGen loss hooks. It depends on acceptance of these hooks.
  • Independent PhysicsNeMo stack

  • Independent packaging proposal

    • FastGen #43 allows the NumPy 2 dependency used by PhysicsNeMo. It does not depend on these hooks.

If FastGen maintainers decline FastGen #44, we would close PhysicsNeMo #2100.
The preceding PhysicsNeMo stack would remain available.

Review process

Reviewers can compare this PR with its immediate parent. After a parent merges,
we update the remaining branches and their PR bases to keep each diff focused.

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant