Materialize composed config snapshots in KFP - #773
Merged
Merged
Conversation
…mposition # Conflicts: # pyproject.toml
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop a redundant genericity test and an implementation-pinning assertion, and assert compiled-pipeline wiring structurally instead of matching the KFP compiler's text layout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Forcing enable_caching=False re-ran composition on every relaunch, so a retry could overwrite the job's resolved snapshots with different dynamically resolved values while downstream cached steps kept the old ones. Recompose by launching with a new job name or with run-level caching disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every YAML read now composes; plain configs and materialized snapshots compose to themselves. compose_yaml_config snapshots and restores any foreign GlobalHydra context instead of refusing to run, so reads work inside a user application under @hydra.main. Local files not named *.yaml are staged like remote URIs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
proto_to_yaml re-serializes the parsed protobuf, so source comments cannot survive composition. Prepend a provenance comment naming the source config instead, prefixed with the docker image when the source is a container-local path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mkolodner-sc
approved these changes
Sep 28, 2026
mkolodner-sc
left a comment
mkolodner-sc
left a comment
Collaborator
There was a problem hiding this comment.
Thanks Kyle! Few small comments/questions
![semgrep-code-snapchat[bot]](https://avatars.githubusercontent.com/u/40505220?s=60&v=4){kind=small}
…p-config-snapshots # Conflicts: # docs/user_guide/config_guides/hydra_composition.md
yliu2-sc
approved these changes
Sep 29, 2026
kmontemayor2-sc
marked this pull request as ready for review
September 29, 2026 23:54
kmontemayor2-sc
enabled auto-merge
September 29, 2026 23:54
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 30, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 30, 2026
kmontemayor2-sc
enabled auto-merge
September 30, 2026 15:40
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolve KFP task and resource configs in ConfigValidator and pass snapshots downstream, so pipeline components consume
the same resolved values. The validator also derives the GLT backend decision from that task config.
Part 2 of 3, based on #726. E2E config adoption follows separately.
Approach
resolved protobufs.
User-facing changes
ConfigValidator's CLI and component inputs change from
task_config_uriandresource_config_uritosource_task_config_uriandsource_resource_config_uri. Other pipeline components retain their existing input names.The validator CLI additionally requires:
--output_file_path_composed_task_config_snapshot_uri--output_file_path_composed_resource_config_snapshot_uri--output_file_path_should_use_glt_backendThe corresponding component outputs contain the two snapshot URIs and GLT decision. The Python
kfp_validation_checkshelper uses the renamed source arguments and returns the resolved configs plus the backend decision.
check_pipeline_has_valid_start_and_stop_flagsacceptsgbml_config_wrapperinstead oftask_config_uri.The separate
glt_backendcomponent helper is removed.Snapshots are written under
<temp-regional-bucket>/<job_name>/config_validator/. Cache hits reuse prior validatoroutputs. Reexecution recomposes sources and can overwrite snapshots for the same job name; use distinct names for runs
that need distinct snapshot paths.
Verification