Skip to content

NIFI-16125 Parallelize initial component validation at startup - #11513

Open
IPL wants to merge 3 commits into
apache:mainfrom
IPL:nifi-16125
Open

NIFI-16125 Parallelize initial component validation at startup#11513
IPL wants to merge 3 commits into
apache:mainfrom
IPL:nifi-16125

Conversation

@IPL

@IPL IPL commented Aug 6, 2026

Copy link
Copy Markdown

Summary

NIFI-16125

FlowController.initializeFlow() performs the initial validation of every component (Controller
Services, Reporting Tasks, Flow Analysis Rules, Parameter Providers, Processors, Flow Registry
Clients, and Connectors) sequentially on a single thread via TriggerValidationTask. For flows
with a large number of components, this can make up a significant portion of startup time.

Per review feedback, this is now split into two dedicated Runnable implementations instead of
branching a single class on whether an ExecutorService was supplied:

  • TriggerValidationTask is restored to its original serial-only behavior and continues to serve
    the periodic re-validation task scheduled every 5 seconds.
  • ParallelTriggerValidationTask is a new class dedicated to the one-time initial sweep performed
    during FlowController initialization. It submits each component's and Connector's validation to
    the existing validationThreadPool and waits for all of them to complete before returning.
    Validating distinct components concurrently is safe because each ComponentNode tracks its own
    validation state independently.

Both classes share a small package-private helper, ValidatableComponents, that gathers the same
set of components (Controller Services, Reporting Tasks, Flow Analysis Rules, Parameter Providers,
Processors, Flow Registry Clients, and Connectors) for both sweeps, so the two Task implementations
cannot silently drift apart on which components they validate. Happy to inline this back into both
classes instead if you'd rather they stay fully independent.

ParallelTriggerValidationTask.isValidationComplete() reports whether the most recent run actually
validated every component. It returns false if the run was interrupted, or the executor rejected
work, before every component could be validated — FlowController logs a warning in that case
instead of unconditionally logging success, avoiding a silent partial-completion that the previous
implementation would not have surfaced.

Unit tests were split accordingly: TriggerValidationTaskTest covers the serial path (unchanged
behavior), and the new ParallelTriggerValidationTaskTest covers the parallel happy path, executor
rejection, interruption while awaiting completion, and the constructor's null-argument checks.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
  • Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using ./mvnw clean install -P contrib-check
    • JDK 21
    • JDK 25

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

@exceptionfactory exceptionfactory left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for proposing this improvement @IPL.

This is a critical part of the framework initialization, so it will require careful consideration.

On initial review, it seems better to have two different Task implementations, versus conditional behavior based on the presence of the ExecutorService. That would clarify the expected behavior at FlowController initialization, versus background scheduling.

@IPL

IPL commented Aug 18, 2026

Copy link
Copy Markdown
Author

@exceptionfactory Thanks for the feedback — pushed a follow-up commit that splits this into two
dedicated Task implementations as you suggested: TriggerValidationTask stays serial-only for the
periodic re-validation, and the new ParallelTriggerValidationTask owns the one-time startup sweep.
PTAL when you get a chance.

@IPL
IPL requested a review from exceptionfactory August 18, 2026 12:59
IPL added 3 commits August 20, 2026 22:10
…iggerValidationTask

Addresses review feedback from exceptionfactory: TriggerValidationTask
previously branched on whether an ExecutorService was supplied to decide
between the one-time parallel startup sweep and the periodic serial
re-validation. ParallelTriggerValidationTask now owns the startup sweep
exclusively; TriggerValidationTask is restored to its original serial-only
behavior for the periodic re-validation task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants