NIFI-16125 Parallelize initial component validation at startup - #11513
NIFI-16125 Parallelize initial component validation at startup#11513IPL wants to merge 3 commits into
Conversation
exceptionfactory
left a comment
There was a problem hiding this comment.
Thanks for proposing this improvement @IPL.
This is a critical part of the framework initialization, so it will require careful consideration.
On initial review, it seems better to have two different Task implementations, versus conditional behavior based on the presence of the ExecutorService. That would clarify the expected behavior at FlowController initialization, versus background scheduling.
|
@exceptionfactory Thanks for the feedback — pushed a follow-up commit that splits this into two |
…iggerValidationTask Addresses review feedback from exceptionfactory: TriggerValidationTask previously branched on whether an ExecutorService was supplied to decide between the one-time parallel startup sweep and the periodic serial re-validation. ParallelTriggerValidationTask now owns the startup sweep exclusively; TriggerValidationTask is restored to its original serial-only behavior for the periodic re-validation task.
Summary
NIFI-16125
FlowController.initializeFlow()performs the initial validation of every component (ControllerServices, Reporting Tasks, Flow Analysis Rules, Parameter Providers, Processors, Flow Registry
Clients, and Connectors) sequentially on a single thread via
TriggerValidationTask. For flowswith a large number of components, this can make up a significant portion of startup time.
Per review feedback, this is now split into two dedicated
Runnableimplementations instead ofbranching a single class on whether an
ExecutorServicewas supplied:TriggerValidationTaskis restored to its original serial-only behavior and continues to servethe periodic re-validation task scheduled every 5 seconds.
ParallelTriggerValidationTaskis a new class dedicated to the one-time initial sweep performedduring
FlowControllerinitialization. It submits each component's and Connector's validation tothe existing
validationThreadPooland waits for all of them to complete before returning.Validating distinct components concurrently is safe because each
ComponentNodetracks its ownvalidation state independently.
Both classes share a small package-private helper,
ValidatableComponents, that gathers the sameset of components (Controller Services, Reporting Tasks, Flow Analysis Rules, Parameter Providers,
Processors, Flow Registry Clients, and Connectors) for both sweeps, so the two Task implementations
cannot silently drift apart on which components they validate. Happy to inline this back into both
classes instead if you'd rather they stay fully independent.
ParallelTriggerValidationTask.isValidationComplete()reports whether the most recent run actuallyvalidated every component. It returns
falseif the run was interrupted, or the executor rejectedwork, before every component could be validated —
FlowControllerlogs a warning in that caseinstead of unconditionally logging success, avoiding a silent partial-completion that the previous
implementation would not have surfaced.
Unit tests were split accordingly:
TriggerValidationTaskTestcovers the serial path (unchangedbehavior), and the new
ParallelTriggerValidationTaskTestcovers the parallel happy path, executorrejection, interruption while awaiting completion, and the constructor's null-argument checks.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation