Repository navigation
build: parallelize static-library compilation; flat static_variant lookup - #229
Merged
Merged
Conversation
…ncy libraries With the Makefile generator every target waits for the targets it links to be fully archived before any of its own objects start. Static libraries do not link, so this is pure serialization: nothing in graphene_chain starts until fc and protocol finish, and no plugin starts until database.cpp (the longest TU) finishes alone on one core. CMAKE_OPTIMIZE_DEPENDENCIES drops those edges for static libraries while keeping dependencies on custom-command outputs and anything added through add_dependencies. graphene_chain therefore keeps only build_hardfork_hpp as an explicit dependency; the protocol/utilities edges already come from target_link_libraries and were defeating the optimization. The generated hardfork.hpp is no longer listed as a source for the same reason. Cold make -j8 vizd cli_wallet on an 8-core laptop: 805 s -> 305 s wall, user time 1762 s -> 1375 s, i.e. ~2.4 -> ~4.8 cores kept busy. A full make of every target, 13/13 fast ctest and consensus_sim_tests pass on the pruned graph.
…rips Bumps fc to the non-recursive static_variant position lookup (fc#6): ops.cpp 8.4 s -> 6.7 s, objects byte-identical. Adds an operation_variant consensus_sim suite that, for each of the 107 alternatives, sets the tag, raw-packs and unpacks, converts to and from fc::variant, and copies/moves/assigns, checking which() throughout; pins the tag numbers of vote (0), transfer (2) and validator_equivocation (count-1); and checks that out-of-range tags are rejected both via set_which and via raw unpack.
Member
Author
|
CI timing note: the job durations on this PR (Docker build 10 min; tests 15/15/10 min) match PR #227's run (14/15/10 min). CI job time is dominated by ccache state, test runtime and image layers, not by compile ordering, so the win reported above is for cold/uncached builds (local dev, Docker production image, cache misses after a header change). No regression on CI either. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two build-time changes, one with a large wall-clock effect and one small frontend win, plus a test.
1.
CMAKE_OPTIMIZE_DEPENDENCIES ON(rootCMakeLists.txt,libraries/chain/CMakeLists.txt)With the Unix Makefiles generator a target does not start compiling until every target it links is fully archived. For static libraries that is pure serialization: nothing in
graphene_chainstarts untilfcandgraphene_protocolare done, and no plugin starts untildatabase.cpp(the longest TU, ~80 s at -O2) finishes alone on one core. Executables still wait for everything they link.CMAKE_OPTIMIZE_DEPENDENCIES(CMake >= 3.19; CI runs 3.28) removes those edges for static libraries while keeping dependencies on custom-command outputs and anything declared throughadd_dependencies. Two things inlibraries/chainwere defeating it and are removed:add_dependencies(graphene_chain graphene_protocol graphene_utilities ...): the library-to-library edges already come fromtarget_link_libraries; onlybuild_hardfork_hppis a real build-order dependency.hardfork.hpplisted as a source ofgraphene_chain.Resulting graph (checked in the generated
Makefile2):graphene_protocoldepends onproject_secp256k1only;graphene_chainand every plugin depend onproject_secp256k1andbuild_hardfork_hpponly;vizdkeeps all 33 of its dependencies.Cold
make -j8 vizd cli_wallet, same tree, same flags, M1 laptop:Caveats: a fanless laptop throttles under a long serialized tail, which inflates the baseline; 4-vCPU CI runners will gain less because they have less idle capacity to fill. The direction is not in doubt: the critical path was
fc -> protocol -> database.cpp -> plugins -> link, and it is nowdatabase.cpp -> link.Ninja users are unaffected (Ninja already orders static libraries this loosely).
2. fc submodule bump to VIZ-Blockchain/fc#6
static_variant'sposition/type_infohelpers were recursive class templates, so every TU that names all 107operationalternatives paid O(N²) instantiations in the frontend. They are nowconstexprfunctions over a pack-expanded array. Dispatch is unchanged;operations.ois byte-identical before and after, and a round-trip digest over every alternative is identical.ops.cpp8.4 s -> 6.7 s;database.cppunchanged (it is backend-bound).A jump-table dispatch was tried and rejected: it tripled
operations.oand sloweddatabase.cpp~15%. Details in the fc PR.3.
tests/consensus_sim/scenarios/test_operation_variant.cppNew
operation_variantsuite: for each of the 107 alternatives, set the tag, raw pack/unpack, to/fromfc::variant, copy/move/assign, checkingwhich()throughout; pinsvote=0,transfer=2,validator_equivocation=count-1; out-of-range tags rejected viaset_whichand via raw unpack. This is the guard for any future change tostatic_variantor to the order ofoperations.hpp.Checks
makeof every target on the pruned graph: exit 0 (confirms no missing generated-file dependency).ctest -E '^consensus_sim$': 13/13 pass.consensus_sim_tests -t operation_variant: pass; whole binary links and lists 135 cases.-std=c++14 -Wall -Wextra.vizd --versionfrom the optimized build runs.Merge order
fc#6 first, then this PR (the submodule pointer here already targets the fc commit).