Tune Maven's own retry/concurrency to reduce Maven Central rate-limiting - #1835
Tune Maven's own retry/concurrency to reduce Maven Central rate-limiting#1835ramakrishnap-nv wants to merge 3 commits into
Conversation
java-static-test has repeatedly hit 429 Too Many Requests resolving plugins like maven-source-plugin from a cold repository -- the cuopt_mvn wrapper's retry loop (fixed in #1823) retries the whole mvn invocation with backoff, but that's compensating for Maven's own resolver never being tuned, and 4 attempts don't reliably outlast a sustained rate limit. cuDF and cuVS already carry this exact fix for their own Java/Maven Central builds: a project-level .mvn/maven.config (auto-applied to every mvn invocation, no wrapper needed) that caps concurrent downloads to reduce burst request rate and adds a real backoff inside Maven's own transport-layer retry handler, rather than only retrying around the outside of a failed process: -Daether.connector.basic.downstreamThreads=1 -Daether.transport.http.retryHandler.count=5 -Daether.transport.http.retryHandler.interval=10000 -Dmaven.wagon.http.retryHandler.count=5 cuopt_mvn's own -D flags target the connector-layer retry handler, which recent Maven resolver versions may no longer consult now that retry logic lives at the transport layer -- this adds the layer that actually gets read, verified via `mvn help:evaluate -Dexpression=aether.transport.http.retryHandler.interval` resolving to 10000. Verified the packaged-jar-tests suite still passes with this config present.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthroughThe Java build now runs Maven from ChangesMaven configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The PR only tunes Maven download concurrency and retry backoff; no actionable merge-blocking risk remains after normal checks and review. Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files. (1 skipped: 1 unsupported.) ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
CI Test Summary✅ 21 passed · 10 cancelled / not completed |
java/cuopt/.mvn/maven.config is only auto-discovered when Maven's directory search (which walks up from the current working directory, not from -f's directory) reaches java/cuopt. build.sh invoked cuopt_mvn with -f but never changed into that directory, so the transport-layer retry/backoff settings added for the Maven Central 429 rate-limiting never took effect. test.sh already cd's into MODULE_DIR for the same reason; mirror that here, matching how cuDF cd's into java/ before its mvn calls for the same .mvn/maven.config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…y mvn test/verify maven-source-plugin and maven-javadoc-plugin were declared unconditionally in <build><plugins>, so every `mvn test`/`verify` invocation -- including ci/test_java.sh's, which never packages anything for publishing -- still had to resolve maven-source-plugin:3.3.1 from Maven Central on every run. That's the same artifact that's repeatedly failed on 429 in the java-static-test investigation on #1818, and there's no reason java-build's plain test path should pay for it either. No script on this branch currently packages a JAR for real publishing (only java/cuopt/scripts/test.sh's `mvn verify`, for testing, and build.sh's `mvn clean package`, for local dev -- neither needs sources/javadoc jars), so nothing needs the new attach-source-javadoc profile activated; this is a pure reduction in unnecessary Maven Central resolution. Verified locally: `mvn compile` no longer resolves maven-source-plugin/maven-javadoc-plugin at all (checked via -X debug output).
|
/ok to test af54fa7 |
Summary
java-static-test(from Explore self-contained Java classifier JARs with a statically linked libcuopt #1818) and, less often,java-buildhave hit429 Too Many Requestsfrom Maven Central while resolving plugins likemaven-source-pluginon a cold repository.cuopt_mvn's retry loop (fixed in Fix cuopt_mvn retry loop being silently skipped under set -e #1823 to actually run underset -e) retries the wholemvninvocation with backoff, but that's compensating for Maven's own resolver never being tuned for CI -- 4 attempts don't reliably outlast a sustained rate-limit window, and by the time the outer wrapper retries, the burst of parallel requests that likely triggered the 429 in the first place repeats..mvn/maven.config, auto-applied to everymvninvocation with no wrapper script needed, that caps concurrent downloads and adds a real backoff inside Maven's own transport-layer retry handler.Fix
Add
java/cuopt/.mvn/maven.config, matching cuDF's (java/.mvn/maven.config) and cuVS's (java/cuvs-java/.mvn/maven.config) content exactly:aether.connector.basic.downstreamThreads=1caps Maven's own concurrent download threads, reducing the burst of parallel requests against Maven Central that likely triggers the rate-limiting in the first place.aether.transport.http.retryHandler.interval=10000adds a real 10s backoff inside Maven's own resolver, at the transport layer.cuopt_mvn's existing-Dflags (java/cuopt/scripts/maven.sh) target the connector-layer retry handler (aether.connector.http.retryHandler.*), which recent Maven resolver versions may no longer consult now that retry logic lives at the transport layer -- this adds the layer that actually gets read. Verified via:cuopt_mvn's outer shell-level retry loop is left in place as a second layer -- it's still useful for failures Maven's own retry can't cover (network drops mid-request, etc).Test plan
maven-help-plugin:evaluate(see above).packaged-jar-testsMaven Java suite locally with this config present; passes cleanly.java-buildandjava-static-testCI run clean without hitting Maven Central 429s.Split out of #1818, where this was found while investigating an unrelated flaky Surefire crash -- this fix is independently useful for the existing
java-buildjob today, not specific to that PR's self-contained classifier JAR work.