Repository navigation
fix(builder): Lock the clone cache against concurrent workers - #658
Open
BourezBastien wants to merge 1 commit into
Open
BourezBastien wants to merge 1 commit into
BourezBastien wants to merge 1 commit into
Conversation
Parallel RQ workers cloning the same app at the same commit race between the os.path.exists cache check and the git init and fetch, which can corrupt the shared clone directory. Hold a per app and commit file lock across the check and the clone so the second worker waits and then hits the cache. filelock is already a dependency, used for the config file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
Author
|
@Mergifyio queue |
Contributor
☑️ Command disallowed due to command restrictions in the Mergify configuration.Details
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel RQ workers cloning the same app at the same commit race between the
os.path.existscache check and thegit init/fetchsequence: the loser re-enters the existing directory and runs git commands over a half-written clone, corrupting it (observed as randomCloneErrors whenever two tenants build the same app version at once; this also blocks the same-machine build batching sketched in frappe/press#7934)._clone_repositorynow holds a per app+commitfilelock.FileLock(600 s timeout) across the cache check and the clone, so the second worker waits and then hits theCACHEDpath.filelockis already a dependency (used for the config file). Lock files are small and flat in the clone directory.No bench was available in this environment; verified by compile and ruff check/format with the repo config.
🤖 Generated with Claude Code