Objects treated as missing despite being present, due to race with geometric repacking - #2207
Open
newren wants to merge 2 commits into
Open
Objects treated as missing despite being present, due to race with geometric repacking#2207newren wants to merge 2 commits into
newren wants to merge 2 commits into
Conversation
When objects involved in the merge cannot be read, the merge machinery will return early with result.clean = -1, and result.tree left as NULL. pick_regular_commit() tested only "if (!result->clean)", ignoring the case where "clean < 0". That causes the code to try to use result->tree, resulting in a SIGSEGV. Handle clean < 0 explicitly; the merge machinery will already have printed messages such as "Could not read <object>" and "collecting merge info failed for trees...", so we don't need to add much detail beyond the fact that the merge failed. Signed-off-by: Elijah Newren <newren@gmail.com>
When a geometric repack runs concurrently with other git processes, it can write a new pack and multi-pack-index and then delete older packs that the new one subsumes. One or more of those older packs may have been indexed by the previous multi-pack-index. A process that already had the previous multi-pack-index open keeps using it, and that stale index still records the removed pack(s) as owning some objects. Because a multi-pack-index attributes each object to exactly one pack, an object that exists in multiple covered packs is served only through its recorded owner. If that owner is the pack a concurrent repack just removed, find_pack_entry() cannot serve the object: fill_midx_entry() routes the lookup to the missing pack (prepare_midx_pack() fails), and the regular pack fallback deliberately skips every multi-pack-index covered pack. The object is reported missing even though a perfectly good copy survives in another covered pack -- for example a large "base" pack that geometric repacking intentionally kept. The false negative is not limited to one caller. Any reader (cat-file, rev-list, pack-objects, ...) can spuriously fail with "unable to read object", and callers that only ask whether an object exists get a wrong answer too, since the OBJECT_INFO_QUICK path never retries. Writers that merge in-core, such as "git replay", are hit hardest: merge-ort treats the unreadable tree as a premature abort, sets result.clean < 0, and returns without a result tree. Teach find_pack_entry() to recover. After the normal multi-pack-index lookup and the regular pack fallback both miss, check whether the object is nonetheless present in a covered multi-pack-index (bsearch_midx()). If it is, its recorded owner must have become unavailable, so scan that index's packs directly for a surviving copy. The bsearch gate keeps genuine misses (i.e. objects absent from the index) on the fast path, and because the recovery lives in find_pack_entry() itself it also fixes the OBJECT_INFO_QUICK callers that never reprepare. This recovers the object without touching the multi-pack-index itself. Reloading the stale index would be a more complete fix but would be much more involved: other code (pack bitmaps, object name disambiguation) borrows and caches the "struct multi_pack_index *" across object reads, so freeing it underneath them would be a use-after-free. Refreshing the index with proper invalidation of those borrowers is left for future work. Signed-off-by: Elijah Newren <newren@gmail.com>
Author
|
/submit |
|
Submitted as pull.2207.git.1787092446.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When an object is found in multiple packs that are in a multi-pack-index, and a subsequent geometric repacking creates a new multi-pack-index and removes the pack that was considered the owner of the object in the old multi-pack-index, then an already-running process that had opened the old multi-pack-index and hadn't yet opened the removed packfile will not be able to access the object -- lookups will return it as missing. Additionally, replay has a separate bug where a missing object causes a SIGSEGV rather than an error message.
This appears to affect a very small percentage of git operations in production since it is a tiny window, but I've found evidence of it occurring in at least eight distinct server-side operations, covering seven different git commands:
There are also commands that could be changing behavior without throwing an error -- e.g. object negotiation thinking an object doesn't exist and instead negotiating based on an older common commit, or cat-file --batch reporting that some objects don't exist.
This series fixes the replay bug first, since it's simpler; investigating it, together with my other recent repacking work, is what led me to the underlying multi-pack-index issue that 2/2 addresses.