Repository navigation
Conversation
ahresse
force-pushed
the
http-mirror-upload
branch
3 times, most recently
from
August 25, 2026 11:27
e9edcda to
9a9282b
Compare
Contributor
|
Hello @ahresse there are conflicts with master in |
ahresse
force-pushed
the
http-mirror-upload
branch
from
October 27, 2026 17:18
9a9282b to
859c1d9
Compare
Contributor
Author
|
@rsanchez87 : branch rebased |
rsanchez87
force-pushed
the
http-mirror-upload
branch
2 times, most recently
from
November 2, 2026 09:02
93157a0 to
8128038
Compare
rsanchez87
requested changes
Nov 2, 2026
| last_exc = e | ||
| if attempt < retries: | ||
| print(f" WARNING: attempt {attempt + 1} failed for {key}: {e}") | ||
| print(f" ERROR: failed to upload {key}: {last_exc}") |
Contributor
There was a problem hiding this comment.
Bucket mode can never fail: a failed upload is only printed here, so main() still prints "Done. Use the bucket as a mirror with:" and exits 0.
Reproduced with a one-URL suite and an unreachable source and bucket (--rgw-host 127.0.0.1:1 --bucket b -j 1 -r 1):
ERROR: failed to upload 127.0.0.1:9/data.bin: Failed to download ... Connection refused
Done. Use the bucket as a mirror with:
exit=0 (identical with -j 4)
The local-directory path raises through pool.map/worker(url) and exits 1 for the same scenario, so the two backends disagree, and a CI job can publish a green but empty mirror.
Suggestion: return a success flag from _upload_one (True for skipped/uploaded, False here), collect them in main() and sys.exit(1) if any failed , or re-raise the last exception.
mirror_sync can now populate a radosgw (Ceph RGW) bucket in addition to a local directory tree. When --rgw-host is given, each test suite resource is downloaded to a temp dir and uploaded to the bucket via HTTP PUT, using the same <netloc>/<path> key layout as the local mirror so the bucket URL works directly with `fluster download --mirror`. - add --rgw-host and --bucket options (--bucket required with --rgw-host) - add HEAD-based skip-if-exists check to avoid re-uploading objects - refactor sync_urls to dispatch a per-URL worker for both modes - add bucket-mode tests using a stdlib PUT/HEAD http.server handler Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Update the README Local Mirror section to describe the two mirror_sync backends (local directory and radosgw bucket), add a radosgw usage example with --rgw-host/--bucket, and refresh the mirror_sync.py options help output. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
PEP 585 builtin generics (list[str]) are not subscriptable at runtime on Python 3.7, the minimum supported version, causing an ImportError when the module is loaded by tests. Use typing.List instead. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
url_to_mirror_path used os.path.join, producing backslashes on Windows that leaked into the radosgw object key (encoded as %5C), breaking the <netloc>/<path> layout that fluster download --mirror expects. Build the key with forward slashes (matching rewrite_url) and convert to OS separators only when writing the local mirror tree. The fake bucket test handler also stored objects at paths containing the host:port ':' character, which is illegal in Windows paths. Map object keys to filesystem-safe paths in the test harness. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
rsanchez87
force-pushed
the
http-mirror-upload
branch
from
November 3, 2026 10:18
8128038 to
214e6d2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request should be merged after: #386
This pull request adds support for using a radosgw (Ceph RGW) bucket as a mirror backend for test vector resources, in addition to the existing local directory option. It updates both the
scripts/mirror_sync.pyscript and the documentation to describe and implement this new feature, and introduces comprehensive tests for the new bucket upload logic.Major new feature: radosgw (Ceph RGW) bucket mirror support
scripts/mirror_sync.py: Added support for uploading resources to a radosgw bucket over HTTP, including logic for checking object existence, uploading via HTTP PUT, and handling retries. New command-line arguments--rgw-hostand--bucketallow users to specify a bucket as the mirror backend. [1] [2] [3] [4]Documentation updates
README.md: Expanded the documentation to describe how to set up and use a radosgw bucket as a mirror, including example commands and notes on authentication. Updated usage instructions and help text formirror_sync.py. [1] [2] [3]Testing improvements
tests/test_mirror.py: Added new tests for the radosgw bucket upload feature, including a minimal HTTP server to simulate bucket behavior, tests for object upload, skipping existing objects, and URL construction. [1] [2]These changes enable faster and more flexible mirroring of test vectors, especially for distributed or CI environments, by allowing the use of object storage buckets as mirrors.