Repository navigation
[OMEGA-482] Fix: Migrate memory transfer to Docker volumes - #397
Open
blackhammer116 wants to merge 8 commits into
Open
blackhammer116 wants to merge 8 commits into
blackhammer116 wants to merge 8 commits into
Conversation
Store archives in omega-memory-transfer at /memory-transfer and expose memory-list, memory-upload, and memory-download through the launcher. Prepare private volume access, stage uploads before publication, and preserve exported archives when cleaning live memory. Remove the obsolete host-group setup from the entrypoint. BREAKING CHANGE: Replace --memory-transfer-dir and --memory-transfer-gid with --memory-transfer-volume. Upload existing host archives before import.
Verify named volume mounts, archive commands, private permissions, duplicate protection, and upload failure cleanup. Cover startup import preflight and preserve the existing agent when an import is missing.
Describe archive storage, launcher transfer commands, startup import, private permissions, and migration from host bind mounts. Document the Windows workflow as WSL2 with Docker Desktop.
…-volumes [OMEGA-482] Fix: Migrate memory transfer to Docker volumes
paul-v-snet
requested changes
Oct 16, 2026
| ${log_config_volume[@]+"${log_config_volume[@]}"} | ||
| ${memory_transfer_volume[@]+"${memory_transfer_volume[@]}"} | ||
| ${memory_transfer_env[@]+"${memory_transfer_env[@]}"} | ||
| --mount "type=volume,source=${memory_transfer_volume_name},target=/memory-transfer" |
Collaborator
There was a problem hiding this comment.
The volume is always mounted, although the docs say:
--enable-memory-exportis required because export is disabled by default.
I suggest mounting the volume only when --enable-memory-export is passed.
Comment on lines
+801
to
+829
| prepare_memory_transfer_volume() { | ||
| docker run --rm --pull=never --network none --read-only --user 0:0 \ | ||
| --entrypoint /bin/sh \ | ||
| --mount "type=volume,source=${memory_transfer_volume_name},target=/memory-transfer" \ | ||
| "${image}" -c 'chown 65534:65534 /memory-transfer && chmod 0700 /memory-transfer' | ||
| } | ||
|
|
||
| require_memory_transfer_volume() { | ||
| if ! docker volume inspect "${memory_transfer_volume_name}" >/dev/null 2>&1; then | ||
| echo "Memory transfer volume not found: ${memory_transfer_volume_name}" >&2 | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| memory_transfer_run() { | ||
| local mode="${1}" | ||
| shift | ||
| local mount="type=volume,source=${memory_transfer_volume_name},target=/memory-transfer" | ||
| local input=() | ||
| if [[ "${mode}" == "ro" ]]; then | ||
| mount+=",readonly" | ||
| else | ||
| input=(-i) | ||
| fi | ||
| docker run --rm --pull=never --network none --read-only --user 65534:65534 \ | ||
| --security-opt no-new-privileges:true \ | ||
| ${input[@]+"${input[@]}"} --mount "${mount}" \ | ||
| --entrypoint python3 "${image}" -B "$@" | ||
| } |
Collaborator
There was a problem hiding this comment.
Why --pull=never used here?
Potential case: a user transfers an archive to another machine and then runs:
image="singularitynet/omega:<tag>"
archive="memory-archive.tar.gz"
scripts/omega memory-upload "$HOME/Downloads/$archive" -d "$image"In this case, the image doesn't exist on that machine, and these commands won't pull it automatically, so the memory-upload command (as well as memory-list) will fail with a No such image error.
I suggest automatically pulling images passed via -d.
Comment on lines
+508
to
+510
| echo -e "\tmemory-list list available memory archives" | ||
| echo -e "\tmemory-upload <host-archive> copy an archive into the transfer volume" | ||
| echo -e "\tmemory-download <filename> <host-destination> copy an archive to your computer" |
Collaborator
There was a problem hiding this comment.
I suggest adding a memory-delete command.
Currently, there is only one way to delete an archive - by using docker volume rm, but this will remove all archives. In addition, archives accumulate indefinitely, and I think this is not an optimal approach.
So I believe a command like memory-delete is necessary.
Co-authored-by: paul-v-snet <paul.vasiletz@singularitynet.io>
Mount the transfer volume at startup only when export is enabled or an import is requested. Let archive helpers pull missing images while reusing locally available images. Add memory-delete to remove one validated archive without removing the transfer volume or other archives.
Check export/import mount combinations, helper image availability, pull-failure cleanup, and single-archive deletion. Cover invalid filenames, non-regular targets, custom volumes, and live-memory preservation. Extend the Docker transfer round-trip to verify that deleting one archive preserves another downloadable archive.
Clarify when the agent mounts the archive volume and how missing helper images are pulled. Document memory-delete and update the README restore workflow.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Move memory archives from host bind mounts to a Docker volume to address the reported export permission failures with Landlock on macOS.
How Has This Been Tested?
On the current PR branch, 64 tests passed. One optional Docker integration test was skipped. Shell syntax and whitespace checks also passed.
The tests cover archive commands, volume mounts, duplicate uploads, interrupted uploads, failed-download cleanup, startup import checks, and archive preservation during cleanup.
Checklist