feat(storage): implement move command#814
Open
nhedger wants to merge 3 commits intoexoscale:masterfrom
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Disclosure:
Description
Closes #604
This PR adds
exo storage move(mv) to move objects within a bucket or across buckets without downloading them locally.The implementation uses server-side copy + delete for regular objects, and multipart server-side copy for large objects. It also preserves object metadata, headers, and ACLs across the move.
A few notes to help review:
cmd/storage/storage_move.goadds the new CLI command, argument parsing, dry-run / verbose behavior, recursive prefix support, and a--concurrencyflag for multipart copy workerspkg/storage/sos/move.goadds the storage-layer move implementation for both single-object and recursive moves, including multipart copy + abort-on-failure handlingpkg/storage/sos/acl.gorefactors ACL translation so the same ACL preservation logic can be reused for bothCopyObjectandCreateMultipartUploadpkg/storage/sos/object.goextracts multipart part-size calculation into a shared helper so upload and move use the same dynamic sizing logic and stay under the 10,000-part limitpkg/storage/sos/s3api.go,pkg/storage/sos/s3api_mock_test.go, andpkg/storage/sos/move_test.goextend the S3 abstraction and test coverage forHeadObject, multipart copy, abort handling, and configurable concurrencyOpen questions
--concurrencyremain a user-facing flag for multipart moves? While it can improve large-object move times, higher values may put additional strain on the object storage backend and increase request pressure on the infrastructure.Checklist
(For exoscale contributors)
CHANGELOG.md)Testing
go test ./...>5 GiBcross-bucket move path