Skip to content

feat(storage): implement move command#814

Open
nhedger wants to merge 3 commits intoexoscale:masterfrom
nhedger:feat/storage-move
Open

feat(storage): implement move command#814
nhedger wants to merge 3 commits intoexoscale:masterfrom
nhedger:feat/storage-move

Conversation

@nhedger
Copy link
Copy Markdown

@nhedger nhedger commented Mar 24, 2026

Note

Disclosure:

  • AI assistance was used to implement this feature
  • Manual review and testing was done

Description

Closes #604

This PR adds exo storage move (mv) to move objects within a bucket or across buckets without downloading them locally.

The implementation uses server-side copy + delete for regular objects, and multipart server-side copy for large objects. It also preserves object metadata, headers, and ACLs across the move.

A few notes to help review:

  • cmd/storage/storage_move.go adds the new CLI command, argument parsing, dry-run / verbose behavior, recursive prefix support, and a --concurrency flag for multipart copy workers
  • pkg/storage/sos/move.go adds the storage-layer move implementation for both single-object and recursive moves, including multipart copy + abort-on-failure handling
  • pkg/storage/sos/acl.go refactors ACL translation so the same ACL preservation logic can be reused for both CopyObject and CreateMultipartUpload
  • pkg/storage/sos/object.go extracts multipart part-size calculation into a shared helper so upload and move use the same dynamic sizing logic and stay under the 10,000-part limit
  • pkg/storage/sos/s3api.go, pkg/storage/sos/s3api_mock_test.go, and pkg/storage/sos/move_test.go extend the S3 abstraction and test coverage for HeadObject, multipart copy, abort handling, and configurable concurrency

Open questions

  • Should --concurrency remain a user-facing flag for multipart moves? While it can improve large-object move times, higher values may put additional strain on the object storage backend and increase request pressure on the infrastructure.

Checklist

(For exoscale contributors)

  • Changelog updated (under Unreleased block, and add the Pull Request #number for each bit you add to the CHANGELOG.md)
  • Testing

Testing

  • go test ./...
  • verified dry-run behavior for single-object moves
  • verified same-bucket object moves
  • verified cross-bucket object moves
  • verified recursive cross-bucket prefix moves
  • verified live >5 GiB cross-bucket move path
  • verified configurable multipart copy concurrency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: implement a exo storage move

1 participant