Enhance tracing and refactor blob-related keyspaces#238
Conversation
# Conflicts: # operations/src/replication/incoming_version_replication.rs # operations/src/replication/version_replication.rs
St4NNi
left a comment
St4NNi
left a comment
There was a problem hiding this comment.
Just a few minor comments, otherwise this looks ok for me.
| "blake3", | ||
| )); | ||
| }; | ||
| let effect = match write_blob_location_effect( |
There was a problem hiding this comment.
Isn't this an issue if the hash already exists ? It just overwrites the metadata without taking existing metadata into account.
| group_id: Ulid, | ||
| requested_restrictions: Option<Vec<CreateS3PathRestriction>>, | ||
| ) -> ServerResult<Option<Vec<PathRestriction>>> { | ||
| let Some(requested_restrictions) = requested_restrictions else { |
There was a problem hiding this comment.
This could be a problem because it drops the restrictions from the auth-context and thus may allow for a broader scope than original granted.
| key_space: BLOB_VERSIONS_KEYSPACE.to_string(), | ||
| prefix: Some(prefix), | ||
| start_after: None, | ||
| limit: 10_000, |
There was a problem hiding this comment.
Unrelated to this PR but when reading through this: What if an object has more than 10000 versions -> This will not work!
| } | ||
|
|
||
| self.state = DeleteBucketState::CheckMultipartUploads; | ||
| smallvec![Effect::Storage(StorageEffect::Iter { |
There was a problem hiding this comment.
This has the same issue as before, it only scan suntil SCAN_LIMIT but what if there is an active multipart beyond the scan_limit ?
| } | ||
|
|
||
| fn defined_keyspaces() -> [&'static str; 18] { | ||
| fn defined_keyspaces() -> [&'static str; 19] { |
There was a problem hiding this comment.
No explores for multipart keyspaces ?
3 participants
This PR combines two main lines of work:
Tracing And Observability
This part of the branch improves end-to-end visibility and debugging across the system.
Blob Keyspace Refactor
This part of the branch changes how blob and object state is modeled and stored.
BLOB_LOCATIONS_KEYSPACE,BLOB_HEAD_KEYSPACE,BLOB_VERSIONS_KEYSPACE, andHASH_PATHS_INDEX_KEYSPACEBlobHeadKey,BlobVersion,HashPathIndexKey, and head-transition utilitiesaruna-doctorto inspect and debug the new blob-related keyspacesSupporting Changes
A few additional changes support the two themes above.