Skip to content

Upgrade testing capability for GC simulations#786

Open
hoytak wants to merge 10 commits intomainfrom
hoytak/260407-upgrade-testing
Open

Upgrade testing capability for GC simulations#786
hoytak wants to merge 10 commits intomainfrom
hoytak/260407-upgrade-testing

Conversation

@hoytak
Copy link
Copy Markdown
Collaborator

@hoytak hoytak commented Apr 7, 2026

Currently, delete_xorb lives on DirectAccessClient and performs unconditional deletion — there is no way for GC workflows to do compare-and-delete style operations that guard against deleting an object that has been rewritten since it was listed. This PR moves delete_xorb to DeletionControlableClient and adds conditional deletion APIs for both xorbs and shards, keyed on an opaque 32-byte ObjectTag.

New DeletionControlableClient methods:

list_xorbs_and_tags / list_shards_with_tags — return object hashes paired with content-derived tags.
delete_xorb_if_tag_matches / delete_shard_if_tag_matches — delete only when the current tag matches the provided one, returning whether the delete occurred.
Tags are derived differently per backend: LocalClient hashes filesystem metadata (modified/created timestamps, size, permissions) to detect overwrites; MemoryClient hashes object content. MemoryClient now also fully implements DeletionControlableClient, so in-memory simulation servers wire through deletion routes instead of returning 501.

verify_integrity on LocalClient gains a third pass that validates every shard referenced by the global dedup table still exists on disk, catching stale references that previously went undetected. Corresponding HTTP routes and SimulationControlClient methods are added for the new operations.


Note

Medium Risk
Touches deletion semantics and integrity verification paths and introduces a new on-disk redb table (FILE_TO_SHARD_TABLE), which could affect GC simulations and file visibility if the mapping logic is wrong. Changes are localized to simulation clients/servers but include concurrency-sensitive rename/tag checks and new API contracts.

Overview
Adds compare-and-delete support to simulation GC workflows by moving delete_xorb off DirectAccessClient onto DeletionControlableClient and introducing an ObjectTag plus new list/conditional-delete methods for both XORBs and shards.

Extends the simulation HTTP surface and SimulationControlClient with endpoints for listing hashes with tags and tag-guarded deletes, and wires deletion controls for the in-memory backend (previously returned 501).

Strengthens LocalClient integrity semantics by tracking active file entries in a new FILE_TO_SHARD_TABLE (with byte offsets for direct-seek reads) and expanding verify_integrity() to validate dedup-table shard references exist on disk; updates upload/delete logic and tests accordingly.

Reviewed by Cursor Bugbot for commit 02a5f75. Bugbot is set up for automated code reviews on this repo. Configure here.

@hoytak hoytak requested review from rajatarya and sirahd April 7, 2026 20:05
Comment thread xet_client/src/cas_client/simulation/memory_client.rs
Comment thread xet_client/src/cas_client/simulation/simulation_server.rs
Comment thread xet_client/src/cas_client/simulation/local_client.rs
Comment thread xet_client/src/cas_client/simulation/local_client.rs
Comment thread xet_client/src/cas_client/simulation/memory_client.rs
Comment thread xet_client/src/cas_client/simulation/local_client.rs
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4f07e21. Configure here.

Comment thread xet_client/src/cas_client/simulation/local_client.rs Outdated
Comment thread xet_client/src/cas_client/simulation/local_client.rs
Comment thread xet_client/src/cas_client/simulation/local_client.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant