Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds runner-level isolation for stored outputs by prepending runner_id to S3/Azure output prefixes, and introduces a new store delete command to delete runner-scoped output data.
Changes:
- Add
cosmotech.runner_idas a required key for S3 and Azure output channels and incorporate it into output prefixes. - Implement Azure blob deletion and wire channel delete operations (plus CLI
store deletecommand). - Update unit tests and PostgreSQL runner metadata handling to use
lastRunInfo.lastRunId.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
cosmotech/coal/store/output/az_storage_channel.py |
Prepends runner_id into Azure file_prefix and implements channel delete via Azure blob deletion. |
cosmotech/coal/store/output/aws_channel.py |
Prepends runner_id into S3 prefix and simplifies sqlite upload path. |
cosmotech/coal/azure/blob.py |
Adds delete_azure_blobs() implementation. |
cosmotech/coal/aws/s3.py |
Adds a file_prefix setter to allow channels to update the effective prefix. |
cosmotech/coal/postgresql/runner.py |
Switches runner metadata access to lastRunInfo.lastRunId for delete + return value. |
cosmotech/csm_data/commands/store/store.py |
Registers the new store delete command. |
cosmotech/csm_data/commands/store/delete.py |
New CLI command that triggers output deletion across configured targets. |
tests/unit/coal/test_store/test_output/test_az_storage_channel.py |
Updates Azure output tests for runner-prefixed paths and delete behavior. |
tests/unit/coal/test_store/test_output/test_aws_channel.py |
Updates AWS output tests for runner-prefixed paths and new upload signature usage. |
tests/unit/coal/test_postgresql/test_postgresql_runner.py |
Updates runner metadata fixture shape to include lastRunInfo. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+76
to
+80
| expected_config = base_azure_storage_config | ||
| expected_config.azure.file_prefix = ( | ||
| base_azure_storage_config.cosmotech.runner_id + "/" + base_azure_storage_config.azure.file_prefix | ||
| ) | ||
| mock_dump.assert_called_once_with(expected_config, selected_tables=None) |
Collaborator
Author
There was a problem hiding this comment.
the channel init function create a new configuration object non prepend is happenning
Comment on lines
+95
to
102
| expected_config = base_azure_storage_config | ||
| expected_config.azure.file_prefix = ( | ||
| base_azure_storage_config.cosmotech.runner_id + "/" + base_azure_storage_config.azure.file_prefix | ||
| ) | ||
| mock_dump.assert_called_once_with( | ||
| base_azure_storage_config, | ||
| expected_config, | ||
| selected_tables=["table1", "table2"], | ||
| ) |
Collaborator
Author
There was a problem hiding this comment.
Still, channel init function create a new configuration object no prepend is happening
Comment on lines
108
to
+119
| """Test delete method (should do nothing).""" | ||
| channel = AzureStorageChannel(base_azure_storage_config) | ||
|
|
||
| # Act | ||
| result = channel.delete() | ||
| channel.delete() | ||
|
|
||
| # Assert | ||
| # Should not raise any exception and return None | ||
| assert result is None | ||
| expected_config = base_azure_storage_config | ||
| expected_config.azure.file_prefix = ( | ||
| base_azure_storage_config.cosmotech.runner_id + "/" + base_azure_storage_config.azure.file_prefix | ||
| ) | ||
| mock_delete.assert_called_once_with(expected_config) |
Comment on lines
105
to
+112
| schema_table = f"{_psql.db_schema}.{_psql.table_prefix}RunnerMetadata" | ||
| sql_delete_from_metatable = f""" | ||
| DELETE FROM {schema_table} | ||
| WHERE last_csm_run_id={runner.get("lastRunId")}; | ||
| WHERE last_csm_run_id={runner.get("lastRunInfo").get("lastRunId")}; | ||
| """ | ||
| curs.execute(sql_delete_from_metatable) | ||
| conn.commit() | ||
| return runner.get("lastRunId") | ||
| return runner.get("lastRunInfo").get("lastRunId") |
Comment on lines
+19
to
+23
| help=T("csm_data.commands.store.output.parameters.filter"), | ||
| type=str, | ||
| ) | ||
| @web_help("csm-data/store/output") | ||
| @translate_help("csm_data.commands.store.output.description") |
Comment on lines
+106
to
+114
| def delete_azure_blobs(configuration: Configuration = Configuration()) -> None: | ||
| container_client = BlobServiceClient( | ||
| account_url=f"https://{configuration.azure.account_name}.blob.core.windows.net/", | ||
| credential=ClientSecretCredential( | ||
| tenant_id=configuration.azure.tenant_id, | ||
| client_id=configuration.azure.client_id, | ||
| client_secret=configuration.azure.client_secret, | ||
| ), | ||
| ).get_container_client(configuration.azure.container_name) |
Collaborator
Author
There was a problem hiding this comment.
This change absolutely nothing, parameter value is set/create a function call
Comment on lines
+28
to
+30
| self.configuration.azure.file_prefix = ( | ||
| self.configuration.cosmotech.runner_id + "/" + self.configuration.azure.file_prefix | ||
| ) |
| @@ -37,11 +38,7 @@ def send(self, filter: Optional[list[str]] = None) -> bool: | |||
| if self._s3.output_type == "sqlite": | |||
| _file_path = _s._database_path | |||
| _file_name = "db.sqlite" | |||
Comment on lines
+15
to
+33
| @click.option( | ||
| "--filter", | ||
| "-f", | ||
| multiple=True, | ||
| help=T("csm_data.commands.store.output.parameters.filter"), | ||
| type=str, | ||
| ) | ||
| @web_help("csm-data/store/output") | ||
| @translate_help("csm_data.commands.store.output.description") | ||
| def delete( | ||
| filter, | ||
| ): | ||
| # Import the function at the start of the command | ||
| from cosmotech.coal.store.output import channel_spliter | ||
| from cosmotech.coal.utils.configuration import Configuration | ||
|
|
||
| try: | ||
| _cs = channel_spliter.ChannelSpliter(Configuration()) | ||
| _cs.delete(list(filter)) |
cosmotech/coal/azure/blob.py
Outdated
Comment on lines
+117
to
+119
| # List and delete blobs | ||
| blob_list = [b.name for b in container_client.list_blobs(name_starts_with=file_prefix)] | ||
| container_client.delete_blobs(*blob_list) |
neomatamune
requested changes
Mar 18, 2026
| @web_help("csm-data/store/delete") | ||
| @translate_help("csm_data.commands.store.delete.description") | ||
| def delete( | ||
| filter, |
Member
There was a problem hiding this comment.
need to remove the filter parameter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add runner differentiation on output S3 and Azure storage and rework on delete to match the modification