Fix default sanitizer to only redact #-prefixed secrets; add input_files_dir to scaffolder#4
Open
matyas-jirat-keboola wants to merge 4 commits intomainfrom
Open
Fix default sanitizer to only redact #-prefixed secrets; add input_files_dir to scaffolder#4matyas-jirat-keboola wants to merge 4 commits intomainfrom
matyas-jirat-keboola wants to merge 4 commits intomainfrom
Conversation
…rameter - Add input_files_dir param to scaffold_from_json() so writer component input CSVs can be supplied without going through the CLI layer - Add TestScaffolder._copy_input_files() static method (previously lived only in datadirtest/__main__.py, invisible to API users) - Expand module docstring with tests/setup/ layout convention and a concrete writer example so other sessions can discover the feature Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously _copy_input_files() was called after all tests were recorded, meaning writer components couldn't find their input CSVs during the live API run. Move the copy into _scaffold_single_test(), after the directory structure is written but before _record_test() is called. The files now live in in/tables/ or in/files/ for both recording and replay. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces extract_values() with _collect_hash_values() so that non-sensitive metadata fields (oauthVersion, id, created, etc.) are skipped. Previously, short values like "2.0" were added to sensitive_values, causing URL path corruption in cassettes (e.g. api.xro/2.0 → api.xro/REDACTED). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous tests used plain keys (e.g. {"api_key": "my-key"}) which
did not reflect the actual config.secrets.json structure. In Keboola,
only #-prefixed keys are encrypted secrets — non-prefixed fields like
oauthVersion or id are plain metadata that must not be redacted from
cassettes (doing so was the bug fixed in 58bd94c).
Update both failing tests to use #-prefixed keys, and add an explicit
assertion that non-prefixed metadata values are not collected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Open
4 tasks
matyas-jirat-keboola
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Fix:
create_default_sanitizer()only redacts#-prefixed secret valuescreate_default_sanitizer()was callingextract_values(), which extracts all string values from the secrets dict regardless of key nameoauthVersion: "2.0"ended up insensitive_values, causing literal string replacement in cassette URLs (e.g.api.xro/2.0→api.xro/REDACTED)_collect_hash_values(), which already exists and correctly respects the Keboola#-prefix convention — only values under#-prefixed keys are treated as sensitiveDefaultSanitizer(config=...)already used_collect_hash_values()correctly; this alignscreate_default_sanitizer()with the same behaviour#-prefixed keys matching the real secrets file formatFeature:
input_files_dirparameter for writer component scaffoldinginput_files_dir: Path | Nonetoscaffold_from_json()and_scaffold_single_test()TestScaffolder._copy_input_files()static method that reads each test'sconfig.jsonand copies files frominput_files_dirinto the test'sin/tables/orin/files/based onstorage.input.tables[].destination/storage.input.files[].destinationentriesinput_files_dirdoes not exist or a referenced source file is missingtests/setup/input_files/repo layout convention in the module docstringTest plan
oauthVersioninsecrets.json(e.g. Xero) and verify URLs are not corruptedCannotOverwriteExistingCassetteException#data,#appSecret) are still redacted in cassettesinput_files_dirpointing to a directory with CSVs and verify files are copied intoin/tables/before recording🤖 Generated with Claude Code