fix: filter concept uploads to training-relevant files only#1406
Open
BitcrushedHeart wants to merge 1 commit intoNerogar:masterfrom
Open
fix: filter concept uploads to training-relevant files only#1406BitcrushedHeart wants to merge 1 commit intoNerogar:masterfrom
BitcrushedHeart wants to merge 1 commit intoNerogar:masterfrom
Conversation
Cloud concept uploads previously transferred every file and directory indiscriminately, including hidden directories like .thumbnails and .trash, plus non-training files such as archives. This caused uploads to appear stuck due to massive per-file SCP/SFTP overhead on hundreds of thousands of irrelevant files. Concept uploads now skip hidden directories and only transfer files with supported image/video extensions and .txt caption files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.thumbnails,.trash,.index, etc.) and non-training files (archives, databases, etc.).txtcaption filesFiles changed
modules/cloud/BaseSSHFileSync.py-sync_up_dir()gains optionalskip_hiddenandallowed_extensionsparametersmodules/cloud/BaseFileSync.py- updated abstract signature to matchmodules/cloud/BaseCloud.py- concept uploads passskip_hidden=Trueand the set of training-relevant extensions