[BUG-FIX]: download progress not reported to Python during file transfers#791
Open
tobocop2 wants to merge 1 commit intohuggingface:mainfrom
Open
[BUG-FIX]: download progress not reported to Python during file transfers#791tobocop2 wants to merge 1 commit intohuggingface:mainfrom
tobocop2 wants to merge 1 commit intohuggingface:mainfrom
Conversation
8e0dd2d to
066a2de
Compare
…progress ItemBridgeState::compute_diff hardcoded all total_transfer_bytes fields to 0, suppressing the fine-grained network-level progress that xet-core already tracks per HTTP chunk. This caused Python callbacks to only receive coarse bytes_completed updates (per disk write, ~256MB batches). - Add transfer_bytes and transfer_bytes_completed to ItemProgressReport - Update ItemBridgeState::compute_diff to diff transfer fields (matching GroupBridgeState which already does this correctly) - Fire callbacks when transfer progress changes, even if bytes_completed has not changed yet
066a2de to
13645b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Download progress bars in
huggingface_hubupdate in large jumps (e.g. 0% -> 26% -> 63% -> 100% for a 4GB file) instead of smoothly. The root cause is thatItemBridgeState::compute_diffhardcodes alltotal_transfer_bytesfields to 0, so the fine-grained per-HTTP-chunk progress that xet-core already tracks internally never reaches Python callbacks.Fix
transfer_bytesandtransfer_bytes_completedtoItemProgressReportItemBridgeState::compute_diffto diff transfer fields, matching the existingGroupBridgeStatebehaviorGroupBridgeState::compute_diffitem inclusion to also trigger on transfer progressReproduction
Single file
Multi-file
Before: tqdm bar jumps 0% -> 26% -> 63% -> 100% with ~4 callbacks for a 4GB file.
After: tqdm bar updates smoothly every 250ms.
See
callback_bridge.rstests for coverage of the new transfer byte diffing.