Decompress / decoding in parquet reader improvements#9577
Open
Dandandan wants to merge 9 commits intoapache:mainfrom
Open
Decompress / decoding in parquet reader improvements#9577Dandandan wants to merge 9 commits intoapache:mainfrom
Dandandan wants to merge 9 commits intoapache:mainfrom
Conversation
Avoid unnecessary buffer zero-fill in Snappy decompression by writing directly into spare capacity, and reduce per-byte overhead in VLQ integer decoding by reading directly from the buffer slice instead of calling get_aligned for each byte. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dandandan
changed the title
Contributor
Author
|
run benchmark arrow_reader_clickbench |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
Benchmark for this request failed. Last 20 lines of output: Click to expand |
Contributor
Author
|
run benchmark arrow_reader_clickbench |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
Dandandan
changed the title
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Contributor
Author
|
I guess 1-3% sounds about right |
Contributor
Author
|
run benchmark arrow_reader_clickbench |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Contributor
Author
|
Nice seems some low single digit improvement across the board (I think mostly from the non-zeroing) |
When bit_width guarantees all possible indices fit within the dictionary, use unchecked indexing to allow LLVM to unroll the dict gather loop 4x with paired loads/stores instead of scalar with per-element bounds checks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmark arrow_reader arrow_reader_clickbench |
Dandandan
changed the title
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
When bit_width guarantees all possible indices fit within the dictionary, use unchecked access to eliminate per-element bounds checks. Also skip buffer management when all dictionary views are inlined (<=12 bytes). Generates a clean 8-instruction gather loop for the common case (all_indices_valid + base_buffer_idx=0) and a branchless 14-instruction loop for the non-zero buffer offset case. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmark arrow_reader arrow_reader_clickbench |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Reserve the full output capacity upfront before the decode loop, eliminating per-chunk reallocation checks inside extend. This gives a ~25% speedup for dictionary-encoded StringView reads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add RleDecoder::get_batch_direct which exposes RLE vs bit-packed batches via a callback, allowing callers to handle each case optimally. For RLE runs, the dict view is looked up once and repeated directly with repeat_n, skipping the index buffer entirely. For bit-packed runs, indices are decoded to a stack-local buffer and gathered immediately. This eliminates the intermediate index buffer roundtrip for the common RLE case and reduces StringView dictionary decoding time by ~49% (137µs → 70µs in benchmarks). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmark arrow_reader arrow_reader_clickbench |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Dandandan
force-pushed
the
optimize-snappy-vlq-decoding
branch
4 times, most recently
from
a0815cd to
4b9a13b
Compare
Replace the if/else checked/unchecked branching in get_batch_with_dict with a single branchless .min(max_idx) clamp. This: - Prevents UB on corrupt parquet files (indices clamped to valid range) - Removes the if/else branch, simplifying codegen - Improves i32 dict perf by ~13% (60µs → 52µs) due to simpler code - StringView dict remains at 75µs (45% faster than 137µs baseline) Remove unused bit_width field from DictIndexDecoder. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dandandan
force-pushed
the
optimize-snappy-vlq-decoding
branch
from
4b9a13b to
0fcda30
Compare
Reserve offsets capacity upfront before the decode loop to avoid per-chunk reallocation. ~3.5% improvement for StringArray dict reads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These are only used by the arrow dictionary_index decoder. Without the arrow feature, they appear as dead code to clippy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmark arrow_reader_clickbench |
Dandandan
changed the title
Dandandan
changed the title
|
🤖 Arrow criterion benchmark running (GKE) | trigger |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Details
Resource Usagebase (merge-base)
branch
|
Contributor
Author
|
NIce - up to 10% with the combined changes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimize dictionary-encoded column reading in the parquet reader, with focus on both primitive (Int32) and StringView types.
Changes
RLE decoder: branchless index clamping (
rle.rs).min(max_idx)clampget_batch_directmethod that exposes RLE vs bit-packed batches via callbackStringView dictionary decoding (
byte_view_array.rs,dictionary_index.rs)repeat_nto fill views directly, skipping the index buffer entirelyByteArray dictionary decoding (
byte_array.rs)Snappy / VLQ
VLQ
Benchmarks
Test plan
arrowfeature🤖 Generated with Claude Code