Skip to content

Add snappy_cpp as optional dependency #9530

@Dandandan

Description

@Dandandan

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

It seems much faster than snap in my testing on compressing, even on e2e benchmarks (clickbench) it is up to 2x as fast:

arrow_reader_clickbench/async_object_store/Q1
                        time:   [906.71 µs 910.64 µs 914.99 µs]
                        change: [−3.1264% −1.9584% −0.9019%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  7 (7.00%) high mild
  3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q10
                        time:   [5.4424 ms 5.4692 ms 5.4999 ms]
                        change: [−2.5400% −1.0993% +0.0171%] (p = 0.09 > 0.05)
                        No change in performance detected.
Found 16 outliers among 100 measurements (16.00%)
  3 (3.00%) high mild
  13 (13.00%) high severe
arrow_reader_clickbench/async_object_store/Q11
                        time:   [6.1418 ms 6.1514 ms 6.1619 ms]
                        change: [−0.4752% +0.1639% +0.7037%] (p = 0.60 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  7 (7.00%) high mild
  3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q12
                        time:   [11.065 ms 11.089 ms 11.125 ms]
                        change: [−17.577% −17.329% −17.043%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
arrow_reader_clickbench/async_object_store/Q13
                        time:   [13.540 ms 13.578 ms 13.620 ms]
                        change: [−13.550% −13.282% −12.955%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
arrow_reader_clickbench/async_object_store/Q14
                        time:   [12.368 ms 12.481 ms 12.643 ms]
                        change: [−16.844% −16.059% −14.979%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q19
                        time:   [2.3651 ms 2.3716 ms 2.3784 ms]
                        change: [−0.2989% +0.4660% +1.1652%] (p = 0.23 > 0.05)
                        No change in performance detected.
arrow_reader_clickbench/async_object_store/Q20
                        time:   [46.271 ms 46.409 ms 46.592 ms]
                        change: [−37.543% −37.315% −37.054%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
Benchmarking arrow_reader_clickbench/async_object_store/Q21: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, or reduce sample count to 90.
arrow_reader_clickbench/async_object_store/Q21
                        time:   [53.187 ms 53.314 ms 53.444 ms]
                        change: [−35.829% −35.618% −35.418%] (p = 0.00 < 0.05)
                        Performance has improved.
Benchmarking arrow_reader_clickbench/async_object_store/Q22: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, or reduce sample count to 70.
arrow_reader_clickbench/async_object_store/Q22
                        time:   [67.068 ms 67.386 ms 67.793 ms]
                        change: [−32.873% −32.418% −31.971%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
Benchmarking arrow_reader_clickbench/async_object_store/Q23: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 15.4s, or reduce sample count to 30.
arrow_reader_clickbench/async_object_store/Q23
                        time:   [147.95 ms 148.29 ms 148.67 ms]
                        change: [−33.220% −33.030% −32.818%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe
arrow_reader_clickbench/async_object_store/Q24
                        time:   [15.013 ms 15.030 ms 15.048 ms]
                        change: [−16.445% −16.294% −16.150%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
arrow_reader_clickbench/async_object_store/Q27
                        time:   [32.775 ms 32.842 ms 32.922 ms]
                        change: [−46.691% −46.261% −45.930%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q28
                        time:   [35.060 ms 35.184 ms 35.342 ms]
                        change: [−42.107% −41.869% −41.569%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe
arrow_reader_clickbench/async_object_store/Q30
                        time:   [14.191 ms 14.216 ms 14.245 ms]
                        change: [−15.213% −15.022% −14.818%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)

Describe the solution you'd like
Add it

Describe alternatives you've considered

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelogperformance

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions