feat: accept distinct kwarg on sum and avg#1556
Open
timsaucer wants to merge 4 commits into
Open
Conversation
Upstream exposes `sum_distinct` / `avg_distinct` / `count_distinct` as sibling functions that call the same underlying UDAF with `distinct: bool = true`. The Rust binding side already routes `distinct=Some(true)` through the aggregate builder for `sum`, `avg`, and `count` — but only `count` exposed the kwarg on the Python wrapper. Add `distinct: bool = False` to `sum()` and `avg()` mirroring the existing `count()` signature, and update SKILL.md so the check-upstream audit does not re-flag the three upstream `*_distinct` shortcuts as gaps. The plan emitted by `sum(col, distinct=True)` matches what upstream's `sum_distinct(col)` builds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the standalone test_sum_distinct_kwarg and test_avg_distinct_kwarg from test_functions.py into the existing test_aggregation::test_aggregation parameterization, matching how distinct is already covered for median, array_agg, count, and bit_xor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the unhelpful "upstream avg_distinct/sum_distinct shortcut" reference in favor of describing the actual behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
timsaucer
changed the title
distinct is inserted before filter on sum and avg for consistency with the other aggregate functions, breaking positional filter callers. Add a DataFusion 54.0.0 upgrade-guide entry covering the migration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #1533
Rationale for this change
Upstream DataFusion now supports the
distinctvariant for these functions. We expose this functionality here.What changes are included in this PR?
sumandavgto take thedistinctkeyword.Are there any user-facing changes?
sumandavggain adistinctkeyword. This is a breaking change:distinctis inserted beforefilterin the argument list to stay consistent with the other aggregate functions, so any caller that passedfilterpositionally (e.g.f.sum(col, my_filter)) will now bind that value todistinct. Such callers must passfilteras a keyword argument. See the DataFusion 54.0.0 entry in the upgrade guide for details.