Collect stage-level metrics#461
Open
gabotechs wants to merge 1 commit into
Open
Conversation
This was referenced May 23, 2026
gabotechs
changed the base branch from
main
to
gabrielmusat/local-worker-connections
gabotechs
force-pushed
the
gabrielmusat/stage-metrics
branch
from
cba86e8 to
aedad16
Compare
gabotechs
changed the base branch from
gabrielmusat/local-worker-connections
to
gabrielmusat/factor-out-distributed-recursion
gabotechs
changed the base branch from
gabrielmusat/factor-out-distributed-recursion
to
gabrielmusat/factor-out-distributed-recursion-fixed
gabotechs
force-pushed
the
gabrielmusat/factor-out-distributed-recursion-fixed
branch
from
e28d614 to
12a761b
Compare
gabotechs
force-pushed
the
gabrielmusat/stage-metrics
branch
from
aedad16 to
af69055
Compare
gabotechs
added a commit
that referenced
this pull request
May 26, 2026
… be the same (#427) This is one PR from the following stack of PRs: - #427 <- you are here - #469 - #461 - #462 - #463 - #464 - #432 This is a preparatory step towards: - #377 This is an optimization that allows workers to communicate in-memory avoiding network calls and serialization in case it needs to communicate to itself. This optimization shows good improvements if using a small number of workers, and fades away as more workers are used. Even if this shows improvements today, it will become very meaningful in adaptative query execution: there will be times when two consecutive stages are assigned 1 task each, and as that was done dynamically, we cannot eagerly do the optimization that collapses those two stages into one, so instead, we assigned both stages to the same worker, and the optimization in this PR kicks in, executing the plan fully locally.
gabotechs
force-pushed
the
gabrielmusat/factor-out-distributed-recursion-fixed
branch
from
12a761b to
c5e321e
Compare
gabotechs
force-pushed
the
gabrielmusat/stage-metrics
branch
from
af69055 to
72c7a9e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is one PR from the following stack of PRs:
This PR introduces stage-level metrics collection to provide better observability into the execution of distributed query plans. By collecting metrics at the stage level, we can understand performance characteristics of individual stages and optimize task distribution and scheduling decisions based on actual runtime data.