perf(physical-optimizer): skip ensure_distribution rebuild when children are unchanged

[zhuqi-lucas]

Which issue does this PR close?

Closes #22520.

Rationale for this change

ensure_distribution in datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs unconditionally calls plan.with_new_children(children_plans) after collecting the (possibly redistributed) children, even when none of those children were actually replaced. For nodes like ProjectionExec, that path runs through try_new and recomputes the schema, equivalence properties, output ordering, and output partitioning, then allocates a new Arc<dyn ExecutionPlan>. When every child Arc is pointer-identical to the input, that work produces a logically identical node — pure overhead.

The cost is amplified by two factors:

1. Plan depth. Workloads dominated by point queries (no join / aggregate / unmet ordering — i.e. nothing for ensure_distribution to inject a RepartitionExec or SortExec for) hit this wasted rebuild at every node in the plan. A 5–30 deep ProjectionExec stack pays the cost N times.
2. Schema width. Most steps inside ProjectionExec::try_new are O(num_columns): per-column data_type / nullable lookup to build the new schema, per-column remapping of equivalence classes through the projection mapping, and per-column lookup when rewriting PhysicalSortExprs into the output ordering. Wide schemas (tens of columns) make every wasted rebuild proportionally heavier.

Profiling a production point-query workload (wide schemas, deep ProjectionExec stacks) showed ProjectionExec::with_new_children as the single largest cost inside ensure_distribution:

• ensure_distribution total: 2.87s of a 60s CPU sample
• ProjectionExec::with_new_children: 1.94s (56% of the rule)
• SortExec::with_new_children: 0.11s
• Other ExecutionPlan nodes: 0.82s

What changes are included in this PR?

After collecting children_plans, compare each new child Arc with the original via Arc::ptr_eq. When every child is unchanged, reuse the existing plan Arc and skip with_new_children. The UnionExec to InterleaveExec special case still runs first because it intentionally produces a new node even when child Arcs are unchanged.

This relies on the fact that ensure_distribution already produces pointer-identical Arcs for children that need no redistribution (it threads the original Arc through unchanged), so Arc::ptr_eq precisely distinguishes "rewritten" from "untouched" children at O(1) per child.

Are these changes tested?

Yes. The existing enforce_distribution suite passes unchanged (66/66):

cargo test --release -p datafusion --test core_integration -- physical_optimizer::enforce_distribution

The behavior is observable only as a CPU reduction; correctness is preserved because ExecutionPlan nodes are immutable, so reusing the original Arc produces the same plan tree as with_new_children(unchanged_children) would have, just without the schema / ordering / equivalence / partitioning recomputation.

Are there any user-facing changes?

No. Same plans, lower planning time.

Micro-benchmark

Plan shape: 30-deep ProjectionExec stack over a sorted parquet scan, 5000 iterations.

• Without fix: 852.74 ms total, 170.55 us/call
• With fix: 296.81 ms total, 59.36 us/call
• ~2.87x speedup, -65% CPU per call

Wider schemas (more projection expressions per node) widen the gap further because each skipped with_new_children avoids more O(num_columns) work.

[Copilot]

Pull request overview

This PR optimizes the physical optimizer’s distribution enforcement by avoiding unnecessary plan-node rebuilds when the computed child plans are identical to the existing children, reducing planning CPU overhead (notably for deep ProjectionExec stacks) without changing correctness.

Changes:

• Detect when all post-processing child plan Arcs are pointer-identical to the original children via Arc::ptr_eq.
• Reuse the existing plan Arc and skip with_new_children when children are unchanged.
• Preserve the existing UnionExec → InterleaveExec special-case behavior by applying it before the “children unchanged” fast-path.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.