Commit f7550ec
docs: user guide page + runnable examples for distributing expressions
Wraps up the Expr-pickle work with the user-facing material:
* docs/source/user-guide/io/distributing_work.rst — new user guide
page covering the multiprocessing, Ray, and datafusion-distributed
patterns. Includes the Security section that is the canonical home
for the cloudpickle / pickle.loads threat model.
* docs/source/user-guide/io/index.rst — toctree entry.
* examples/multiprocessing_pickle_expr.py — runnable example: a
Pool.map of a closure-capturing UDF across processes, with worker
context registration in the initializer.
* examples/ray_pickle_expr.py — Ray actor analogue.
* examples/datafusion-ffi-example/python/tests/_test_pickle_strict_ffi.py
— exercises the strict-mode refusal end to end against an FFI
capsule scalar UDF (kept under the FFI example crate because the
test needs that crate's compiled artifacts).
* examples/README.md — index entries for the new files.
Also tightens three docstrings that previously duplicated the
security warning so they point at the canonical Security section
instead:
* PythonLogicalCodec::with_python_udf_inlining (rustdoc): one-line
summary plus a relative pointer to distributing_work.rst and the
upstream Python pickle module security warning.
* SessionContext.with_python_udf_inlining: one-sentence summary plus
:doc: link to the user guide.
* datafusion.ipc module docstring: cross-reference to the user guide
for the full pattern.
The crate-level codec.rs module rustdoc also updates "pure-Python
scalar UDFs" to "scalar / aggregate / window UDFs" now that all three
are covered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 081325a commit f7550ec
7 files changed
Lines changed: 765 additions & 12 deletions
File tree
- crates/core/src
- docs/source/user-guide/io
- examples
- datafusion-ffi-example/python/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
256 | 256 | | |
257 | 257 | | |
258 | 258 | | |
259 | | - | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
260 | 265 | | |
261 | 266 | | |
262 | 267 | | |
| |||
433 | 438 | | |
434 | 439 | | |
435 | 440 | | |
436 | | - | |
| 441 | + | |
437 | 442 | | |
438 | 443 | | |
439 | 444 | | |
| |||
563 | 568 | | |
564 | 569 | | |
565 | 570 | | |
566 | | - | |
567 | | - | |
568 | | - | |
569 | | - | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
570 | 575 | | |
571 | 576 | | |
572 | 577 | | |
| |||
806 | 811 | | |
807 | 812 | | |
808 | 813 | | |
809 | | - | |
| 814 | + | |
810 | 815 | | |
811 | 816 | | |
812 | 817 | | |
| |||
0 commit comments