Merge remote-tracking branch 'origin/main' into python3.14

Author: tobiasraabe

.github/workflows/main.yml

@@ -12,6 +12,7 @@ on:
   pull_request:
     branches:
     - '*'
+  merge_group:
 
 jobs:
 
@@ -27,8 +28,11 @@ jobs:
           enable-cache: true
       - name: Install just
         uses: extractions/setup-just@v3
+      - name: Install graphviz
+        run: |
+          sudo apt-get update
+          sudo apt-get install graphviz graphviz-dev
       - run: just typing
-      - run: just typing-nb
 
   run-tests:
 

.pre-commit-config.yaml

@@ -25,12 +25,12 @@ repos:
     -   id: python-no-log-warn
     -   id: text-unicode-replacement-char
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.14.9
+    rev: v0.14.10
     hooks:
       - id: ruff-format
       - id: ruff-check
 -   repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.9.17
+    rev: 0.9.18
     hooks:
       - id: uv-lock
 -   repo: https://github.com/executablebooks/mdformat
@@ -59,7 +59,7 @@ repos:
     -   id: nbstripout
         exclude: (docs)
 -   repo: https://github.com/crate-ci/typos
-    rev: v1
+    rev: typos-dict-v0.13.13
     hooks:
     -   id: typos
         exclude: (\.ipynb)

CHANGELOG.md

@@ -7,10 +7,16 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 
 ## Unreleased
 
+- {pull}`739` closes file descriptors for the capture manager between CLI runs and
+  disposes stale database engines to prevent hitting OS file descriptor limits in
+  large test runs.
 - {pull}`725` fixes the pickle node hash test by accounting for Python 3.14's
   default pickle protocol.
-- {pull}`???` adapts the interactive debugger integration to Python 3.14's
+- {pull}`726` adapts the interactive debugger integration to Python 3.14's
   updated `pdb` behaviour and keeps pytest-style capturing intact.
+- {pull}`734` migrates from mypy to ty for type checking.
+- {pull}`736` updates the comparison to other tools documentation and adds a section on
+  the Common Workflow Language (CWL) and WorkflowHub.
 
 ## 0.5.7 - 2025-11-22
 

docs/source/explanations/comparison_to_other_tools.md

@@ -10,124 +10,111 @@ in other WMFs.
 
 ## [snakemake](https://github.com/snakemake/snakemake)
 
-Pros
-
-- Very mature library and probably the most adapted library in the realm of scientific
-  workflow software.
-- Can scale to clusters and use Docker images.
-- Supports Python and R.
-- Automatic test case generation.
-
-Cons
-
-- Need to learn snakemake's syntax which is a mixture of Make and Python.
-- No debug mode.
-- Seems to have no plugin system.
+Snakemake is one of the most widely adopted workflow systems in scientific computing. It
+scales from local execution to clusters and cloud environments, with built-in support
+for containers and conda environments. Workflows are defined using a DSL that combines
+Make-style rules with Python, and can be exported to CWL for portability.
 
 ## [ploomber](https://github.com/ploomber/ploomber)
 
-General
-
-- Strong focus on machine learning pipelines, training, and deployment.
-- Integration with tools such as MLflow, Docker, AWS Batch.
-- Tasks can be defined in yaml, python files, Jupyter notebooks or SQL.
-
-Pros
-
-- Conversion from Jupyter notebooks to tasks via
-  [soorgeon](https://github.com/ploomber/soorgeon).
-
-Cons
-
-- Programming in Jupyter notebooks increases the risk of coding errors (e.g.
-  side-effects).
-- Supports parametrizations in form of cartesian products in `yaml` files, but not more
-  powerful parametrizations.
+Ploomber focuses on machine learning pipelines with strong integration into MLflow,
+Docker, and AWS Batch. Tasks can be defined in YAML, Python files, Jupyter notebooks, or
+SQL, and it can convert notebooks into pipeline tasks.
 
 ## [Waf](https://waf.io)
 
-Pros
-
-- Mature library.
-- Can be extended.
-
-Cons
-
-- Focus on compiling binaries, not research projects.
-- Bus factor of 1.
+Waf is a mature build system primarily designed for compiling software projects. It
+handles complex build dependencies and can be extended with Python.
 
 ## [nextflow](https://github.com/nextflow-io/nextflow)
 
-- Tasks are scripted using Groovy which is a superset of Java.
-- Supports AWS, Google, Azure.
-- Supports Docker, Shifter, Podman, etc.
+Nextflow is a workflow system popular in bioinformatics that runs on AWS, Google Cloud,
+and Azure. It uses Groovy (a JVM language) for scripting and has strong support for
+containers including Docker, Singularity, and Podman.
 
 ## [Kedro](https://github.com/kedro-org/kedro)
 
-Pros
-
-- Mature library, used by some institutions and companies. Created inside McKinsey.
-- Provides the full package: templates, pipelines, deployment
+Kedro is a mature workflow framework developed at McKinsey that provides project
+templates, data catalogs, and deployment tooling. It is designed for production machine
+learning pipelines with a focus on software engineering best practices.
 
 ## [pydoit](https://github.com/pydoit/doit)
 
-General
-
-- A general task runner which focuses on command line tools.
-- You can think of it as an replacement for make.
-- Powers Nikola, a static site generator.
+pydoit is a general-purpose task runner that serves as a Python replacement for Make. It
+focuses on executing command-line tools and powers projects like Nikola, a static site
+generator.
 
 ## [Luigi](https://github.com/spotify/luigi)
 
-General
-
-- A build system written by Spotify.
-- Designed for any kind of long-running batch processes.
-- Integrates with many other tools like databases, Hadoop, Spark, etc..
-
-Cons
-
-- Very complex interface and a lot of stuff you probably don't need.
-- [Development](https://github.com/spotify/luigi/graphs/contributors) seems to stall.
+Luigi is a workflow system built by Spotify for long-running batch processes. It
+integrates with Hadoop, Spark, and various databases for large-scale data pipelines.
+Development has slowed in recent years.
 
 ## [sciluigi](https://github.com/pharmbio/sciluigi)
 
-sciluigi aims to be a lightweight wrapper around luigi.
-
-Cons
-
-- [Development](https://github.com/pharmbio/sciluigi/graphs/contributors) has basically
-  stalled since 2018.
-- Not very popular compared to its lifetime.
+sciluigi is a lightweight wrapper around Luigi aimed at simplifying scientific workflow
+development. It reduces some of Luigi's boilerplate for research use cases. Development
+has stalled since 2018.
 
 ## [scipipe](https://github.com/scipipe/scipipe)
 
-Cons
+SciPipe is a workflow library written in Go for building robust, flexible pipelines
+using Flow-Based Programming principles. It compiles workflows to fast binaries and is
+designed for bioinformatics and cheminformatics applications involving command-line
+tools.
 
-- [Development](https://github.com/scipipe/scipipe/graphs/contributors) slowed down.
-- Written in Go.
+## [SCons](https://github.com/SCons/scons)
 
-## [Scons](https://github.com/SCons/scons)
-
-Pros
-
-- Mature library.
-
-Cons
-
-- Seems to have no plugin system.
+SCons is a mature, cross-platform software construction tool that serves as an improved
+substitute for Make. It uses Python scripts for configuration and has built-in support
+for C, C++, Java, Fortran, and automatic dependency analysis.
 
 ## [pypyr](https://github.com/pypyr/pypyr)
 
-General
+pypyr is a task-runner for automation pipelines defined in YAML. It provides built-in
+steps for common operations like loops, conditionals, retries, and error handling
+without requiring custom code, and is often used for CI/CD and DevOps automation.
+
+## [ZenML](https://github.com/zenml-io/zenml)
 
-- A general task-runner with task defined in yaml files.
+ZenML is an MLOps framework for building portable ML pipelines that can run on various
+orchestrators including Kubernetes, AWS SageMaker, GCP Vertex AI, Kubeflow, and Airflow.
+It focuses on productionizing ML workflows with features like automatic
+containerization, artifact tracking, and native caching.
 
-## [zenml](https://github.com/zenml-io/zenml)
+## [Flyte](https://github.com/flyteorg/flyte)
 
-## [flyte](https://github.com/flyteorg/flyte)
+Flyte is a Kubernetes-native workflow orchestration platform for building
+production-grade data and ML pipelines. It provides automatic retries, checkpointing,
+failure recovery, and scales dynamically across cloud providers including AWS, GCP, and
+Azure.
 
 ## [pipefunc](https://github.com/pipefunc/pipefunc)
 
-A tool for executing graphs made out of functions. More focused on computational
-compared to workflow graphs.
+pipefunc is a lightweight library for creating function pipelines as directed acyclic
+graphs (DAGs) in pure Python. It automatically handles execution order, supports
+map-reduce operations, parallel execution, and provides resource profiling.
+
+## [Common Workflow Language (CWL)](https://www.commonwl.org/)
+
+CWL is an open standard for describing data analysis workflows in a portable,
+language-agnostic format. Its primary goal is to enable workflows to be written once and
+executed across different computing environments—from local workstations to clusters,
+cloud, and HPC systems—without modification. Workflows described in CWL can be
+registered on [WorkflowHub](https://workflowhub.eu/) for sharing and discovery following
+FAIR (Findable, Accessible, Interoperable, Reusable) principles.
+
+CWL is particularly prevalent in bioinformatics and life sciences where reproducibility
+across institutions is critical. Tools that support CWL include
+[cwltool](https://github.com/common-workflow-language/cwltool) (the reference
+implementation), [Toil](https://github.com/DataBiosphere/toil),
+[Arvados](https://arvados.org/), and [REANA](https://reanahub.io/). Some workflow
+systems like Snakemake and Nextflow can export workflows to CWL format.
+
+pytask is not a CWL-compliant tool because it operates on a fundamentally different
+model. CWL describes workflows as graphs of command-line tool invocations where data
+flows between tools via files. pytask, in contrast, orchestrates Python functions that
+can execute arbitrary code, manipulate data in memory, call APIs, or perform any
+operation available in Python. This Python-native approach enables features like
+interactive debugging but means pytask workflows cannot be represented in CWL's
+command-line-centric specification.

justfile

@@ -10,17 +10,9 @@ test *FLAGS:
 test-cov *FLAGS:
     uv run --group test pytest --nbmake --cov=src --cov=tests --cov-report=xml -n auto {{FLAGS}}
 
-# Run tests with notebook validation
-test-nb:
-    uv run --group test pytest --nbmake -n auto
-
 # Run type checking
 typing:
-    uv run --group typing --no-dev --isolated mypy
-
-# Run type checking on notebooks
-typing-nb:
-    uv run --group typing --no-dev --isolated nbqa mypy --ignore-missing-imports .
+    uv run --group typing --group test ty check src/ tests/
 
 # Run linting
 lint:

pyproject.toml

@@ -47,7 +47,6 @@ name = "Tobias Raabe"
 email = "raabe@posteo.de"
 
 [dependency-groups]
-dev = ["pygraphviz>=1.12;platform_system=='Linux'"]
 docs = [
     "furo>=2024.8.6",
     "ipython>=8.13.2",
@@ -65,6 +64,7 @@ docs = [
 ]
 plugin-list = ["httpx>=0.27.0", "tabulate[widechars]>=0.9.0", "tqdm>=4.66.3"]
 test = [
+    "cloudpickle>=3.0.0",
     "deepdiff>=7.0.0",
     # nbmake requires pywin32 on Windows, which has no wheels for Python 3.14 yet
     "nbmake>=1.5.5; platform_system != 'Windows' or python_version < '3.14'",
@@ -74,11 +74,11 @@ test = [
     "pytest-cov>=5.0.0",
     "pytest-xdist>=3.6.1",
     "syrupy>=4.5.0",
-    "aiohttp>=3.11.0",      # For HTTPPath tests.
+    "aiohttp>=3.11.0",     # For HTTPPath tests.
     "coiled>=1.42.0",
-    "cloudpickle>=3.0.0",
+    "pygraphviz>=1.12;platform_system=='Linux'",
 ]
-typing = ["mypy>=1.11.0", "nbqa>=1.8.5"]
+typing = ["ty>=0.0.7"]
 
 [project.urls]
 Changelog = "https://pytask-dev.readthedocs.io/en/stable/changes.html"
@@ -170,33 +170,14 @@ filterwarnings = [
     "ignore:'asyncio\\..*' is deprecated:DeprecationWarning",
 ]
 
-[tool.mypy]
-files = ["src", "tests"]
-check_untyped_defs = true
-disallow_any_generics = true
-disallow_incomplete_defs = true
-disallow_untyped_defs = true
-no_implicit_optional = true
-warn_redundant_casts = true
-warn_unused_ignores = true
-disable_error_code = ["import-untyped"]
-
-[[tool.mypy.overrides]]
-module = "tests.*"
-disallow_untyped_defs = false
-ignore_errors = true
-
-[[tool.mypy.overrides]]
-module = ["click_default_group", "networkx"]
-ignore_missing_imports = true
-
-[[tool.mypy.overrides]]
-module = ["_pytask.coiled_utils"]
-disable_error_code = ["import-not-found"]
-
-[[tool.mypy.overrides]]
-module = ["_pytask.hookspecs"]
-disable_error_code = ["empty-body"]
+[tool.ty.rules]
+unused-ignore-comment = "error"
+
+[tool.ty.src]
+exclude = ["src/_pytask/_hashlib.py"]
+
+[tool.ty.terminal]
+error-on-warning = true
 
 [tool.coverage.report]
 exclude_also = [

src/_pytask/build.py

@@ -9,6 +9,7 @@
 from typing import TYPE_CHECKING
 from typing import Any
 from typing import Literal
+from typing import cast
 
 import click
 
@@ -65,7 +66,7 @@ def pytask_unconfigure(session: Session) -> None:
     path.write_text(json.dumps(HashPathCache._cache))
 
 
-def build(  # noqa: C901, PLR0912, PLR0913
+def build(  # noqa: C901, PLR0912, PLR0913, PLR0915
     *,
     capture: Literal["fd", "no", "sys", "tee-sys"] | CaptureMethod = CaptureMethod.FD,
     check_casing_of_paths: bool = True,
@@ -230,10 +231,22 @@ def build(  # noqa: C901, PLR0912, PLR0913
 
             raw_config = {**DEFAULTS_FROM_CLI, **raw_config}
 
-            raw_config["paths"] = parse_paths(raw_config["paths"])
+            paths_value = raw_config["paths"]
+            # Convert tuple to list since parse_paths expects Path | list[Path]
+            if isinstance(paths_value, tuple):
+                paths_value = list(paths_value)
+            if not isinstance(paths_value, (Path, list)):
+                msg = f"paths must be Path or list, got {type(paths_value)}"
+                raise TypeError(msg)  # noqa: TRY301
+            # Cast is justified - we validated at runtime
+            raw_config["paths"] = parse_paths(cast("Path | list[Path]", paths_value))
 
             if raw_config["config"] is not None:
-                raw_config["config"] = Path(raw_config["config"]).resolve()
+                config_value = raw_config["config"]
+                if not isinstance(config_value, (str, Path)):
+                    msg = f"config must be str or Path, got {type(config_value)}"
+                    raise TypeError(msg)  # noqa: TRY301
+                raw_config["config"] = Path(config_value).resolve()
                 raw_config["root"] = raw_config["config"].parent
             else:
                 (