diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 120b6acb..944cb382 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -8,6 +8,7 @@ repos: - id: check-merge-conflict - id: check-vcs-permalinks - id: check-yaml + exclude: (^mkdocs\.yml$) - id: debug-statements exclude: (debugging\.py) - id: end-of-file-fixer diff --git a/docs/source/_static/md/commands/command-list.md b/docs/source/_static/md/commands/command-list.md index 093f3ad2..af659a97 100644 --- a/docs/source/_static/md/commands/command-list.md +++ b/docs/source/_static/md/commands/command-list.md @@ -4,5 +4,6 @@ | [`clean`](clean.md) | Clean the provided paths by removing files unknown to pytask. | | [`collect`](collect.md) | Collect tasks and report information about them. | | [`dag`](dag.md) | Create a visualization of the directed acyclic graph. | +| [`lock`](lock.md) | Inspect and update recorded task state in the lockfile. | | [`markers`](markers.md) | Show all registered markers. | | [`profile`](profile.md) | Show information about resource consumption. | diff --git a/docs/source/_static/md/commands/lock-accept-options.md b/docs/source/_static/md/commands/lock-accept-options.md new file mode 100644 index 00000000..6a8442ee --- /dev/null +++ b/docs/source/_static/md/commands/lock-accept-options.md @@ -0,0 +1,10 @@ +| Option | Default | Description | +| ------------------------------ | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | +| `-k, --expression TEXT` | - | Select tasks by expression. | +| `-m, --marker-expression TEXT` | - | Select tasks by marker expression. | +| `--with-ancestors` | `false` | Also include preceding tasks of the selected tasks. | +| `--with-descendants` | `false` | Also include descending tasks of the selected tasks. | +| `--run-on TEXT` | `task_change,dependency_change,product_change` | Choose which kinds of changes are considered when determining whether a task would normally require execution. | +| `--dry-run` | `false` | Show which recorded states would be updated without writing changes. | +| `-y, --yes` | `false` | Apply the changes without prompting for confirmation. | +| `-h, --help` | - | Show this message and exit. | diff --git a/docs/source/_static/md/commands/lock-clean-options.md b/docs/source/_static/md/commands/lock-clean-options.md new file mode 100644 index 00000000..de51a3d1 --- /dev/null +++ b/docs/source/_static/md/commands/lock-clean-options.md @@ -0,0 +1,5 @@ +| Option | Default | Description | +| ------------ | ------- | ------------------------------------------------------------------ | +| `--dry-run` | `false` | Show which stale entries would be removed without writing changes. | +| `-y, --yes` | `false` | Apply the changes without prompting for confirmation. | +| `-h, --help` | - | Show this message and exit. | diff --git a/docs/source/_static/md/commands/lock-reset-options.md b/docs/source/_static/md/commands/lock-reset-options.md new file mode 100644 index 00000000..13695157 --- /dev/null +++ b/docs/source/_static/md/commands/lock-reset-options.md @@ -0,0 +1,9 @@ +| Option | Default | Description | +| ------------------------------ | ------- | -------------------------------------------------------------------- | +| `-k, --expression TEXT` | - | Select tasks by expression. | +| `-m, --marker-expression TEXT` | - | Select tasks by marker expression. | +| `--with-ancestors` | `false` | Also include preceding tasks of the selected tasks. | +| `--with-descendants` | `false` | Also include descending tasks of the selected tasks. | +| `--dry-run` | `false` | Show which recorded states would be removed without writing changes. | +| `-y, --yes` | `false` | Apply the changes without prompting for confirmation. | +| `-h, --help` | - | Show this message and exit. | diff --git a/docs/source/_static/md/lock-accept-dry-run.md b/docs/source/_static/md/lock-accept-dry-run.md new file mode 100644 index 00000000..3d71c89c --- /dev/null +++ b/docs/source/_static/md/lock-accept-dry-run.md @@ -0,0 +1,23 @@ +
+ +```console + +$ pytask lock accept -k train --dry-run +────────────────────────── Start pytask session ───────────────────────── +Platform: win32 -- Python 3.12.0, pytask 0.5.3, pluggy 1.3.0 +Root: C:\Users\pytask-dev\git\my_project +Collected 2 tasks. + +The following recorded states would be updated: + +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Task ┃ Reason ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ +│ task_train.py::task_train_model │ task_change │ +│ task_train.py::task_evaluate_model │ dependency_change │ +└────────────────────────────────────────────────┴──────────────────────┘ + +No changes were written because --dry-run was used. +``` + +
diff --git a/docs/source/_static/md/lock-accept-interactive.md b/docs/source/_static/md/lock-accept-interactive.md new file mode 100644 index 00000000..69392eaa --- /dev/null +++ b/docs/source/_static/md/lock-accept-interactive.md @@ -0,0 +1,31 @@ +
+ +```console + +$ pytask lock accept -k train +────────────────────────── Start pytask session ───────────────────────── +Platform: win32 -- Python 3.12.0, pytask 0.5.3, pluggy 1.3.0 +Root: C:\Users\pytask-dev\git\my_project +Collected 2 tasks. + +The following recorded states would be updated: + +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Task ┃ Reason ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ +│ task_train.py::task_train_model │ task_change │ +│ task_train.py::task_evaluate_model │ dependency_change │ +└────────────────────────────────────────────────┴──────────────────────┘ + +Accept recorded state for task_train.py::task_train_model? Yes + +Accept recorded state for task_train.py::task_evaluate_model? + Yes + No +❯ Accept all remaining tasks + Quit + +Accepted recorded state for 2 task(s). +``` + +
diff --git a/docs/source/_static/md/lock-clean.md b/docs/source/_static/md/lock-clean.md new file mode 100644 index 00000000..1382a42d --- /dev/null +++ b/docs/source/_static/md/lock-clean.md @@ -0,0 +1,30 @@ +
+ +```console + +$ pytask lock clean +────────────────────────── Start pytask session ───────────────────────── +Platform: win32 -- Python 3.12.0, pytask 0.5.3, pluggy 1.3.0 +Root: C:\Users\pytask-dev\git\my_project + +The following stale lockfile entries would be removed: + +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Lockfile entry ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ +│ task_old_pipeline.py::task_train_model │ +│ task_old_pipeline.py::task_evaluate_model │ +└───────────────────────────────────────────────────────────────────────┘ + +Remove stale entry task_old_pipeline.py::task_train_model? Yes + +Remove stale entry task_old_pipeline.py::task_evaluate_model? + Yes + No +❯ Remove all remaining entries + Quit + +Removed 2 stale lockfile entries. +``` + +
diff --git a/docs/source/commands/lock.md b/docs/source/commands/lock.md new file mode 100644 index 00000000..59440892 --- /dev/null +++ b/docs/source/commands/lock.md @@ -0,0 +1,94 @@ +# lock + +Inspect and update recorded task state in the lockfile. + +## Usage + +```bash +pytask lock SUBCOMMAND [OPTIONS] [PATHS] +``` + +Subcommands: + +- `accept`: Accept the current state for selected tasks. +- `reset`: Remove recorded state for selected tasks. +- `clean`: Remove stale lockfile entries which no longer correspond to collected tasks. + +If no selectors are provided for `accept` or `reset`, the command applies to all +collected tasks in the provided paths. + +## Examples + +```bash +# Preview which selected tasks would be accepted. +pytask lock accept -k train --dry-run + +# Accept the current state for all collected tasks in the current directory. +pytask lock accept --yes + +# Also include descendants of the selected tasks. +pytask lock accept -k train --with-descendants + +# Restrict the staleness decision to task and dependency changes. +pytask lock accept -k train --run-on task_change,dependency_change + +# Reset recorded state for selected tasks. +pytask lock reset -k train --yes + +# Remove stale entries from the lockfile. +pytask lock clean --yes +``` + +## Subcommands + +### accept + +Accept the current state for selected tasks without executing them. + +```bash +pytask lock accept [OPTIONS] [PATHS] +``` + +Options: + +--8<-- "docs/source/_static/md/commands/lock-accept-options.md" + +### reset + +Remove recorded state for selected tasks. + +```bash +pytask lock reset [OPTIONS] [PATHS] +``` + +Options: + +--8<-- "docs/source/_static/md/commands/lock-reset-options.md" + +### clean + +Remove stale lockfile entries. + +```bash +pytask lock clean [OPTIONS] [PATHS] +``` + +Options: + +--8<-- "docs/source/_static/md/commands/lock-clean-options.md" + +## Interaction Modes + +- Default: Run interactively, show the planned changes, and step through them one by one + with an interactive selector for each task or lockfile entry. +- `--dry-run`: Show the planned changes without writing them. +- `--yes`: Apply the planned changes without prompting. + +The options `--dry-run` and `--yes` are mutually exclusive. + +## Related + +- [Reconciling Lockfile State](../how_to_guides/reconciling_lockfile_state.md) +- [Selecting tasks](../tutorials/selecting_tasks.md) +- [Making Tasks Persist](../tutorials/making_tasks_persist.md) +- [Lockfile](../reference_guides/lockfile.md) diff --git a/docs/source/explanations/execution_policies.md b/docs/source/explanations/execution_policies.md new file mode 100644 index 00000000..c0aa4c25 --- /dev/null +++ b/docs/source/explanations/execution_policies.md @@ -0,0 +1,108 @@ +# Task Execution Model + +The task execution model can be confusing because several questions seem to arrive at +once: is the task still current, should it run, and what happens to the recorded state? + +It becomes easier to follow once we stop looking at all of these questions at the same +time and instead watch an ordinary task move through pytask. + +## The Normal Case + +In the normal case, a task passes through four phases: setup, execution, teardown, and +processing the report. + +### Setup + +Setup is where pytask decides whether a task needs attention. + +It checks a few basic questions first. + +- Are all dependencies available and have they changed? +- Do the products already exist and have they changed? +- Has the task itself changed? + +Generally, change means whether the current state of a node matches the recorded state +in the lockfile. For most filepath-based nodes, the state is the hash of the file +content. + +If everything still matches the previously recorded state, the task is considered +unchanged and pytask will not run it. + +If something does not match, the task is stale and pytask prepares to run it. + +### Execution + +Here pytask calls the task function itself. + +### Teardown + +Once the task function has returned, pytask checks whether the promised products are +really there. + +This matters because a task can finish without raising an exception and still leave the +project in an incomplete state. Teardown is where pytask notices that kind of mismatch. + +### Processing The Report + +After teardown, pytask knows the outcome of the task. In case the task succeeded, this +is also the moment where pytask updates the recorded state for the task. + +## Exceptions to the Rule + +Not every task follows this path all the way through. + +Markers and runtime options can short-circuit parts of the model. A task can be skipped, +persisted, or forced to run even when it would otherwise be left alone. + +### Skipping Tasks + +Sometimes a task and its subgraph should not run at all. This might be because the tasks +are too costly, buggy, or irrelevant for the current run. + +For temporary exclusions, you would use the [`-m`](<>) or [`-k`](<>) options, maybe in +combination with a `not ...` expression. + +For a permanent exclusion, you would use the +[`@pytask.mark.skip`](../api/marks.md#pytask.mark.skip) or +[`@pytask.mark.skipif`](../api/marks.md#pytask.mark.skipif) markers. + +Skipping hooks into setup because pytask should stop before execution starts. All +dependent tasks are marked to be skipped as well. + +### Persisting Tasks + +Persisting is useful when a task would normally be considered stale, but rerunning it is +not desirable. A common example is a change to the task file that does not change the +actual result, such as formatting. + +Persisting also has to hook into setup because pytask needs to decide early that the +task should not execute even though something changed. Later, when the report is +processed, pytask accepts the current state as the new recorded state. + +Users can opt into this behavior with the +[`persist`](../tutorials/making_tasks_persist.md) marker. The disadvantage of the +decorator is that it is permanent and removing it triggers a rerun of the task. + +What is missing is a temporary option of the persisting mechanism that would also allow +to integrate new or renamed tasks into an existing project without triggering a rerun. + +### Forcing Tasks to Run + +Sometimes the opposite is needed: a task should run even though pytask would otherwise +consider it unchanged. + +This has to affect setup as well because setup is where pytask normally decides that an +unchanged task can be left alone. Forcing a task to run overrides that decision and lets +the task continue to execution. + +Users can request this behavior with the `--force` option. + +### Fine-grained Staleness Decision + +The current model mostly answers staleness as a yes-or-no question based on recorded +state and the current state of the task, its dependencies, and its products. + +Issue [#403](https://github.com/pytask-dev/pytask/issues/403) explores whether this +decision could become more fine-grained. For example, users might want to treat changes +to the task source differently from changes to dependencies or products, or they might +want to override the default decision for one run without changing the task itself. diff --git a/docs/source/explanations/index.md b/docs/source/explanations/index.md index ec626fd0..22b584b5 100644 --- a/docs/source/explanations/index.md +++ b/docs/source/explanations/index.md @@ -5,4 +5,5 @@ systems in general as well as its design. - [Why Pytask](why_pytask.md) - [Comparison To Other Tools](comparison_to_other_tools.md) +- [Execution Policies](execution_policies.md) - [Pluggy](pluggy.md) diff --git a/docs/source/how_to_guides/customizing_when_tasks_run.md b/docs/source/how_to_guides/customizing_when_tasks_run.md new file mode 100644 index 00000000..0ede6b83 --- /dev/null +++ b/docs/source/how_to_guides/customizing_when_tasks_run.md @@ -0,0 +1,125 @@ +# Customizing when tasks run + +By default, pytask decides whether a task should run by comparing the current state of +the task, its dependencies, and its products with the recorded state from a previous +run. + +Sometimes this default is not precise enough. You may want to treat one kind of change +as relevant and another one as irrelevant. + +For these cases, a task can define `run_on`. + +## When is this useful? + +Typical situations are: + +- A task should rerun when dependencies change, but not when only the task file changed. +- A task should ignore changes to selected products for the purpose of deciding whether + to rerun. +- A task should use a project-specific rule to decide whether the current state still + requires execution. + +## Use declarative run conditions + +For many cases, it is enough to describe which changes should trigger execution. + +The default is `None`, which means that pytask uses its default behavior. At the moment, +this means considering all three change kinds: `task_change`, `dependency_change`, and +`product_change`. + +```python +from pathlib import Path + +import pytask + + +@pytask.task( + kwargs={"depends_on": Path("input.txt"), "produces": Path("output.txt")}, + run_on=("dependency_change",), +) +def task_copy(depends_on: Path, produces: Path) -> None: + produces.write_text(depends_on.read_text()) +``` + +In this example, pytask reruns the task when dependencies change, but not when only the +task itself or its products changed. + +Supported values are: + +- `None`, which means pytask's default behavior +- `all`, which is shorthand for `task_change`, `dependency_change`, and `product_change` +- `True`, which means always execute the task +- `task_change` +- `dependency_change` +- `product_change` + +The change kinds can be passed as a single value or as an iterable. + +Time-based values can also express expiration-like rules. + +- `"7d"` +- `timedelta(days=7)` + +Other time-based interfaces could be added later. Cron expressions and RRULEs are both +candidate interfaces, but this part of the design is still open. + +This also makes combinations possible such as: + +- `run_on=("all", "7d")` + +For example, `True` can be used as a shorthand for always running a task. + +```python +from pathlib import Path + +import pytask + + +@pytask.task( + kwargs={"depends_on": Path("input.txt"), "produces": Path("output.txt")}, + run_on=True, +) +def task_copy(depends_on: Path, produces: Path) -> None: + produces.write_text(depends_on.read_text()) +``` + +## Use a callable for advanced rules + +If declarative values are not enough, `run_on` can also be a callable. + +```python +from pathlib import Path + +import pytask + + +def run_on(ctx): ... + + +@pytask.task( + kwargs={"depends_on": Path("input.txt"), "produces": Path("output.txt")}, + run_on=run_on, +) +def task_copy(depends_on: Path, produces: Path) -> None: + produces.write_text(depends_on.read_text()) +``` + +The callable receives a context object as its first positional argument. + +The exact shape of `ctx` is still to be determined. The goal is to keep enough +information available for advanced decisions without making the API harder to understand +than necessary. + +The relevant use cases for a callable are currently: + +- expiration-like rules which are more complex than a simple duration +- project-specific combinations of change kinds +- selective ignore rules for parts of the task state + +This part of the interface is still more open than the declarative values above. + +## Related + +- [Task Execution Model](../explanations/execution_policies.md) +- [Reconciling Lockfile State](reconciling_lockfile_state.md) +- [Hashing Inputs Of Tasks](hashing_inputs_of_tasks.md) diff --git a/docs/source/how_to_guides/index.md b/docs/source/how_to_guides/index.md index ce4084f1..7a92af18 100644 --- a/docs/source/how_to_guides/index.md +++ b/docs/source/how_to_guides/index.md @@ -14,6 +14,8 @@ specific tasks with pytask. - [Functional Interface](functional_interface.md) - [Capture Warnings](capture_warnings.md) - [Manage Logging](logging.md) +- [Customizing When Tasks Run](customizing_when_tasks_run.md) +- [Reconciling Lockfile State](reconciling_lockfile_state.md) - [How To Influence Build Order](how_to_influence_build_order.md) - [Hashing Inputs Of Tasks](hashing_inputs_of_tasks.md) - [Using Task Returns](using_task_returns.md) diff --git a/docs/source/how_to_guides/reconciling_lockfile_state.md b/docs/source/how_to_guides/reconciling_lockfile_state.md new file mode 100644 index 00000000..37b13c22 --- /dev/null +++ b/docs/source/how_to_guides/reconciling_lockfile_state.md @@ -0,0 +1,127 @@ +# Reconciling lockfile state + +The [`lock`](../commands/lock.md) command is useful when the current project state +should be reconciled with the lockfile without running tasks. + +This is an advanced workflow. Most of the time, [`pytask build`](../commands/build.md) +is the right command. Reach for `pytask lock` when the current files are already correct +and only the recorded state needs to catch up. + +## When is this useful? + +Typical situations are: + +- You renamed or moved a task and want to accept the current outputs for the new task. +- You reformatted a task file and do not want to rerun an expensive task. +- You produced outputs manually or elsewhere and now want to register them in the + lockfile. +- You want to remove old lockfile entries after deleting or renaming tasks. + +## Preview changes first + +By default, `pytask lock` runs interactively. It shows the planned changes and then +steps through them one by one. + +To preview changes without writing them, use `--dry-run`. + +--8<-- "docs/source/_static/md/lock-accept-dry-run.md" + +To apply changes without prompting, use `--yes`. + +```bash +pytask lock accept -k train --yes +``` + +## Accept the current state for selected tasks + +Use `pytask lock accept` when the current dependencies and products are already correct +and should become the new recorded state. + +--8<-- "docs/source/_static/md/lock-accept-interactive.md" + +If no selectors are provided, `pytask lock accept` applies to all collected tasks in the +provided paths. + +By default, the command applies only to the selected tasks. It does not automatically +include ancestors or descendants. + +If you want to accept a wider part of the DAG, widen the task selection yourself. + +```bash +pytask lock accept -k "train or evaluate" +``` + +You can also expand the selection structurally. + +```bash +pytask lock accept -k train --with-descendants +``` + +```bash +pytask lock accept -k evaluate --with-ancestors +``` + +If a selected task is missing a required dependency or product, the command fails for +that task. + +## Control which changes matter + +`pytask lock accept` can use `--run-on` to decide which kinds of changes are considered +when determining whether a task would normally require execution. + +Supported values are: + +- `task_change` +- `dependency_change` +- `product_change` + +For example, if only changes to the task definition and dependencies should matter for +the current command, you can write: + +```bash +pytask lock accept -k train --run-on task_change,dependency_change +``` + +This is useful when you want to treat one kind of change as relevant for the current +operation while ignoring another one. + +## Reset recorded state for selected tasks + +Use `pytask lock reset` when recorded state for selected tasks should be removed. + +```bash +pytask lock reset -k train +``` + +On the next build, pytask will determine again whether these tasks require execution. + +This is useful when state was accepted too broadly or when you want selected tasks to be +reconsidered from scratch. + +## Remove stale lockfile entries + +Use `pytask lock clean` to remove entries from the lockfile which no longer correspond +to collected tasks in the current project. + +```bash +pytask lock clean +``` + +--8<-- "docs/source/_static/md/lock-clean.md" + +This is useful after deleting, renaming, or moving tasks when old entries should no +longer remain in the lockfile. + +## Be explicit about scope + +`pytask lock` is a sharp tool. It changes recorded state without executing tasks. + +Use narrow task selections first, preview changes with `--dry-run`, and widen the +selection only when needed. + +## Related + +- [`pytask lock`](../commands/lock.md) +- [Making Tasks Persist](../tutorials/making_tasks_persist.md) +- [Task Execution Model](../explanations/execution_policies.md) +- [Lockfile](../reference_guides/lockfile.md) diff --git a/docs/source/reference_guides/lockfile.md b/docs/source/reference_guides/lockfile.md index 15b29c39..42d15ada 100644 --- a/docs/source/reference_guides/lockfile.md +++ b/docs/source/reference_guides/lockfile.md @@ -55,6 +55,13 @@ Use [`pytask build --clean-lockfile`](../commands/build.md#options) to rewrite `pytask.lock` with only currently collected tasks. The rewrite happens after a successful build and recomputes current state values without executing tasks again. +!!! warning + + This area is currently under discussion. If + [`pytask lock clean`](../commands/lock.md#clean) is introduced, it is still an open + question whether `--clean-lockfile` should remain as a build flag or be deprecated in + favor of the dedicated `lock` subcommand. + ## File Format Reference ### Top-Level diff --git a/mkdocs.yml b/mkdocs.yml index 257741e6..b8e18c63 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -42,6 +42,8 @@ nav: - Functional Interface: how_to_guides/functional_interface.md - Capture Warnings: how_to_guides/capture_warnings.md - Manage Logging: how_to_guides/logging.md + - Customizing When Tasks Run: how_to_guides/customizing_when_tasks_run.md + - Reconciling Lockfile State: how_to_guides/reconciling_lockfile_state.md - How To Influence Build Order: how_to_guides/how_to_influence_build_order.md - Hashing Inputs Of Tasks: how_to_guides/hashing_inputs_of_tasks.md - Using Task Returns: how_to_guides/using_task_returns.md @@ -59,6 +61,7 @@ nav: - Overview: explanations/index.md - Why pytask: explanations/why_pytask.md - Comparison To Other Tools: explanations/comparison_to_other_tools.md + - Execution Policies: explanations/execution_policies.md - Pluggy: explanations/pluggy.md - Reference Guides: - Overview: reference_guides/index.md @@ -83,6 +86,7 @@ nav: - clean: commands/clean.md - collect: commands/collect.md - dag: commands/dag.md + - lock: commands/lock.md - markers: commands/markers.md - profile: commands/profile.md - Plugin List: plugin_list.md @@ -159,7 +163,11 @@ markdown_extensions: combine_header_slug: true - pymdownx.snippets: check_paths: true - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - toc: permalink: true