Skip to content

Commit e0c1c3c

Browse files
tobiasraabetobiasraabe
committed
Add lock and execution policy docs
1 parent 6278277 commit e0c1c3c

16 files changed

Lines changed: 586 additions & 1 deletion

‎docs/source/_static/md/commands/command-list.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,6 @@
44
| [`clean`](clean.md) | Clean the provided paths by removing files unknown to pytask. |
55
| [`collect`](collect.md) | Collect tasks and report information about them. |
66
| [`dag`](dag.md) | Create a visualization of the directed acyclic graph. |
7+
| [`lock`](lock.md) | Inspect and update recorded task state in the lockfile. |
78
| [`markers`](markers.md) | Show all registered markers. |
89
| [`profile`](profile.md) | Show information about resource consumption. |

‎docs/source/_static/md/commands/lock-accept-options.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
| Option | Default | Description |
2+
| ------------------------------ | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
3+
| `-k, --expression TEXT` | - | Select tasks by expression. |
4+
| `-m, --marker-expression TEXT` | - | Select tasks by marker expression. |
5+
| `--with-ancestors` | `false` | Also include preceding tasks of the selected tasks. |
6+
| `--with-descendants` | `false` | Also include descending tasks of the selected tasks. |
7+
| `--run-on TEXT` | `task_change,dependency_change,product_change` | Choose which kinds of changes are considered when determining whether a task would normally require execution. |
8+
| `--dry-run` | `false` | Show which recorded states would be updated without writing changes. |
9+
| `-y, --yes` | `false` | Apply the changes without prompting for confirmation. |
10+
| `-h, --help` | - | Show this message and exit. |

‎docs/source/_static/md/commands/lock-clean-options.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
| Option | Default | Description |
2+
| ------------ | ------- | ------------------------------------------------------------------ |
3+
| `--dry-run` | `false` | Show which stale entries would be removed without writing changes. |
4+
| `-y, --yes` | `false` | Apply the changes without prompting for confirmation. |
5+
| `-h, --help` | - | Show this message and exit. |

‎docs/source/_static/md/commands/lock-reset-options.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
| Option | Default | Description |
2+
| ------------------------------ | ------- | -------------------------------------------------------------------- |
3+
| `-k, --expression TEXT` | - | Select tasks by expression. |
4+
| `-m, --marker-expression TEXT` | - | Select tasks by marker expression. |
5+
| `--with-ancestors` | `false` | Also include preceding tasks of the selected tasks. |
6+
| `--with-descendants` | `false` | Also include descending tasks of the selected tasks. |
7+
| `--dry-run` | `false` | Show which recorded states would be removed without writing changes. |
8+
| `-y, --yes` | `false` | Apply the changes without prompting for confirmation. |
9+
| `-h, --help` | - | Show this message and exit. |

‎docs/source/_static/md/lock-accept-dry-run.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
<div class="termy">
2+
3+
```console
4+
5+
$ pytask lock accept -k train --dry-run
6+
────────────────────────── Start pytask session ─────────────────────────
7+
Platform: win32 -- Python 3.12.0, pytask 0.5.3, pluggy 1.3.0
8+
Root: C:\Users\pytask-dev\git\my_project
9+
Collected 2 tasks.
10+
11+
The following recorded states would be updated:
12+
13+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
14+
┃ Task ┃ Reason ┃
15+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
16+
│ <span class="termynal-dim">task_train.py::</span>task_train_model │ task_change │
17+
│ <span class="termynal-dim">task_train.py::</span>task_evaluate_model │ dependency_change │
18+
└────────────────────────────────────────────────┴──────────────────────┘
19+
20+
<span class="termynal-warning">No changes were written because --dry-run was used.</span>
21+
```
22+
23+
</div>

‎docs/source/_static/md/lock-accept-interactive.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<div class="termy">
2+
3+
```console
4+
5+
$ pytask lock accept -k train
6+
────────────────────────── Start pytask session ─────────────────────────
7+
Platform: win32 -- Python 3.12.0, pytask 0.5.3, pluggy 1.3.0
8+
Root: C:\Users\pytask-dev\git\my_project
9+
Collected 2 tasks.
10+
11+
The following recorded states would be updated:
12+
13+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
14+
┃ Task ┃ Reason ┃
15+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
16+
│ <span class="termynal-dim">task_train.py::</span>task_train_model │ task_change │
17+
│ <span class="termynal-dim">task_train.py::</span>task_evaluate_model │ dependency_change │
18+
└────────────────────────────────────────────────┴──────────────────────┘
19+
20+
Accept recorded state for <span class="termynal-dim">task_train.py::</span>task_train_model? Yes
21+
22+
Accept recorded state for <span class="termynal-dim">task_train.py::</span>task_evaluate_model?
23+
Yes
24+
No
25+
❯ Accept all remaining tasks
26+
Quit
27+
28+
<span class="termynal-success">Accepted recorded state for 2 task(s).</span>
29+
```
30+
31+
</div>

‎docs/source/_static/md/lock-clean.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<div class="termy">
2+
3+
```console
4+
5+
$ pytask lock clean
6+
────────────────────────── Start pytask session ─────────────────────────
7+
Platform: win32 -- Python 3.12.0, pytask 0.5.3, pluggy 1.3.0
8+
Root: C:\Users\pytask-dev\git\my_project
9+
10+
The following stale lockfile entries would be removed:
11+
12+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
13+
┃ Lockfile entry ┃
14+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
15+
│ <span class="termynal-dim">task_old_pipeline.py::</span>task_train_model │
16+
│ <span class="termynal-dim">task_old_pipeline.py::</span>task_evaluate_model │
17+
└───────────────────────────────────────────────────────────────────────┘
18+
19+
Remove stale entry <span class="termynal-dim">task_old_pipeline.py::</span>task_train_model? Yes
20+
21+
Remove stale entry <span class="termynal-dim">task_old_pipeline.py::</span>task_evaluate_model?
22+
Yes
23+
No
24+
❯ Remove all remaining entries
25+
Quit
26+
27+
<span class="termynal-success">Removed 2 stale lockfile entries.</span>
28+
```
29+
30+
</div>

‎docs/source/commands/lock.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# lock
2+
3+
Inspect and update recorded task state in the lockfile.
4+
5+
## Usage
6+
7+
```bash
8+
pytask lock SUBCOMMAND [OPTIONS] [PATHS]
9+
```
10+
11+
Subcommands:
12+
13+
- `accept`: Accept the current state for selected tasks.
14+
- `reset`: Remove recorded state for selected tasks.
15+
- `clean`: Remove stale lockfile entries which no longer correspond to collected tasks.
16+
17+
If no selectors are provided for `accept` or `reset`, the command applies to all
18+
collected tasks in the provided paths.
19+
20+
## Examples
21+
22+
```bash
23+
# Preview which selected tasks would be accepted.
24+
pytask lock accept -k train --dry-run
25+
26+
# Accept the current state for all collected tasks in the current directory.
27+
pytask lock accept --yes
28+
29+
# Also include descendants of the selected tasks.
30+
pytask lock accept -k train --with-descendants
31+
32+
# Restrict the staleness decision to task and dependency changes.
33+
pytask lock accept -k train --run-on task_change,dependency_change
34+
35+
# Reset recorded state for selected tasks.
36+
pytask lock reset -k train --yes
37+
38+
# Remove stale entries from the lockfile.
39+
pytask lock clean --yes
40+
```
41+
42+
## Subcommands
43+
44+
### accept
45+
46+
Accept the current state for selected tasks without executing them.
47+
48+
```bash
49+
pytask lock accept [OPTIONS] [PATHS]
50+
```
51+
52+
Options:
53+
54+
--8<-- "docs/source/_static/md/commands/lock-accept-options.md"
55+
56+
### reset
57+
58+
Remove recorded state for selected tasks.
59+
60+
```bash
61+
pytask lock reset [OPTIONS] [PATHS]
62+
```
63+
64+
Options:
65+
66+
--8<-- "docs/source/_static/md/commands/lock-reset-options.md"
67+
68+
### clean
69+
70+
Remove stale lockfile entries.
71+
72+
```bash
73+
pytask lock clean [OPTIONS] [PATHS]
74+
```
75+
76+
Options:
77+
78+
--8<-- "docs/source/_static/md/commands/lock-clean-options.md"
79+
80+
## Interaction Modes
81+
82+
- Default: Run interactively, show the planned changes, and step through them one by one
83+
with an interactive selector for each task or lockfile entry.
84+
- `--dry-run`: Show the planned changes without writing them.
85+
- `--yes`: Apply the planned changes without prompting.
86+
87+
The options `--dry-run` and `--yes` are mutually exclusive.
88+
89+
## Related
90+
91+
- [Reconciling Lockfile State](../how_to_guides/reconciling_lockfile_state.md)
92+
- [Selecting tasks](../tutorials/selecting_tasks.md)
93+
- [Making Tasks Persist](../tutorials/making_tasks_persist.md)
94+
- [Lockfile](../reference_guides/lockfile.md)

‎docs/source/explanations/execution_policies.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Task Execution Model
2+
3+
The task execution model can be confusing because several questions seem to arrive at
4+
once: is the task still current, should it run, and what happens to the recorded state?
5+
6+
It becomes easier to follow once we stop looking at all of these questions at the same
7+
time and instead watch an ordinary task move through pytask.
8+
9+
## The Normal Case
10+
11+
In the normal case, a task passes through four phases: setup, execution, teardown, and
12+
processing the report.
13+
14+
### Setup
15+
16+
Setup is where pytask decides whether a task needs attention.
17+
18+
It checks a few basic questions first.
19+
20+
- Are all dependencies available and have they changed?
21+
- Do the products already exist and have they changed?
22+
- Has the task itself changed?
23+
24+
Generally, change means whether the current state of a node matches the recorded state
25+
in the lockfile. For most filepath-based nodes, the state is the hash of the file
26+
content.
27+
28+
If everything still matches the previously recorded state, the task is considered
29+
unchanged and pytask will not run it.
30+
31+
If something does not match, the task is stale and pytask prepares to run it.
32+
33+
### Execution
34+
35+
Here pytask calls the task function itself.
36+
37+
### Teardown
38+
39+
Once the task function has returned, pytask checks whether the promised products are
40+
really there.
41+
42+
This matters because a task can finish without raising an exception and still leave the
43+
project in an incomplete state. Teardown is where pytask notices that kind of mismatch.
44+
45+
### Processing The Report
46+
47+
After teardown, pytask knows the outcome of the task. In case the task succeeded, this
48+
is also the moment where pytask updates the recorded state for the task.
49+
50+
## Exceptions to the Rule
51+
52+
Not every task follows this path all the way through.
53+
54+
Markers and runtime options can short-circuit parts of the model. A task can be skipped,
55+
persisted, or forced to run even when it would otherwise be left alone.
56+
57+
### Skipping Tasks
58+
59+
Sometimes a task and its subgraph should not run at all. This might be because the tasks
60+
are too costly, buggy, or irrelevant for the current run.
61+
62+
For temporary exclusions, you would use the [`-m`](<>) or [`-k`](<>) options, maybe in
63+
combination with a `not ...` expression.
64+
65+
For a permanent exclusion, you would use the
66+
[`@pytask.mark.skip`](../api/marks.md#pytask.mark.skip) or
67+
[`@pytask.mark.skipif`](../api/marks.md#pytask.mark.skipif) markers.
68+
69+
Skipping hooks into setup because pytask should stop before execution starts. All
70+
dependent tasks are marked to be skipped as well.
71+
72+
### Persisting Tasks
73+
74+
Persisting is useful when a task would normally be considered stale, but rerunning it is
75+
not desirable. A common example is a change to the task file that does not change the
76+
actual result, such as formatting.
77+
78+
Persisting also has to hook into setup because pytask needs to decide early that the
79+
task should not execute even though something changed. Later, when the report is
80+
processed, pytask accepts the current state as the new recorded state.
81+
82+
Users can opt into this behavior with the
83+
[`persist`](../tutorials/making_tasks_persist.md) marker. The disadvantage of the
84+
decorator is that it is permanent and removing it triggers a rerun of the task.
85+
86+
What is missing is a temporary option of the persisting mechanism that would also allow
87+
to integrate new or renamed tasks into an existing project without triggering a rerun.
88+
89+
### Forcing Tasks to Run
90+
91+
Sometimes the opposite is needed: a task should run even though pytask would otherwise
92+
consider it unchanged.
93+
94+
This has to affect setup as well because setup is where pytask normally decides that an
95+
unchanged task can be left alone. Forcing a task to run overrides that decision and lets
96+
the task continue to execution.
97+
98+
Users can request this behavior with the `--force` option.
99+
100+
### Fine-grained Staleness Decision
101+
102+
The current model mostly answers staleness as a yes-or-no question based on recorded
103+
state and the current state of the task, its dependencies, and its products.
104+
105+
Issue [#403](https://github.com/pytask-dev/pytask/issues/403) explores whether this
106+
decision could become more fine-grained. For example, users might want to treat changes
107+
to the task source differently from changes to dependencies or products, or they might
108+
want to override the default decision for one run without changing the task itself.

‎docs/source/explanations/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,5 @@ systems in general as well as its design.
55

66
- [Why Pytask](why_pytask.md)
77
- [Comparison To Other Tools](comparison_to_other_tools.md)
8+
- [Execution Policies](execution_policies.md)
89
- [Pluggy](pluggy.md)

0 commit comments

Comments
 (0)