Implement a portable lockfile.

Author: tobiasraabe

CHANGELOG.md

@@ -7,7 +7,9 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 
 ## Unreleased
 
-- Nothing yet.
+- {issue}`735` adds the `pytask.lock` lockfile as the primary state backend with a
+  portable format, documentation, and a one-run SQLite fallback when no lockfile
+  exists.
 
 ## 0.5.8 - 2025-12-30
 

docs/source/how_to_guides/index.md

@@ -13,6 +13,7 @@ maxdepth: 1
 ---
 migrating_from_scripts_to_pytask
 interfaces_for_dependencies_products
+portability
 remote_files
 functional_interface
 capture_warnings

docs/source/how_to_guides/portability.md

@@ -0,0 +1,32 @@
+# Portability
+
+This guide explains how to keep pytask state portable across machines.
+
+## Two Portability Concerns
+
+1. **Portable IDs**
+
+   - The lockfile stores task and node IDs.
+   - IDs must be project‑relative and stable across machines.
+   - pytask builds these IDs from the project root; no action required for most users.
+
+1. **Portable State Values**
+
+   - `state.value` is opaque and comes from `PNode.state()` / `PTask.state()`.
+   - Content hashes are portable; timestamps or absolute paths are not.
+   - Custom nodes should avoid machine‑specific paths in `state()`.
+
+## Tips
+
+- Commit `pytask.lock` to your repository. If you ship the repository together with the
+  build artifacts (for example, a zipped project folder including `pytask.lock` and the
+  produced files), you can move it to another machine and runs will skip recomputation.
+- Prefer file content hashes over timestamps for custom nodes.
+- For `PythonNode` values that are not natively stable, provide a custom hash function.
+- If inputs live outside the project root, IDs will include `..` segments to remain
+  relative; this is expected.
+
+## Legacy SQLite
+
+SQLite is the old state format. It is used only when no lockfile exists, and the
+lockfile is written during that run. Subsequent runs rely on the lockfile.

docs/source/reference_guides/configuration.md

@@ -44,11 +44,12 @@ are welcome to also support macOS.
 
 ````{confval} database_url
 
-pytask uses a database to keep track of tasks, products, and dependencies over runs. By
-default, it will create an SQLite database in the project's root directory called
-`.pytask/pytask.sqlite3`. If you want to use a different name or a different dialect
-[supported by sqlalchemy](https://docs.sqlalchemy.org/en/latest/core/engines.html#backend-specific-urls),
-use either {option}`pytask build --database-url` or `database_url` in the config.
+SQLite is the legacy state format. pytask now uses `pytask.lock` as the primary state
+backend and only consults the database when no lockfile exists. During that first run,
+the lockfile is written and subsequent runs use the lockfile only.
+
+The `database_url` option remains for backwards compatibility and controls the legacy
+database location and dialect ([supported by sqlalchemy](https://docs.sqlalchemy.org/en/latest/core/engines.html#backend-specific-urls)).
 
 ```toml
 database_url = "sqlite:///.pytask/pytask.sqlite3"

docs/source/reference_guides/index.md

@@ -9,6 +9,7 @@ maxdepth: 1
 ---
 command_line_interface
 configuration
+lockfile
 hookspecs
 api
 ```

docs/source/reference_guides/lockfile.md

@@ -0,0 +1,90 @@
+# The Lock File
+
+`pytask.lock` is the default state backend. It stores task state in a portable,
+git-friendly format so runs can be resumed or shared across machines.
+
+```{note}
+SQLite is the legacy format. It is still read when no lockfile exists, and a lockfile
+is written during that first run. Subsequent runs use the lockfile only.
+```
+
+## Example
+
+```toml
+# This file is automatically @generated by pytask.
+# It is not intended for manual editing.
+
+lock-version = "1.0"
+
+[[task]]
+id = "src/tasks/data.py::task_clean_data"
+
+[task.state]
+value = "f9e8d7c6..."
+
+[[task.depends_on]]
+id = "data/raw/input.csv"
+
+[task.depends_on.state]
+value = "e5f6g7h8..."
+
+[[task.produces]]
+id = "data/processed/clean.parquet"
+
+[task.produces.state]
+value = "m3n4o5p6..."
+```
+
+## Behavior
+
+On each run, pytask:
+
+1. Reads `pytask.lock` (if present).
+1. Compares current dependency/product/task `state()` to stored `state.value`.
+1. Skips tasks whose states match; runs the rest.
+1. Updates `pytask.lock` after each completed task (atomic write).
+
+`pytask-parallel` uses a single coordinator to write the lock file, so writes are
+serialized even when tasks execute in parallel.
+
+## Portability
+
+There are two portability concerns:
+
+1. **IDs**: Lockfile IDs must be project‑relative and stable across machines.
+1. **State values**: `state.value` is opaque; portability depends on each node’s
+   `state()` implementation. Content hashes are portable; timestamps are not.
+
+## File Format Reference
+
+### Top-Level
+
+| Field          | Required | Description                        |
+| -------------- | -------- | ---------------------------------- |
+| `lock-version` | Yes      | Schema version (currently `"1.0"`) |
+
+### Task Entry
+
+| Field   | Required | Description                                  |
+| ------- | -------- | -------------------------------------------- |
+| `id`    | Yes      | Portable task identifier                     |
+| `state` | Yes      | State dictionary with a single `value` field |
+
+### Dependency/Product Entry
+
+| Field   | Required | Description                                  |
+| ------- | -------- | -------------------------------------------- |
+| `id`    | Yes      | Node identifier                              |
+| `state` | Yes      | State dictionary with a single `value` field |
+
+### State Dictionary
+
+| Field   | Required | Description         |
+| ------- | -------- | ------------------- |
+| `value` | Yes      | Opaque state string |
+
+## Version Compatibility
+
+- **Upgrade**: newer pytask upgrades old lock files in memory and writes the new format
+  on the next update.
+- **Downgrade**: older pytask errors with a clear upgrade message.

docs/source/tutorials/making_tasks_persist.md

@@ -9,7 +9,7 @@ In this case, you can apply the {func}`@pytask.mark.persist <pytask.mark.persist
 decorator to the task, which will skip its execution as long as all products exist.
 
 Internally, the state of the dependencies, the source file, and the products are updated
-in the database such that the subsequent execution will skip the task successfully.
+in the lockfile such that the subsequent execution will skip the task successfully.
 
 ## When is this useful?
 

pyproject.toml

@@ -30,6 +30,7 @@ dependencies = [
     "pluggy>=1.3.0",
     "rich>=13.8.0",
     "sqlalchemy>=2.0.31",
+    "msgspec[toml]>=0.18.6",
     'tomli>=1; python_version < "3.11"',
     'typing-extensions>=4.8.0; python_version < "3.11"',
     "universal-pathlib>=0.2.2",

src/_pytask/console.py

@@ -111,10 +111,26 @@ def render_to_string(
     example, render warnings with colors or text in exceptions.
 
     """
-    buffer = console.render(renderable)
+    render_console = console
+    if not strip_styles and console.no_color and console.color_system is not None:
+        theme: Theme | None
+        try:
+            theme = Theme(console._theme_stack._entries[-1])  # type: ignore[attr-defined]
+        except (AttributeError, IndexError, TypeError):
+            theme = None
+        render_console = Console(
+            color_system=console.color_system,
+            force_terminal=True,
+            width=console.width,
+            no_color=False,
+            markup=getattr(console, "_markup", True),
+            theme=theme,
+        )
+
+    buffer = render_console.render(renderable)
     if strip_styles:
         buffer = Segment.strip_styles(buffer)
-    return console._render_buffer(buffer)
+    return render_console._render_buffer(buffer)
 
 
 def format_task_name(task: PTask, editor_url_scheme: str) -> Text:

src/_pytask/execute.py

@@ -20,9 +20,6 @@
 from _pytask.dag_utils import TopologicalSorter
 from _pytask.dag_utils import descending_tasks
 from _pytask.dag_utils import node_and_neighbors
-from _pytask.database_utils import get_node_change_info
-from _pytask.database_utils import has_node_changed
-from _pytask.database_utils import update_states_in_database
 from _pytask.exceptions import ExecutionError
 from _pytask.exceptions import NodeLoadError
 from _pytask.exceptions import NodeNotFoundError
@@ -46,6 +43,9 @@
 from _pytask.pluginmanager import hookimpl
 from _pytask.provisional_utils import collect_provisional_products
 from _pytask.reports import ExecutionReport
+from _pytask.state import get_node_change_info
+from _pytask.state import has_node_changed
+from _pytask.state import update_states
 from _pytask.traceback import remove_traceback_from_exc_info
 from _pytask.tree_util import tree_leaves
 from _pytask.tree_util import tree_map
@@ -196,7 +196,7 @@ def pytask_execute_task_setup(session: Session, task: PTask) -> None:  # noqa: C
             # Check if node changed and collect detailed info if in explain mode
             if session.config["explain"]:
                 has_changed, reason, details = get_node_change_info(
-                    task=task, node=node, state=node_state
+                    session=session, task=task, node=node, state=node_state
                 )
                 if has_changed:
                     needs_to_be_executed = True
@@ -222,7 +222,9 @@ def pytask_execute_task_setup(session: Session, task: PTask) -> None:  # noqa: C
                         )
                     )
             else:
-                has_changed = has_node_changed(task=task, node=node, state=node_state)
+                has_changed = has_node_changed(
+                    session=session, task=task, node=node, state=node_state
+                )
                 if has_changed:
                     needs_to_be_executed = True
 
@@ -232,6 +234,8 @@ def pytask_execute_task_setup(session: Session, task: PTask) -> None:  # noqa: C
 
     if not needs_to_be_executed:
         collect_provisional_products(session, task)
+        if not session.config["dry_run"] and not session.config["explain"]:
+            update_states(session, task)
         raise SkippedUnchanged
 
     # Create directory for product if it does not exist. Maybe this should be a `setup`
@@ -326,7 +330,7 @@ def pytask_execute_task_process_report(
     task = report.task
 
     if report.outcome == TaskOutcome.SUCCESS:
-        update_states_in_database(session, task.signature)
+        update_states(session, task)
     elif report.exc_info and isinstance(report.exc_info[1], WouldBeExecuted):
         report.outcome = TaskOutcome.WOULD_BE_EXECUTED