Skip to content

Latest commit

 

History

History
851 lines (618 loc) · 50.9 KB

File metadata and controls

851 lines (618 loc) · 50.9 KB

eigenpal workflow

Manage workflows: push, pull, run, evaluate.

Contents

Surface

workflow
├── list|ls
├── pull <workflow-id>
├── push
├── move <workflow-id>
├── evaluators
│   ├── pull <workflow-id>
│   ├── push <workflow-id>
│   └── validate [path]
├── dataset
│   ├── list|ls <workflow-id>
│   ├── pull <workflow-id>
│   ├── push <workflow-id>
│   ├── example
│   │   ├── create <workflow-id>
│   │   ├── update <workflow-id> <exampleId>
│   │   ├── delete <workflow-id> <exampleId>
│   │   └── get <workflow-id> <exampleId>
│   └── validate [path]
├── experiment|exp
│   ├── list|ls <workflow-id>
│   ├── run <workflow-id>
│   ├── status <workflow-id> <batchId>
│   ├── cancel <workflow-id> <batchId>
│   ├── results <workflow-id> [batchId]
│   ├── compare|diff <batchIdA> <batchIdB>
│   └── watch <workflow-id> <batchId>
├── execution|exec
│   ├── run <workflow-id> [examples...]
│   ├── get <executionId>
│   ├── list|ls <workflow-id>
│   ├── watch <executionId>
│   ├── compare|diff <executionA> <executionB>
│   └── cancel <executionId>
├── versions
│   ├── list|ls <workflow-id>
│   └── restore <workflow-id> <versionId>
├── step-type
│   ├── list|ls
│   └── get <type>
├── evaluator-type
│   ├── list|ls
│   └── get <type>
├── validate
├── clear-local [examples...]
└── step
    └── exec <type>

Commands

Core

| Command | Description | | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | eigenpal workflow list | ls [options] | List workflows the caller can read. | | eigenpal workflow pull [options] <workflow-id> | Download the YAML definition of the workflow at its current version. | | eigenpal workflow push [options] | Create or update a workflow from a YAML file. | | eigenpal workflow move [options] <workflow-id> | Move a workflow to a folder path, creating folders as needed | | eigenpal workflow validate [options] | Local-only validation against the templated project layout: ./workflow.yaml + ./evaluators.yaml + ./dataset/. For targeted validation use evaluators validate or dataset validate. | | eigenpal workflow clear-local [options] [examples...] | Delete local execution artifacts under ./dataset/examples/. Keeps the latest run per example by default. |

Evaluators

Command Description
eigenpal workflow evaluators pull [options] <workflow-id> Download the workflow's evaluators YAML.
eigenpal workflow evaluators push [options] <workflow-id> Overwrite the workflow's evaluator config from a YAML file.
eigenpal workflow evaluators validate [options] [path] Validate an evaluators YAML file against the EvalConfig schema. Defaults to ./evaluators.yaml.

Dataset

| Command | Description | | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | ------------------------------------ | | eigenpal workflow dataset list | ls [options] <workflow-id> | List eval examples for the workflow. | | eigenpal workflow dataset pull [options] <workflow-id> | Download the workflow's dataset as a ZIP archive. | | eigenpal workflow dataset push [options] <workflow-id> | Replace or extend the workflow's dataset from a ZIP or folder. | | eigenpal workflow dataset example create [options] <workflow-id> | Create one eval example without re-uploading the dataset. | | eigenpal workflow dataset example update [options] <workflow-id> <exampleId> | Patch one eval example. Omitted flags are left alone. | | eigenpal workflow dataset example delete [options] <workflow-id> <exampleId> | Delete one eval example by id. Non-TTY shells require --yes. | | eigenpal workflow dataset example get [options] <workflow-id> <exampleId> | Fetch one eval example with full triggerInput, expectedOutput, and metadata. | | eigenpal workflow dataset validate [options] [path] | Validate a dataset folder against the examples//{input,expected,meta} convention. Defaults to ./dataset/. |

Experiment

| Command | Description | | ----------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------ | | eigenpal workflow experiment | exp list | ls [options] <workflow-id> | List executions for the workflow, newest first. | | eigenpal workflow experiment | exp run [options] <workflow-id> | Start a batch eval against the workflow's dataset. | | eigenpal workflow experiment | exp status [options] <workflow-id> <batchId> | Aggregate progress for a batch by batchId. | | eigenpal workflow experiment | exp cancel [options] <workflow-id> <batchId> | Cancel every execution in a batch. Idempotent. | | eigenpal workflow experiment | exp results [options] <workflow-id> [batchId] | Download eval results in CSV or JSON. | | eigenpal workflow experiment | exp compare | diff [options] <batchIdA> <batchIdB> | Diff eval scores between two experiment batches. | | eigenpal workflow experiment | exp watch [options] <workflow-id> <batchId> | Poll until terminal, then auto-pull results — replaces status --watch + results --out. |

Execution

| Command | Description | | ---------------------------- | ----------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------- | | eigenpal workflow execution | exec run [options] <workflow-id> [examples...] | Run a saved workflow against local dataset examples. | | eigenpal workflow execution | exec get [options] <executionId> | Fetch a single execution payload. Optionally narrow to one step. | | eigenpal workflow execution | exec list | ls [options] <workflow-id> | List recent executions for a workflow. | | eigenpal workflow execution | exec watch [options] <executionId> | Stream live execution status until terminal or 30-min detach. | | eigenpal workflow execution | exec compare | diff [options] <executionA> <executionB> | Diff two executions side-by-side, per step. | | eigenpal workflow execution | exec cancel [options] <executionId> | Cancel an execution. Idempotent on already-terminal runs. |

Versions

| Command | Description | | ------------------------------------------------------------------------ | ------------------------------------------- | ------------------------------------------------ | | eigenpal workflow versions list | ls [options] <workflow-id> | List historical workflow versions, newest first. | | eigenpal workflow versions restore [options] <workflow-id> <versionId> | Restore the workflow to a previous version. |

Step-type

| Command | Description | | -------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------- | | eigenpal workflow step-type list | ls [options] | List every step type the deployment supports. | | eigenpal workflow step-type get [options] <type> | Return the full schema and behavioral docs for one step type. |

Evaluator-type

| Command | Description | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | eigenpal workflow evaluator-type list | ls [options] | List every evaluator type with a one-line description. | | eigenpal workflow evaluator-type get [options] <type> | Fetch the JSON Schema for one evaluator type. Pipe through jq to inspect specific fields. |

Step

Command Description
eigenpal workflow step exec [options] <type> DISABLED — local mimic runners removed pending server-side redesign (EIG-104). Use workflow execution run or workflow experiment run instead.

Details

eigenpal workflow list|ls [options]

List workflows the caller can read.

Options

Flag Required Default Description
--search <q> no Filter by name
--limit <n> no 50 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow pull [options] <workflow-id>

Download the YAML definition of the workflow at its current version.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--out <path> no Write YAML to file instead of stdout
--base-url <url> no Server base URL

eigenpal workflow push [options]

Create or update a workflow from a YAML file.

Options

Flag Required Default Description
--file <yaml> no Path to YAML file
--workflow-id <id> no Update existing workflow (default: create new)
--bump <level> no Auto-bump from the server's current version: patch | minor | major. Mutually exclusive with --version and with a top-level version: in the YAML.
--set-version <semver> no Push at this exact semver (e.g. 1.4.0). Mutually exclusive with --bump and with a top-level version: in the YAML. (Named --set-version to avoid the global -v, --version flag.)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow move [options] <workflow-id>

Move a workflow to a folder path, creating folders as needed

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--folder <path> yes Target folder path (/ for root)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow validate [options]

Local-only validation against the templated project layout: ./workflow.yaml + ./evaluators.yaml + ./dataset/. For targeted validation use evaluators validate or dataset validate.

Options

Flag Required Default Description
--dir <path> no Project root (defaults to cwd; resolves the three default paths from here)

eigenpal workflow clear-local [options] [examples...]

Delete local execution artifacts under ./dataset/examples/. Keeps the latest run per example by default.

Arguments

Name Required Variadic Description
examples no yes

Options

Flag Required Default Description
--dir <dir> no Local eigenpal directory
--all no false Remove all artifacts, including the latest kept by default

eigenpal workflow evaluators pull [options] <workflow-id>

Download the workflow's evaluators YAML.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--out <path> no Write YAML to file instead of stdout
--base-url <url> no Server base URL

eigenpal workflow evaluators push [options] <workflow-id>

Overwrite the workflow's evaluator config from a YAML file.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--file <yaml> yes Path to evaluators YAML file
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow evaluators validate [options] [path]

Validate an evaluators YAML file against the EvalConfig schema. Defaults to ./evaluators.yaml.

Arguments

Name Required Variadic Description
path no no

eigenpal workflow dataset list|ls [options] <workflow-id>

List eval examples for the workflow.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--limit <n> no 100 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow dataset pull [options] <workflow-id>

Download the workflow's dataset as a ZIP archive.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--out <zip> no Write the dataset ZIP to this path. When omitted, the binary streams to stdout.
--base-url <url> no Server base URL

eigenpal workflow dataset push [options] <workflow-id>

Replace or extend the workflow's dataset from a ZIP or folder.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--file <path> yes Path to a dataset ZIP file or a dataset/ folder (folder is zipped in memory before upload)
--mode <append|replace> no "append" Import mode (default: append)
--yes no false Skip the destructive confirmation prompt for --mode replace (use in CI)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow dataset example create [options] <workflow-id>

Create one eval example without re-uploading the dataset.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--name <name> yes Example name (1–64 chars)
--input-json <json> no Trigger input as a JSON literal
--input-file <path> no Trigger input from a JSON file (or - for stdin)
--expected-json <json> no Expected output as a JSON literal
--expected-file <path> no Expected output from a JSON file (or - for stdin)
--annotation <text> no Free-form annotation
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow dataset example update [options] <workflow-id> <exampleId>

Patch one eval example. Omitted flags are left alone.

Arguments

Name Required Variadic Description
workflow-id yes no
exampleId yes no

Options

Flag Required Default Description
--name <name> no Rename the example (1–64 chars)
--input-json <json> no Replace trigger input with this JSON literal
--input-file <path> no Replace trigger input from a JSON file (or - for stdin)
--expected-json <json> no Replace expected output with this JSON literal
--expected-file <path> no Replace expected output from a JSON file (or - for stdin)
--annotation <text> no Replace annotation; pass empty string to clear
--row-order <n> no Reorder the row (0-based)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow dataset example delete [options] <workflow-id> <exampleId>

Delete one eval example by id. Non-TTY shells require --yes.

Arguments

Name Required Variadic Description
workflow-id yes no
exampleId yes no

Options

Flag Required Default Description
--yes no false Required for non-TTY shells; explicit acknowledgment that this is destructive
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow dataset example get [options] <workflow-id> <exampleId>

Fetch one eval example with full triggerInput, expectedOutput, and metadata.

Arguments

Name Required Variadic Description
workflow-id yes no
exampleId yes no

Options

Flag Required Default Description
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow dataset validate [options] [path]

Validate a dataset folder against the examples//{input,expected,meta} convention. Defaults to ./dataset/.

Arguments

Name Required Variadic Description
path no no

eigenpal workflow experiment|exp list|ls [options] <workflow-id>

List executions for the workflow, newest first.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--batch-id <id> no Filter by batch
--limit <n> no 50 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow experiment|exp run [options] <workflow-id>

Start a batch eval against the workflow's dataset.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--example-id <id> no [] Run only this example (repeatable)
--wait no false Poll until terminal; non-zero exit on passRate < 1.0
--interval <n> no 10 Polling interval in seconds (default 10)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow experiment|exp status [options] <workflow-id> <batchId>

Aggregate progress for a batch by batchId.

Arguments

Name Required Variadic Description
workflow-id yes no
batchId yes no

Options

Flag Required Default Description
--watch no false Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit
--short no false Single-line plain-text summary on stdout (e.g. 6/6 done failed=0 cancelled=0 rejected=0). Pipe-friendly for monitoring loops; mutually exclusive with --watch.
--interval <seconds> no 5 Poll interval in seconds when --watch is set (default 5)
--max-wait <seconds> no 1800 Hard ceiling for --watch in seconds (default 1800 = 30 min)
--include <kinds> no "" Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow experiment|exp cancel [options] <workflow-id> <batchId>

Cancel every execution in a batch. Idempotent.

Arguments

Name Required Variadic Description
workflow-id yes no
batchId yes no

Options

Flag Required Default Description
--yes no Required for non-TTY shells (CI, pipes). Acts immediately, no prompt.
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow experiment|exp results [options] <workflow-id> [batchId]

Download eval results in CSV or JSON.

Arguments

Name Required Variadic Description
workflow-id yes no
batchId no no

Options

Flag Required Default Description
--format <csv|json> yes Output format
--out <path> no Output file. When omitted, the binary streams to stdout.
--base-url <url> no Server base URL

eigenpal workflow experiment|exp compare|diff [options] <batchIdA> <batchIdB>

Diff eval scores between two experiment batches.

Arguments

Name Required Variadic Description
batchIdA yes no
batchIdB yes no

Options

Flag Required Default Description
--sort <abs-delta-desc|delta-asc|delta-desc|name> no "abs-delta-desc" Row sort order (default: biggest movers first)
--regression-threshold <n> no 0.05 Δ below this is flagged as a regression (default 0.05)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow experiment|exp watch [options] <workflow-id> <batchId>

Poll until terminal, then auto-pull results — replaces status --watch + results --out.

Arguments

Name Required Variadic Description
workflow-id yes no
batchId yes no

Options

Flag Required Default Description
--interval <seconds> no 5 Poll interval in seconds (default 5)
--max-wait <seconds> no 1800 Hard ceiling in seconds (default 1800 = 30 min)
--pull-on-complete <path> no Destination for the results file. Default: ./results-.
--format <csv|json> no "json" Results export format (default json)
--no-pull no Skip auto-pulling results on terminal (just watch)
--base-url <url> no Server base URL

eigenpal workflow execution|exec run [options] <workflow-id> [examples...]

Run a saved workflow against local dataset examples.

Arguments

Name Required Variadic Description
workflow-id yes no
examples no yes

Options

Flag Required Default Description
--dir <dir> no Local eigenpal directory
--concurrency <n> no Max examples to run in parallel (default: 3)
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow execution|exec get [options] <executionId>

Fetch a single execution payload. Optionally narrow to one step.

Arguments

Name Required Variadic Description
executionId yes no

Options

Flag Required Default Description
--step <name> no Show only this step (or comma-separated list)
--include <kinds> no "input,output,error,duration" Comma-separated subset of input,output,error,duration. input projects the resolved templated config (= what the processor actually received). inputRef returns the minimal predecessor-id reference instead.
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow execution|exec list|ls [options] <workflow-id>

List recent executions for a workflow.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--status <status> no Filter by status: pending|running|completed|failed|cancelled
--limit <n> no 50 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow execution|exec watch [options] <executionId>

Stream live execution status until terminal or 30-min detach.

Arguments

Name Required Variadic Description
executionId yes no

Options

Flag Required Default Description
--max-wait <seconds> no 1800 Detach after N seconds (default 1800 = 30 min)
--base-url <url> no Server base URL

eigenpal workflow execution|exec compare|diff [options] <executionA> <executionB>

Diff two executions side-by-side, per step.

Arguments

Name Required Variadic Description
executionA yes no
executionB yes no

Options

Flag Required Default Description
--step <name> no Restrict comparison to one step
--base-url <url> no Server base URL

eigenpal workflow execution|exec cancel [options] <executionId>

Cancel an execution. Idempotent on already-terminal runs.

Arguments

Name Required Variadic Description
executionId yes no

Options

Flag Required Default Description
--yes no Required for non-TTY shells (CI, pipes). Acts immediately, no prompt.
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow versions list|ls [options] <workflow-id>

List historical workflow versions, newest first.

Arguments

Name Required Variadic Description
workflow-id yes no

Options

Flag Required Default Description
--limit <n> no 50 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow versions restore [options] <workflow-id> <versionId>

Restore the workflow to a previous version.

Arguments

Name Required Variadic Description
workflow-id yes no
versionId yes no

Options

Flag Required Default Description
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow step-type list|ls [options]

List every step type the deployment supports.

Options

Flag Required Default Description
--search <q> no Filter
--limit <n> no 50 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow step-type get [options] <type>

Return the full schema and behavioral docs for one step type.

Arguments

Name Required Variadic Description
type yes no

Options

Flag Required Default Description
--out <path> no Write to file

eigenpal workflow evaluator-type list|ls [options]

List every evaluator type with a one-line description.

Options

Flag Required Default Description
--search <q> no Filter by type, name, or description
--limit <n> no 50 Page size
--offset <n> no 0 Page offset
--base-url <url> no Server base URL
--json no Output the raw server response as JSON

eigenpal workflow evaluator-type get [options] <type>

Fetch the JSON Schema for one evaluator type. Pipe through jq to inspect specific fields.

Arguments

Name Required Variadic Description
type yes no

Options

Flag Required Default Description
--out <path> no Write to file

eigenpal workflow step exec [options] <type>

DISABLED — local mimic runners removed pending server-side redesign (EIG-104). Use workflow execution run or workflow experiment run instead.

Arguments

Name Required Variadic Description
type yes no

Options

Flag Required Default Description
--config-json <json> no (unused; kept for back-compat parsing)
--config-file <path> no (unused)
--inputs <pairs...> no (unused)
--output-schema <path> no (unused)