diff --git a/docs.json b/docs.json index 8bc7f09d99..ff922e5298 100644 --- a/docs.json +++ b/docs.json @@ -391,8 +391,17 @@ "models/automations/create-automations/webhook" ] }, + { + "group": "Tutorials", + "pages": [ + "models/automations/tutorial", + "models/automations/project-automation-tutorial", + "models/automations/registry-automation-tutorial" + ] + }, "models/automations/view-automation-history", - "models/automations/automation-events" + "models/automations/automation-events", + "models/automations/api" ] }, { diff --git a/ja/models/automations/create-automations/webhook.mdx b/ja/models/automations/create-automations/webhook.mdx index f5ed624d19..1ddb052be8 100644 --- a/ja/models/automations/create-automations/webhook.mdx +++ b/ja/models/automations/create-automations/webhook.mdx @@ -291,7 +291,6 @@ W&B が `POST` リクエストに使用する形式の詳細については、** * `PAYLOAD` * `API_ENDPOINT` - ```bash webhook_test.sh #!/bin/bash diff --git a/ko/models/automations/automation-events.mdx b/ko/models/automations/automation-events.mdx index 80ed20fbbf..c7ca96051e 100644 --- a/ko/models/automations/automation-events.mdx +++ b/ko/models/automations/automation-events.mdx @@ -129,7 +129,6 @@ z-score 자동화를 생성하려면 다음을 구성합니다: - z-score가 -2.0이면 메트릭이 평균보다 2 표준 편차 낮음을 의미합니다. - ±2를 벗어나는 값은 종종 통계적으로 유의미한 이상치(outlier)로 간주됩니다. - #### Run 필터 이 섹션에서는 자동화가 평가할 run을 선택하는 방법에 대해 설명합니다. diff --git a/ko/models/automations/create-automations/webhook.mdx b/ko/models/automations/create-automations/webhook.mdx index 6445a0f3ea..34d160b3dd 100644 --- a/ko/models/automations/create-automations/webhook.mdx +++ b/ko/models/automations/create-automations/webhook.mdx @@ -291,7 +291,6 @@ webhook 문제를 해결하려면 아래 코드를 쉘 스크립트에 복사하 * `PAYLOAD` * `API_ENDPOINT` - ```bash webhook_test.sh #!/bin/bash diff --git a/models/automations.mdx b/models/automations.mdx index 7ed76575ad..96add7dbe8 100644 --- a/models/automations.mdx +++ b/models/automations.mdx @@ -1,57 +1,72 @@ --- title: Automations overview +sidebarTitle: Overview description: Use W&B Automations for triggering workflows based on events in W&B --- +import AutomationsMentalModel from "/snippets/en/_includes/automations/mental-model.mdx"; +import AutomationsActionsList from "/snippets/en/_includes/automations/actions-list.mdx"; +import AutomationsBestPractices from "/snippets/en/_includes/automations/best-practices.mdx"; +import AutomationsWhereToFind from "/snippets/en/_includes/automations/where-to-find-automations.mdx"; -This page describes _automations_ in W&B. [Create an automation](/models/automations/create-automations/) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B. +Automations exist for both **projects** and **registries**. Where you create an automation, which events you can use, and how scope works all differ. For event types by scope, see [Automation events and scopes](/models/automations/automation-events). -For example, an automation can notify a Slack channel when a new version is created, trigger an automated testing webhook when the `production` alias is added to an artifact, or start a validation job only when a run's `loss` is within acceptable bounds. + - -Looking for companion tutorials for automations? -- [Learn to automatically trigger a Github Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw). -- [Watch a video demonstrating automatically deploying a model to a Sagemaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ). -- [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6&feature=shared). - +```mermaid +flowchart LR + Event[Event] --> Condition[Condition] + Condition --> Action[Action] +``` + +**Example:** Run fails (event) and optional run name filter (condition) then Slack notification (action). Or: alias `production` added (event) then webhook (action). -## Automation events +## Where to create automations -Automations can be triggered by events related to runs or artifacts in a collection, project, or registry. An automation can start: + -- When a run metric meets a defined absolute or relative threshold. For example, trigger when a run's `loss` is less than 0.01 or when a run's `accuracy` improves by 5%. -- When a run's z-score (standard score) deviates from the mean by a given amount. A z-score of 0 indicates that the result aligns with the mean. A z-score of +2 means that the result is two standard deviations higher than the mean, and a z-score of -2 means that the result is two standard deviations lower than the mean. -- When an event occurs in a project or registry. For example, trigger when a new version of a model artifact is created or when the `production` alias is added to a model artifact. +## Use cases -For more details, see [Automation events and scopes](/models/automations/automation-events/). +- **Run monitoring and alerting**: Notify the team when a run fails or when a metric crosses a threshold (for example, loss goes to NaN or accuracy drops). +- **Registry CI/CD**: When a new model version is linked or an alias (such as `staging` or `production`) is added, trigger a webhook to run tests or deploy. +- **Project artifact workflows**: When a new artifact version is created or an alias is added in a project, run a downstream job or post to Slack. + +For full event and scope details, see [Automation events and scopes](/models/automations/automation-events). ## Automation actions When an event triggers an automation, it can perform one of these actions: -### Slack notification -Send a message to a Slack channel with details about the triggering event. The message summarizes the event, with a link to view more details in W&B. - -### Webhook -Call a webhook URL with a JSON payload containing information about the triggering event. This enables integration with external systems like CI/CD pipelines, model deployment services, or custom workflows. The body of the webhook request may be any JSON-serializable payload. + -For implementation details, see: -- [Create a Slack automation](/models/automations/create-automations/slack/) -- [Create a webhook automation](/models/automations/create-automations/webhook/) +For implementation details, see [Create a Slack automation](/models/automations/create-automations/slack) and [Create a webhook automation](/models/automations/create-automations/webhook). ## How automations work -To [create an automation](/models/automations/create-automations/), you: +To [create an automation](/models/automations/create-automations), you: -1. If required, configure [secrets](/platform/secrets/) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations to securely pass credentials or tokens to the webhook's external service without exposing it in plain text or hard-coding it in the webhook's payload. +1. If required, configure [secrets](/platform/secrets) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations to securely pass credentials or tokens to the webhook's external service without exposing it in plain text or hard-coding it in the webhook's payload. 1. Configure team-level webhook or Slack integrations to authorize W&B to post to Slack or run the webhook on your behalf. A single automation action (webhook or Slack notification) can be used by multiple automations. These actions are defined in your **Team Settings**. 1. In the project or registry, create the automation: - 1. Define the [event](#automation-events) to watch for, such as when a new artifact version is added. + 1. Define the [event](/models/automations/automation-events) to watch for, such as when a new artifact version is added. 1. Define the action to take when the event occurs (posting to a Slack channel or running a webhook). For a webhook, specify a secret to use for the access token and/or a secret to send with the payload, if required. +## Recommendations + + + ## Limitations [Run metric automations](/models/automations/automation-events/#run-metrics-events) and [run metrics z-score change automations](/models/automations/automation-events/#run-metrics-z-score-change-automations) are currently supported only in [W&B Multi-tenant Cloud](/platform/hosting/#wb-multi-tenant-cloud). ## Next steps -- [Create an automation](/models/automations/create-automations/). -- Learn about [Automation events and scopes](/models/automations/automation-events/). -- [Create a secret](/platform/secrets/). +- [Automations tutorial](/models/automations/tutorial): Guides you to create a project automation to alert on run failures and a Registry automation to run a webhoook when an alias is added. The tutorial provides both UI and API instructions. +- [Create an automation](/models/automations/create-automations). +- [Manage automations with the API](/models/automations/api). +- [Automation events and scopes](/models/automations/automation-events). +- [Create a secret](/platform/secrets). + + +Looking for companion tutorials for automations? +- [Learn to automatically trigger a Github Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw). +- [Watch a video demonstrating automatically deploying a model to a Sagemaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ). +- [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6&feature=shared). + diff --git a/models/automations/api.mdx b/models/automations/api.mdx new file mode 100644 index 0000000000..59a4c1abfb --- /dev/null +++ b/models/automations/api.mdx @@ -0,0 +1,132 @@ +--- +title: Manage automations with the API +description: Create, list, get, update, and delete W&B automations using the Python API +--- + +You can manage W&B [automations](/models/automations) programmatically with the public Python API. This page summarizes the main operations. For full method signatures and parameters, see the [Automations API reference](/models/ref/python/public-api/automations). + +## List automations + +Use `api.automations()` to iterate over automations. You can filter by entity and optionally by name. + +```python +import wandb + +api = wandb.Api() + +# All automations for an entity (team or user) +for automation in api.automations(entity="my-team"): + print(automation.name, automation.scope) + +# Filter by name (returns matching automations) +for automation in api.automations(entity="my-team", name="my-automation"): + print(automation.name) +``` + +Automations can be scoped to a [project](/models/automations/automation-events/#project) or a [registry](/models/automations/automation-events/#registry). The returned `Automation` objects include scope information. + +## Get one automation + +Use `api.automation()` when you know the automation name. The call returns the only automation matching the name (and entity, if provided). + +```python +import wandb + +api = wandb.Api() + +automation = api.automation(name="my-automation") +print(automation.description, automation.enabled) + +# From a specific entity +automation = api.automation(name="my-automation", entity="my-team") +``` + +If zero or multiple automations match, the API raises `ValueError`. + +## Create an automation + +Create an automation by defining an **event** (what triggers it) and an **action** (what runs when it triggers), then calling `api.create_automation()`. Events and actions are built from classes in `wandb.automations`; the scope (project or registry/collection) determines which event types you can use. + +**Example: project automation (run state to Slack)** + +This example creates a project-scoped automation that sends a Slack notification when a run in the project finishes in a **Failed** state. + +```python +import wandb +from wandb.automations import OnRunState, RunEvent, SendNotification + +api = wandb.Api() + +project = api.project("my-project", entity="my-team") +slack_integration = next(api.slack_integrations(entity="my-team")) + +event = OnRunState( + scope=project, + filter=RunEvent.state.in_(["failed"]), +) +action = SendNotification.from_integration(slack_integration) + +automation = api.create_automation( + event >> action, + name="run-failure-alert", + description="Notify the team when a run fails.", +) +``` + +**Example: run metric threshold to Slack (project)** + +The [Api reference](/models/ref/python/public-api/api/#api-create_automation) shows creating an automation that triggers when a run metric exceeds a threshold. You can combine other event types (for example, [OnAddArtifactAlias](/models/ref/python/automations/onaddartifactalias) or [OnCreateArtifact](/models/ref/python/automations/oncreateartifact)) with [SendNotification](/models/ref/python/automations/sendnotification) or [SendWebhook](/models/ref/python/automations/sendwebhook) in the same way: build the event and action, then pass `event >> action` to `api.create_automation()`. + +## Update an automation + +Fetch the automation, change its attributes, then call `api.update_automation()`. You can pass keyword arguments to update in place without mutating the object. + +```python +import wandb + +api = wandb.Api() + +automation = api.automation(name="my-automation") +automation.enabled = False +automation.description = "Temporarily disabled for maintenance." +api.update_automation(automation) +``` + +Or pass updates as keyword arguments: + +```python +updated = api.update_automation( + automation, + enabled=False, + description="Temporarily disabled for maintenance.", +) +``` + +## Delete an automation + +Pass the automation object or its ID to `api.delete_automation()`. + +```python +import wandb + +api = wandb.Api() + +automation = api.automation(name="my-automation") +api.delete_automation(automation) +``` + +Or delete by ID: + +```python +api.delete_automation(automation.id) +``` + +{/* TODO: Fix when https://github.com/wandb/examples/pull/618 merges */} +For a runnable walkthrough that includes these operations and the two tutorial automations (project run failure alert and registry alias to webhook), see the [Automations tutorial notebook](https://raw.githubusercontent.com/mdlinville/examples/a43b31213c8e0642a30a202f82e174772eb687f6/colabs/automations/automations-tutorial.ipynb) on GitHub. + +## Next steps + +- [Automations overview](/models/automations) +- [Automation events and scopes](/models/automations/automation-events) (project vs registry events) +- [Create a Slack automation](/models/automations/create-automations/slack) or [Create a webhook automation](/models/automations/create-automations/webhook) (UI) +- [Automations API reference](/models/ref/python/public-api/automations) diff --git a/models/automations/automation-events.mdx b/models/automations/automation-events.mdx index c6f701c229..25a4e4a50f 100644 --- a/models/automations/automation-events.mdx +++ b/models/automations/automation-events.mdx @@ -130,7 +130,6 @@ For example, trigger an automation when `accuracy` has a z-score above 2, which - A z-score of -2.0 means the metric is 2 standard deviations below average. - Values beyond ±2 are often considered statistically significant outliers. - #### Run filters This section describes how the automation selects runs to evaluate. diff --git a/models/automations/create-automations.mdx b/models/automations/create-automations.mdx index 5ab6e28c93..3be172fbb5 100644 --- a/models/automations/create-automations.mdx +++ b/models/automations/create-automations.mdx @@ -3,24 +3,24 @@ title: Overview description: Create and manage W&B automations to streamline your ML workflows --- -This page gives an overview of creating and managing W&B [automations](/models/automations/). For more detailed instructions, refer to [Create a Slack automation](/models/automations/create-automations/slack/) or [Create a webhook automation](/models/automations/create-automations/webhook/). +This page gives an overview of creating and managing W&B [automations](/models/automations). For more detailed instructions, refer to [Create a Slack automation](/models/automations/create-automations/slack) or [Create a webhook automation](/models/automations/create-automations/webhook). Looking for companion tutorials for automations? -- [Learn to automatically triggers a Github Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw). +- [Learn to automatically trigger a Github Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw). - [Watch a video demonstrating automatically deploying a model to a Sagemaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ). - [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6&feature=shared). ## Requirements -- A team admin can create and manage automations for the team's projects, as well as components of their automations, such as webhooks, secrets, and Slack integrations. Refer to [Team settings](/platform/app/settings-page/teams/). +- A team admin can create and manage automations for the team's projects, as well as components of their automations, such as webhooks, secrets, and Slack integrations. Refer to [Team settings](/platform/app/settings-page/teams). - To create a registry automation, you must have access to the registry. Refer to [Configure Registry access](/models/registry/configure_registry/#registry-roles). - To create a Slack automation, you must have permission to post to the Slack instance and channel you select. ## Create an automation Create an automation from the project or registry's **Automations** tab. At a high level, to create an automation, follow these steps: -1. If necessary, [create a W&B secret](/platform/secrets/) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations. +1. If necessary, [create a W&B secret](/platform/secrets) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations. 1. Configure team-level webhook or Slack integrations to authorize W&B to post to Slack or run the webhook on your behalf. A single webhook or Slack integration can be used by multiple automations. These actions are defined in your **Team Settings**. 1. In the project or registry, create the automation, which specifies the event to watch for and the action to take (such as posting to Slack or running a webhook). When you create a webhook automation, you configure the payload it sends. @@ -35,8 +35,8 @@ Or, from a line plot in the workspace, you can quickly create a [run metric auto For details, refer to: -- [Create a Slack automation](/models/automations/create-automations/slack/) -- [Create a webhook automation](/models/automations/create-automations/webhook/) +- [Create a Slack automation](/models/automations/create-automations/slack) +- [Create a webhook automation](/models/automations/create-automations/webhook) ## View and manage automations View and manage automations from a project or registry's **Automations** tab. @@ -47,7 +47,9 @@ View and manage automations from a project or registry's **Automations** tab. - To delete an automation, click its action `...` menu, then click **Delete automation**. ## Next steps -- Learn more about [automation events and scopes](/models/automations/automation-events/) -- [Create a Slack automation](/models/automations/create-automations/slack/). -- [Create a webhook automation](/models/automations/create-automations/webhook/). -- [Create a secret](/platform/secrets/). +- [Automations tutorials](/models/automations/tutorial): Build a [project run failure alert](/models/automations/project-automation-tutorial) or a [registry alias-to-webhook automation](/models/automations/registry-automation-tutorial) (UI or API). +- [Manage automations with the API](/models/automations/api). +- [Automation events and scopes](/models/automations/automation-events). +- [Create a Slack automation](/models/automations/create-automations/slack). +- [Create a webhook automation](/models/automations/create-automations/webhook). +- [Create a secret](/platform/secrets). diff --git a/models/automations/create-automations/slack.mdx b/models/automations/create-automations/slack.mdx index 2eeaaccf57..aa3428abae 100644 --- a/models/automations/create-automations/slack.mdx +++ b/models/automations/create-automations/slack.mdx @@ -3,11 +3,11 @@ title: Create a Slack automation description: "Set up a Slack integration and create a W&B Automation that sends notifications to a Slack channel on specific events." --- -This page shows how to create a Slack [automation](/models/automations/). To create a webhook automation, refer to [Create a webhook automation](/models/automations/create-automations/webhook/) instead. +This page shows how to create a Slack [automation](/models/automations). To create a webhook automation, refer to [Create a webhook automation](/models/automations/create-automations/webhook) instead. At a high level, to create a Slack automation, you take these steps: 1. [Add a Slack integration](#add-a-slack-integration), which authorizes W&B to post to the Slack instance and channel. -1. [Create the automation](#create-an-automation), which defines the [event](/models/automations/automation-events/) to watch for and the channel to notify. +1. [Create the automation](#create-an-automation), which defines the [event](/models/automations/automation-events) to watch for and the channel to notify. ## Add a Slack integration A team admin can add a Slack integration to the team. @@ -29,7 +29,7 @@ A team admin can view and manage the team's Slack instances and channels. 1. Delete a destination by clicking its trash icon. ## Create an automation -After you [add a Slack integration](#add-a-slack-integreation), select **Registry** or **Project**, then follow these steps to create an automation that notifies the Slack channel. +After you [add a Slack integration](#add-a-slack-integration), select **Registry** or **Project**, then follow these steps to create an automation that notifies the Slack channel. diff --git a/models/automations/create-automations/webhook.mdx b/models/automations/create-automations/webhook.mdx index de59ec3eec..25e49c2355 100644 --- a/models/automations/create-automations/webhook.mdx +++ b/models/automations/create-automations/webhook.mdx @@ -3,12 +3,12 @@ title: Create a webhook automation description: "Create a webhook automation in W&B to send HTTP requests to external services when specific events occur." --- -This page shows how to create a webhook [automation](/models/automations/). To create a Slack automation, refer to [Create a Slack automation](/models/automations/create-automations/slack/) instead. +This page shows how to create a webhook [automation](/models/automations). To create a Slack automation, refer to [Create a Slack automation](/models/automations/create-automations/slack) instead. At a high level, to create a webhook automation, you take these steps: -1. If necessary, [create a W&B secret](/platform/secrets/) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **Team Settings**. +1. If necessary, [create a W&B secret](/platform/secrets) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **Team Settings**. 1. [Create a webhook](#create-a-webhook) to define the endpoint and authorization details and grant the integration access to any secrets it needs. -1. [Create the automation](#create-an-automation) to define the [event](/models/automations/automation-events/) to watch for and the payload W&B will send. Grant the automation access to any secrets it needs for the payload. +1. [Create the automation](#create-an-automation) to define the [event](/models/automations/automation-events) to watch for and the payload W&B will send. Grant the automation access to any secrets it needs for the payload. ## Create a webhook A team admin can add a webhook for the team. @@ -21,7 +21,7 @@ If the webhook requires a Bearer token or its payload requires a sensitive strin 1. In the **Webhooks** section, click **New webhook**. 1. Provide a name for the webhook. 1. Provide the endpoint URL for the webhook. -1. If the webhook requires a Bearer token, set **Access token** to the [secret](/platform/secrets/) that contains it. When using the webhook automation, W&B sets the `Authorization: Bearer` HTTP header to the access token, and you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). Learn more about the structure of the `POST` request W&B sends to the webhook service in [Troubleshoot your webhook](#troubleshoot-your-webhook). +1. If the webhook requires a Bearer token, set **Access token** to the [secret](/platform/secrets) that contains it. When using the webhook automation, W&B sets the `Authorization: Bearer` HTTP header to the access token, and you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). Learn more about the structure of the `POST` request W&B sends to the webhook service in [Troubleshoot your webhook](#troubleshoot-your-webhook). 1. If the webhook requires a password or other sensitive string in its payload, set **Secret** to the secret that contains it. When you configure the automation that uses the webhook, you can access the secret as a [payload variable](#payload-variables) by prefixing its name with `$`. If the webhook's access token is stored in a secret, you must _also_ complete the next step to specify the secret as the access token. @@ -54,7 +54,7 @@ A Registry admin can create automations in that registry. Registry automations a Click **Next step**. 1. Select the team that owns the [webhook](#create-a-webhook). -1. Set **Action type** to **Webhooks**. then select the [webhook](#create-a-webhook) to use. +1. Set **Action type** to **Webhooks**. Then select the [webhook](#create-a-webhook) to use. 1. If you configured an access token for the webhook, you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). If you configured a secret for the webhook, you can access it in the payload by prefixing its name with `$`. Your webhook's requirements are determined by the webhook's service. 1. Click **Next step**. 1. Provide a name for the automation. Optionally, provide a description. Click **Create automation**. @@ -81,7 +81,7 @@ A W&B admin can create automations in a project. The automation is applied to all collections in the project, including those added in the future. 1. Click **Next step**. 1. Select the team that owns the [webhook](#create-a-webhook). -1. Set **Action type** to **Webhooks**. then select the [webhook](#create-a-webhook) to use. +1. Set **Action type** to **Webhooks**. Then select the [webhook](#create-a-webhook) to use. 1. If your webhook requires a payload, construct it and paste it into the **Payload** field. If you configured an access token for the webhook, you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). If you configured a secret for the webhook, you can access it in the payload by prefixing its name with `$`. Your webhook's requirements are determined by the webhook's service. 1. Click **Next step**. 1. Provide a name for the automation. Optionally, provide a description. Click **Create automation**. @@ -122,7 +122,7 @@ This section describes the variables you can use to construct your webhook's pay | `${tag}` | Contains an artifact's tags if the automation is triggered by the **An artifact tag is added** event. For other automations, this variable is blank. | | `${artifact_collection_name}` | The name of the artifact collection that the artifact version is linked to. | | `${artifact_metadata.}` | The value of an arbitrary top-level metadata key from the artifact version that triggered the action. Replace `` with the name of a top-level metadata key. Only top-level metadata keys are available in the webhook's payload. | -| `${artifact_version}` | The [`Wandb.Artifact`](/models/ref/python/experiments/artifact.md/) representation of the artifact version that triggered the action. | +| `${artifact_version}` | The [`Wandb.Artifact`](/models/ref/python/experiments/artifact) representation of the artifact version that triggered the action. | | `${artifact_version_string}` | The `string` representation of the artifact version that triggered the action. | | `${ACCESS_TOKEN}` | The value of the access token configured in the [webhook](#create-a-webhook), if an access token is configured. The access token is automatically passed in the `Authorization: Bearer` HTTP header. | | `${SECRET_NAME}` | If configured, the value of a secret configured in the [webhook](#create-a-webhook). Replace `SECRET_NAME` with the name of the secret. | @@ -177,7 +177,7 @@ ${project_name} --> "model-registry" ${entity_name} --> "" ``` -Use template strings to dynamically pass context from W&B to GitHub Actions and other tools. If those tools can call Python scripts, they can consume the registered model artifacts through the [W&B API](/models/artifacts/download-and-use-an-artifact/). +Use template strings to dynamically pass context from W&B to GitHub Actions and other tools. If those tools can call Python scripts, they can consume the registered model artifacts through the [W&B API](/models/artifacts/download-and-use-an-artifact). - For more information about repository dispatch, see the [official documentation on the GitHub Marketplace](https://github.com/marketplace/actions/repository-dispatch). @@ -292,7 +292,6 @@ Copy and paste the code below into a shell script to troubleshoot your webhook. * `PAYLOAD` * `API_ENDPOINT` - ```bash webhook_test.sh #!/bin/bash diff --git a/models/automations/project-automation-tutorial.mdx b/models/automations/project-automation-tutorial.mdx new file mode 100644 index 0000000000..517aba8cdf --- /dev/null +++ b/models/automations/project-automation-tutorial.mdx @@ -0,0 +1,75 @@ +--- +title: "Tutorial: Project run-failure alert automation" +description: Build a run-failure alert that sends a Slack notification when a run in your project fails. +--- +import TutorialDiagramProject from "/snippets/en/_includes/automations/tutorial-diagram-project.mdx"; +import TutorialNotebook from "/snippets/en/_includes/automations/tutorial-notebook.mdx"; +import TutorialGoFurther from "/snippets/en/_includes/automations/tutorial-go-further.mdx"; + +This tutorial walks you through building a **project** automation triggered by run status: when a run in your project transitions to **Failed**, W&B sends a Slack notification. + + + +For guidance creating a Registry automation, see [Tutorial: Registry artifact alias automation](/models/automations/registry-automation-tutorial). + +## Prerequisites + +- A W&B project. +- A [Slack integration](/models/automations/create-automations/slack#add-a-slack-integration) configured in **Team Settings**. + +## Create a project automation + +Follow these instructions to set up a project-scoped automation: When a run in the project transitions to **Failed**, W&B sends a Slack notification. + + + +1. Open the project and click the **Automations** tab in the sidebar, then click **Create automation**. +1. Choose the event **Run state change**. Set the state to **Failed**. Optionally add a run name or user filter to limit which runs trigger the automation. +1. Click **Next step**. Set **Action type** to **Slack notification** and select the Slack channel. +1. Click **Next step**. Give the automation a name (for example, "Run failure alert") and an optional description, then click **Create automation**. + +For more detail, see [Create a Slack automation](/models/automations/create-automations/slack#create-an-automation) (Project tab). + + +Use the public API to create the same automation with a project scope and run-state filter: + +```python +import wandb +from wandb.automations import OnRunState, RunEvent, SendNotification + +api = wandb.Api() + +project = api.project("my-project", entity="my-team") +slack_integration = next(api.slack_integrations(entity="my-team")) + +event = OnRunState( + scope=project, + filter=RunEvent.state.in_(["failed"]), +) +action = SendNotification.from_integration(slack_integration) + +automation = api.create_automation( + event >> action, + name="run-failure-alert", + description="Notify the team when a run fails.", +) +``` + + + +## Test the automation +Create a run and log it to the project, explicitly marking it as failed: + +```python +import wandb + +with wandb.init(project="my-project") as run: + run.log({"loss": 1.23}) + run.finish(exit_code=1) +``` + +Within a short time you should see a Slack message with the run link and status. + + + + diff --git a/models/automations/registry-automation-tutorial.mdx b/models/automations/registry-automation-tutorial.mdx new file mode 100644 index 0000000000..2741d3d67b --- /dev/null +++ b/models/automations/registry-automation-tutorial.mdx @@ -0,0 +1,95 @@ +--- +title: "Tutorial: Registry artifact alias automation" +description: Build an automation that runs a webhook when a Registry artifact gets a specific alias like "production". +--- +import TutorialDiagramRegistry from "/snippets/en/_includes/automations/tutorial-diagram-registry.mdx"; +import TutorialNotebook from "/snippets/en/_includes/automations/tutorial-notebook.mdx"; +import TutorialGoFurther from "/snippets/en/_includes/automations/tutorial-go-further.mdx"; + +This tutorial walks you through building a **registry** automation triggered by artifact metadata: when an artifact in your registry gets a specific alias (for example, **production**), W&B sends a POST request to your webhook. + + + +For guidance creating a project automation, see [Tutorial: Project run-failure alert automation](/models/automations/project-automation-tutorial). + +## Prerequisites + +- A [webhook](/models/automations/create-automations/webhook#create-a-webhook) configured in **Team Settings**. +- A W&B [registry](/models/registry/create_registry) with at least one collection, or reuse an existing registry. + +## Create a registry automation + +Follow these instructions to set up a registry-scoped automation: When an artifact in any collection in that registry gets a specific alias (for example, **production**), W&B sends a POST request to your webhook. + + + +1. Open the registry and click the **Automations** tab, then click **Create automation**. +1. Choose the event **An artifact alias is added**. Enter an **Alias regex** that matches the alias you care about (for example, **production** or **staging**). +1. Click **Next step**. Set **Action type** to **Webhooks** and select your webhook. If the webhook expects a payload, paste a JSON body and use [payload variables](/models/automations/create-automations/webhook#payload-variables) such as `${artifact_collection_name}` and `${artifact_version_string}`. +1. Click **Next step**. Give the automation a name and optional description, then click **Create automation**. + +For more detail, see [Create a webhook automation](/models/automations/create-automations/webhook#create-an-automation) (Registry tab). + + +Use the public API to create the same automation with a collection scope and alias filter: + +```python +import wandb +from wandb.automations import OnAddArtifactAlias, ArtifactEvent, SendWebhook + +api = wandb.Api() + +collection = api.artifact_collection(name="my-model", type_name="model") +webhook_integration = next(api.webhook_integrations(entity="my-team")) + +event = OnAddArtifactAlias( + scope=collection, + filter=ArtifactEvent.alias.eq("production"), +) +action = SendWebhook.from_integration( + webhook_integration, + payload={ + "event": "${event_type}", + "model": "${artifact_collection_name}", + "version": "${artifact_version_string}", + }, +) + +automation = api.create_automation( + event >> action, + name="production-alias-webhook", + description="Trigger webhook when production alias is added.", +) +``` + + + +## Test the automation + +Add the alias (for example, **production**) to an artifact version in the registry, using the W&B App or the public API. For example: + +```python +import wandb + +with wandb.init(project="my-project") as run: + artifact = wandb.Artifact("my-model", type="model") + # ... log files or metadata to artifact as needed ... + run.log_artifact(artifact) + run.wait() # Ensure the artifact is logged before proceeding + +# Add an alias to the latest version in the collection +api = wandb.Api() +collection = api.artifact_collection(name="my-model", type_name="model") +version = next(collection.versions()) # Get the latest version + +version.aliases.append("production") +version.save() +print("Added alias 'production' to", version.name) + +``` + +Within a short time your webhook endpoint should receive a POST with the payload you configured. + + + + diff --git a/models/automations/tutorial.mdx b/models/automations/tutorial.mdx new file mode 100644 index 0000000000..b477e126fd --- /dev/null +++ b/models/automations/tutorial.mdx @@ -0,0 +1,15 @@ +--- +title: Automation tutorial overview +sidebarTitle: Overview +description: Learn to build a project run-failure alert or a registry alias automation. +--- +import AutomationsMentalModel from "/snippets/en/_includes/automations/mental-model.mdx"; +import TutorialDiagramProject from "/snippets/en/_includes/automations/tutorial-diagram-project.mdx"; +import TutorialDiagramRegistry from "/snippets/en/_includes/automations/tutorial-diagram-registry.mdx"; + + + +Select a tutorial for detailed guidance. + +- **[Project automation tutorial](/models/automations/project-automation-tutorial)**: Alert when a run fails (Slack notification). +- **[Registry automation tutorial](/models/automations/registry-automation-tutorial)**: Trigger a webhook when an alias (for example, `production`) is added to an artifact. diff --git a/snippets/en/_includes/automations/actions-list.mdx b/snippets/en/_includes/automations/actions-list.mdx new file mode 100644 index 0000000000..341a0ca9a4 --- /dev/null +++ b/snippets/en/_includes/automations/actions-list.mdx @@ -0,0 +1,2 @@ +- **Slack notification**: Send a message to a Slack channel with details about the triggering event. The message summarizes the event, with a link to view more details in W&B. +- **Webhook**: Call a webhook URL with a JSON payload containing information about the triggering event. This enables integration with external systems like CI/CD pipelines, model deployment services, or custom workflows. diff --git a/snippets/en/_includes/automations/best-practices.mdx b/snippets/en/_includes/automations/best-practices.mdx new file mode 100644 index 0000000000..51096dbfce --- /dev/null +++ b/snippets/en/_includes/automations/best-practices.mdx @@ -0,0 +1,3 @@ +- **Start small**: Begin with one or two automations for high-value events (for example, run failures or production alias changes). Validate that they work as expected before adding more. +- **Test before production**: Create automations in a test project or with a test webhook or Slack channel first. Manually trigger the event and confirm the action runs and the payload or message looks correct. +- **Avoid alert fatigue**: Use run filters, metric thresholds, or alias patterns to limit how often an automation fires. Route different severities to different channels if needed. diff --git a/snippets/en/_includes/automations/mental-model.mdx b/snippets/en/_includes/automations/mental-model.mdx new file mode 100644 index 0000000000..a2f747639d --- /dev/null +++ b/snippets/en/_includes/automations/mental-model.mdx @@ -0,0 +1,7 @@ +W&B Automations follow a simple pattern: when an **event** occurs and optional **conditions** are met, an **action** runs automatically. For example: +- When a run fails (event), notify a Slack channel (action). + +- When the `production` alias is added to an artifact (event), call a webhook to trigger deployment (action). + + +Events and available conditions differ for automations scoped to a [project](/models/automations/automation-events#project) or a [registry](/models/automations/automation-events#registry). See [Automation events and scopes](/models/automations/automation-events). diff --git a/snippets/en/_includes/automations/tutorial-diagram-project.mdx b/snippets/en/_includes/automations/tutorial-diagram-project.mdx new file mode 100644 index 0000000000..1e3d0cc1c4 --- /dev/null +++ b/snippets/en/_includes/automations/tutorial-diagram-project.mdx @@ -0,0 +1,9 @@ +

+```mermaid +%%{init: {'flowchart': {'rankSpacing': 200;}}}%% +flowchart LR + Event[Run state change to Failed] + Action[Slack notification] + Event --> Action +``` +
diff --git a/snippets/en/_includes/automations/tutorial-diagram-registry.mdx b/snippets/en/_includes/automations/tutorial-diagram-registry.mdx new file mode 100644 index 0000000000..eaf18b5ff8 --- /dev/null +++ b/snippets/en/_includes/automations/tutorial-diagram-registry.mdx @@ -0,0 +1,8 @@ +

+```mermaid +flowchart LR + Event[Artifact alias added] + Action[Webhook] + Event --> Action +``` +
diff --git a/snippets/en/_includes/automations/tutorial-go-further.mdx b/snippets/en/_includes/automations/tutorial-go-further.mdx new file mode 100644 index 0000000000..b35b6d63f3 --- /dev/null +++ b/snippets/en/_includes/automations/tutorial-go-further.mdx @@ -0,0 +1,6 @@ +## Go further + +- [Automation events and scopes](/models/automations/automation-events) for all project and registry event types. +- [Create a Slack automation](/models/automations/create-automations/slack) and [Create a webhook automation](/models/automations/create-automations/webhook) for full UI and payload details. +- [Manage automations with the API](/models/automations/api) for list, get, update, and delete examples. +- [Automations API reference](/models/ref/python/public-api/automations) for all event and action classes. diff --git a/snippets/en/_includes/automations/tutorial-notebook.mdx b/snippets/en/_includes/automations/tutorial-notebook.mdx new file mode 100644 index 0000000000..2e9c797e56 --- /dev/null +++ b/snippets/en/_includes/automations/tutorial-notebook.mdx @@ -0,0 +1,5 @@ +## Run the full tutorial in a notebook + +Run the [Automations tutorial notebook](https://raw.githubusercontent.com/mdlinville/examples/a43b31213c8e0642a30a202f82e174772eb687f6/colabs/automations/automations-tutorial.ipynb) to create and test project-scoped and registry-scoped automations. + +Download or clone the repo and open the notebook in Jupyter, Google Colab, or your preferred notebook environment. diff --git a/snippets/en/_includes/automations/where-to-find-automations.mdx b/snippets/en/_includes/automations/where-to-find-automations.mdx new file mode 100644 index 0000000000..e8bb61f9d6 --- /dev/null +++ b/snippets/en/_includes/automations/where-to-find-automations.mdx @@ -0,0 +1,2 @@ +- **In a project**: Open the project, then click the **Automations** tab in the project sidebar. +- **In a registry**: Open the registry, then click the **Automations** tab.