Automation events and scopes

This feature requires a Pro or Enterprise plan.

An automation can start when a specific event occurs within a project or registry. Learn more about automations in the Automations overview or Create an automation.

Action	Registry scope	Project scope
Artifact version linked	✓
Artifact version aliased	✓	✓
Artifact version created		✓
Artifact version tagged		✓
Run status changed		✓
Run metric threshold met		✓

Artifact events

In a particular registry, an automation can start when an artifact version is linked or aliased. Automations in a given registry apply to all of the registry’s collections.

In a project, an automation can start when an artifact version is created, tagged, or aliased. Project automations are scoped to the team and apply to all of the project’s collections.

Regular expressions are supported for aliases and tags.

Run status change

Currently available only in W&B Multi-tenant Cloud.

In a project, an automation can start when a run changes its status to Running, Finished, or Failed. An automation cannot start when a run’s status is Killed. This status indicates that the run was stopped forcibly by an admin user.

Optionally, you can further limit the runs that can trigger an automation by specifying a user or run name filter.

Because run status is a property of the entire run, you can create a run status automation only from the Automations page.

Run metric events

Currently available only in W&B Multi-tenant Cloud.

In a project, an automation can start when a logged value for a metric meets a given threshold or changes by a given threshold. A system metric, such as cpu, can trigger an automation. W&B logs system metrics automatically every 15 seconds.

You can create a run metrics automation from the project’s Automations tab or directly from a line plot panel in a workspace.

To set up a run metric automation, you configure how to compare the metric’s value with the threshold you specify. Your choices depend on the event type and on any filters you specify.

Optionally, you can further limit the runs that can trigger an automation by specifying a user or run name filter.

Threshold

For Run metrics threshold met events, you configure:

1. The window of most recently logged values to consider (defaults to 5).
2. Whether to evaluate the Average, Min, or Max value within the window.
3. The comparison to make:
   • Above
   • Above or equal to
   • Below
   • Below or equal to
   • Not equal to
   • Equal to

For example, trigger an automation when average accuracy is above .6.

Change threshold

For Run metrics change threshold met events, the automation uses two “windows” of values to check whether to start:

• The current window of recently logged values to consider (defaults to 10).
• The prior window of recently logged values to consider (defaults to 50).

The current and prior windows are consecutive and do not overlap.

To create the automation, you configure:

1. The current window of logged values (defaults to 10).
2. The prior window of logged values (defaults to 50).
3. Whether to evaluate the values as relative or absolute (defaults to Relative).
4. The comparison to make:
   • Increases by at least
   • Decreases by at least
   • Increases or decreases by at least

For example, trigger an automation when average loss decreases by at least .25.

Run filters

This section describes how the automation selects runs to evaluate.

• By default, any run in the project triggers the automation when the event occurs. You can limit which runs trigger an automation by configuring one of the following filters:

  • Filter to one user’s runs: Include only runs created by the specified user.
  • Filter on run name: Include only runs whose names match the given regular expression.

  For details, see Create automations.

• Each run is considered individually and can potentially trigger the automation.

• Each run’s values are put into a separate window and compared to the threshold separately.

• In a 24 hour period, a particular automation can fire at most once per run.

Next steps

• Create a Slack automation
• Create a webhook automation