Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,8 @@ with an external configuration file, like TOML, decoupling connection settings f

- [**Eager Workflow Start**](./eager-workflow-start): Demonstrates how to start a workflow in eager mode, an experimental latency optimization.

- [**Task Queue Priority and Fairness**](./task-queue-priority-fairness): Demonstrates Activity Task dispatch using `PriorityKey`, `FairnessKey`, and `FairnessWeight` with a multi-tenant backlog.

### Dynamic Workflow logic examples

These samples demonstrate some common control flow patterns using Temporal's Go SDK API.
Expand Down
172 changes: 172 additions & 0 deletions task-queue-priority-fairness/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
# Task Queue Priority and Fairness

This sample demonstrates Temporal Task Queue Priority and Fairness for Activity Tasks using a multi-tenant media rendering workload.

A SaaS rendering platform receives urgent previews, normal renders, and background archive jobs from multiple tenants into one shared Activity Task Queue. The sample intentionally creates backlog so dispatch order is visible:

- `PriorityKey` chooses which priority level is dispatched first. Lower values run first, so urgent preview jobs use `PriorityKey=1`, normal render jobs use `PriorityKey=3`, and background archive jobs use `PriorityKey=5`.
- `FairnessKey` groups work by tenant within a priority level. Every Activity Task in this sample uses its tenant ID as a non-empty fairness key.
- `FairnessWeight` gives `premium-media` a larger proportional share while it has backlog. The premium tenant uses `FairnessWeight=3.0`; regular tenants use `FairnessWeight=1.0`.

Priority determines which priority sub-queue Tasks go into. Fairness determines ordering within a given priority level. Fairness ordering is probabilistic and observational, so the exact order can vary between runs.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I need to change up this wording as it is misleading

Suggested change
Priority determines which priority sub-queue Tasks go into. Fairness determines ordering within a given priority level. Fairness ordering is probabilistic and observational, so the exact order can vary between runs.
Priority determines which priority sub-queue Tasks go into. Within a priority level, Fairness selects among fairness-key virtual queues using weighted, probabilistic dispatch. Tasks with the same fairness key are still dispatched FIFO, but the interleaving between different fairness keys is not a deterministic row-by-row sequence and can vary between runs.


## Workload definition

This sample uses a fixed workload on one shared Activity Task Queue so readers
can predict expected behavior before running it.

Tenants and fairness weights:

- `premium-media`: `FairnessWeight=3.0`
- `large-studio`: `FairnessWeight=1.0`
- `small-studio-a`: `FairnessWeight=1.0`
- `small-studio-b`: `FairnessWeight=1.0`

Job mix:

- Urgent preview (`PriorityKey=1`)
- `premium-media`: 2 jobs
- `small-studio-a`: 2 jobs
- Normal render (`PriorityKey=3`)
- `large-studio`: 18 jobs
- `small-studio-a`: 3 jobs
- `small-studio-b`: 3 jobs
- `premium-media`: 9 jobs
- Background archive (`PriorityKey=5`)
- `large-studio`: 4 jobs

Total jobs: `41` (`4 urgent + 33 normal + 4 background`)

`BuildJobs()` scheduling order:

1. `large-studio` normal jobs
2. `small-studio-a` normal jobs
3. `small-studio-b` normal jobs
4. `premium-media` normal jobs
5. `large-studio` background jobs
6. Urgent jobs last

Urgent jobs are intentionally enqueued last so the sample can show that
dispatching follows `PriorityKey` (with backlog), not submission order.

## Expected ordering at runtime

In the backlog sample flow (workflow worker first, then starter, then activity
worker), expected behavior is:

1. Priority across levels:
all urgent (`priority=1`) jobs should appear before queued normal
(`priority=3`) and background (`priority=5`) jobs.
2. Fairness within normal priority:
small-tenant fairness keys should appear before `large-studio` drains all of
its normal backlog.
3. Weighted fairness within normal priority:
`premium-media` should appear repeatedly near the front of normal-priority
dispatch while it still has backlog.

Fairness is probabilistic, so exact row-by-row ordering can vary between runs.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one as well.

Suggested change
Fairness is probabilistic, so exact row-by-row ordering can vary between runs.
Fairness uses weighted, probabilistic dispatch across fairness keys, so this sample validates expected weighted/proportional patterns rather than an exact row-by-row sequence.

The sample is designed to validate these patterns, not a deterministic sequence.

## Run the backlog sample

### 1. Start Temporal Server with Fairness enabled

The following dev server config is known to work well for this sample:

```bash
temporal server start-dev \
--dynamic-config-value matching.useNewMatcher=true \
--dynamic-config-value matching.enableFairness=true \
--dynamic-config-value matching.numTaskqueueReadPartitions=1 \
--dynamic-config-value matching.numTaskqueueWritePartitions=1
```

Why these options:

- `matching.useNewMatcher=true`: enables the matcher implementation that supports task queue priority/fairness behavior used by this sample.
- `matching.enableFairness=true`: turns on fairness-aware dispatch so `FairnessKey` and `FairnessWeight` affect ordering within a priority level.
- `matching.numTaskqueueReadPartitions=1` and `matching.numTaskqueueWritePartitions=1`: forces a single partition for this sample, which makes observed ordering easier to interpret and less noisy than multi-partition dispatch.

### 2. Start only the Workflow Worker

```bash
go run task-queue-priority-fairness/worker/main.go -mode workflow
```

### 3. Start the Workflow

In another terminal:

```bash
go run task-queue-priority-fairness/starter/main.go
```

The starter prints a reminder to start the Activity Worker. At this point, the Workflow has scheduled many Activity Tasks, but no Activity Worker is polling yet, so the Activity Task Queue has backlog.

### 4. Start the constrained Activity Worker

In another terminal:

```bash
go run task-queue-priority-fairness/worker/main.go -mode activity
```

The Activity Worker uses `MaxConcurrentActivityExecutionSize: 1`, making the dispatch/start order easy to observe in logs and in the starter output.

## What to look for

The starter prints a table of observed Activity start order and a summary. In the backlog-focused flow, urgent preview jobs submitted last should be dispatched before lower-priority queued work. Within normal render jobs, small tenants should appear before the large tenant drains its backlog, and `premium-media` should receive repeated dispatches while it remains backlogged.

## Example output

A successful run should look similar to this.
> [!NOTE]
> Timestamps, Run IDs, and Worker IDs will differ on your machine.

```text
2026/05/05 00:45:09 Started workflow. WorkflowID task-queue-priority-fairness-<example-id> RunID <example-run-id>
If you are running the full backlog sample, start the Activity Worker now:
go run task-queue-priority-fairness/worker/main.go -mode activity
Activity start order:

01 started_at=<ts> priority=1 fairness_key=premium-media weight=3.0 kind=urgent-preview job=premium-media-urgent-preview-00
02 started_at=<ts> priority=1 fairness_key=premium-media weight=3.0 kind=urgent-preview job=premium-media-urgent-preview-01
03 started_at=<ts> priority=1 fairness_key=small-studio-a weight=1.0 kind=urgent-preview job=small-studio-a-urgent-preview-01
04 started_at=<ts> priority=1 fairness_key=small-studio-a weight=1.0 kind=urgent-preview job=small-studio-a-urgent-preview-00
05 started_at=<ts> priority=3 fairness_key=premium-media weight=3.0 kind=normal-render job=premium-media-normal-render-06
06 started_at=<ts> priority=3 fairness_key=small-studio-b weight=1.0 kind=normal-render job=small-studio-b-normal-render-00
07 started_at=<ts> priority=3 fairness_key=small-studio-a weight=1.0 kind=normal-render job=small-studio-a-normal-render-02
...
37 started_at=<ts> priority=3 fairness_key=large-studio weight=1.0 kind=normal-render job=large-studio-normal-render-17
38 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-03
39 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-02
40 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-01
41 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-00

Summary:

Priority:
Urgent jobs use PriorityKey=1.
Normal jobs use PriorityKey=3.
Background jobs use PriorityKey=5.
Urgent work was dispatched before lower-priority queued work: OBSERVED

Fairness:
Tenant IDs are used as FairnessKey values.
Small tenants appeared before the large tenant drained all normal jobs: OBSERVED

Weighted fairness:
premium-media uses FairnessWeight=3.0.
Other tenants use FairnessWeight=1.0.
Premium work received repeated dispatches while backlogged: OBSERVED

Note:
Fairness ordering is probabilistic and can vary between runs.
```

How to read this output:

- Rows `01` to `04` are all urgent (`priority=1`) even though urgent jobs were scheduled last in `BuildJobs()`. This shows priority overtaking queued lower-priority work.
- Normal jobs (`priority=3`) include multiple fairness keys near the front (`premium-media`, `small-studio-a`, `small-studio-b`, `large-studio`) instead of draining one tenant first.
- `premium-media` appears more often in the early normal-priority dispatches while it has queued work, reflecting its `FairnessWeight=3.0` compared with `1.0` for the other tenants.
- Background jobs (`priority=5`) start only after normal-priority work in this run, matching the expected priority behavior.
35 changes: 35 additions & 0 deletions task-queue-priority-fairness/activity.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package task_queue_priority_fairness

import (
"context"
"time"

"go.temporal.io/sdk/activity"
)

func ProcessRenderJob(ctx context.Context, job RenderJob) (RenderResult, error) {
startedAt := time.Now().UTC()

logger := activity.GetLogger(ctx)
logger.Info(
"Started render job",
"started_at", startedAt,
"priority", job.PriorityKey,
"tenant", job.Tenant,
"weight", job.FairnessWeight,
"kind", job.Kind,
"job_id", job.JobID,
)

time.Sleep(150 * time.Millisecond)

return RenderResult{
StartedAt: startedAt,
JobID: job.JobID,
Tenant: job.Tenant,
Kind: job.Kind,
PriorityKey: job.PriorityKey,
FairnessKey: job.FairnessKey,
FairnessWeight: job.FairnessWeight,
}, nil
}
46 changes: 46 additions & 0 deletions task-queue-priority-fairness/starter/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package main

import (
"context"
"fmt"
"log"
"time"

"go.temporal.io/sdk/client"
"go.temporal.io/sdk/contrib/envconfig"

task_queue_priority_fairness "github.com/temporalio/samples-go/task-queue-priority-fairness"
)

func main() {
c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
if err != nil {
log.Fatalln("Unable to create client", err)
}
defer c.Close()

jobs := task_queue_priority_fairness.BuildJobs()
workflowID := fmt.Sprintf("task-queue-priority-fairness-%d", time.Now().UnixNano())
workflowOptions := client.StartWorkflowOptions{
ID: workflowID,
TaskQueue: task_queue_priority_fairness.WorkflowTaskQueue,
}

we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, task_queue_priority_fairness.RenderWorkflow, jobs)
if err != nil {
log.Fatalln("Unable to execute workflow", err)
}

log.Println("Started workflow.", "WorkflowID", we.GetID(), "RunID", we.GetRunID())
fmt.Println("If you are running the full backlog sample, start the Activity Worker now:")
fmt.Println("go run task-queue-priority-fairness/worker/main.go -mode activity")

var results []task_queue_priority_fairness.RenderResult
if err := we.Get(context.Background(), &results); err != nil {
log.Fatalln("Unable to get workflow result", err)
}

fmt.Println(task_queue_priority_fairness.FormatResults(results))
summary := task_queue_priority_fairness.SummarizeResults(results)
fmt.Print(task_queue_priority_fairness.FormatSummary(summary))
}
Loading
Loading