-
Notifications
You must be signed in to change notification settings - Fork 238
Add Task Queue Priority and Fairness sample #466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sivagirish81
wants to merge
2
commits into
temporalio:main
Choose a base branch
from
sivagirish81:task_queue_priority_fairness
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,172 @@ | ||||||
| # Task Queue Priority and Fairness | ||||||
|
|
||||||
| This sample demonstrates Temporal Task Queue Priority and Fairness for Activity Tasks using a multi-tenant media rendering workload. | ||||||
|
|
||||||
| A SaaS rendering platform receives urgent previews, normal renders, and background archive jobs from multiple tenants into one shared Activity Task Queue. The sample intentionally creates backlog so dispatch order is visible: | ||||||
|
|
||||||
| - `PriorityKey` chooses which priority level is dispatched first. Lower values run first, so urgent preview jobs use `PriorityKey=1`, normal render jobs use `PriorityKey=3`, and background archive jobs use `PriorityKey=5`. | ||||||
| - `FairnessKey` groups work by tenant within a priority level. Every Activity Task in this sample uses its tenant ID as a non-empty fairness key. | ||||||
| - `FairnessWeight` gives `premium-media` a larger proportional share while it has backlog. The premium tenant uses `FairnessWeight=3.0`; regular tenants use `FairnessWeight=1.0`. | ||||||
|
|
||||||
| Priority determines which priority sub-queue Tasks go into. Fairness determines ordering within a given priority level. Fairness ordering is probabilistic and observational, so the exact order can vary between runs. | ||||||
|
|
||||||
| ## Workload definition | ||||||
|
|
||||||
| This sample uses a fixed workload on one shared Activity Task Queue so readers | ||||||
| can predict expected behavior before running it. | ||||||
|
|
||||||
| Tenants and fairness weights: | ||||||
|
|
||||||
| - `premium-media`: `FairnessWeight=3.0` | ||||||
| - `large-studio`: `FairnessWeight=1.0` | ||||||
| - `small-studio-a`: `FairnessWeight=1.0` | ||||||
| - `small-studio-b`: `FairnessWeight=1.0` | ||||||
|
|
||||||
| Job mix: | ||||||
|
|
||||||
| - Urgent preview (`PriorityKey=1`) | ||||||
| - `premium-media`: 2 jobs | ||||||
| - `small-studio-a`: 2 jobs | ||||||
| - Normal render (`PriorityKey=3`) | ||||||
| - `large-studio`: 18 jobs | ||||||
| - `small-studio-a`: 3 jobs | ||||||
| - `small-studio-b`: 3 jobs | ||||||
| - `premium-media`: 9 jobs | ||||||
| - Background archive (`PriorityKey=5`) | ||||||
| - `large-studio`: 4 jobs | ||||||
|
|
||||||
| Total jobs: `41` (`4 urgent + 33 normal + 4 background`) | ||||||
|
|
||||||
| `BuildJobs()` scheduling order: | ||||||
|
|
||||||
| 1. `large-studio` normal jobs | ||||||
| 2. `small-studio-a` normal jobs | ||||||
| 3. `small-studio-b` normal jobs | ||||||
| 4. `premium-media` normal jobs | ||||||
| 5. `large-studio` background jobs | ||||||
| 6. Urgent jobs last | ||||||
|
|
||||||
| Urgent jobs are intentionally enqueued last so the sample can show that | ||||||
| dispatching follows `PriorityKey` (with backlog), not submission order. | ||||||
|
|
||||||
| ## Expected ordering at runtime | ||||||
|
|
||||||
| In the backlog sample flow (workflow worker first, then starter, then activity | ||||||
| worker), expected behavior is: | ||||||
|
|
||||||
| 1. Priority across levels: | ||||||
| all urgent (`priority=1`) jobs should appear before queued normal | ||||||
| (`priority=3`) and background (`priority=5`) jobs. | ||||||
| 2. Fairness within normal priority: | ||||||
| small-tenant fairness keys should appear before `large-studio` drains all of | ||||||
| its normal backlog. | ||||||
| 3. Weighted fairness within normal priority: | ||||||
| `premium-media` should appear repeatedly near the front of normal-priority | ||||||
| dispatch while it still has backlog. | ||||||
|
|
||||||
| Fairness is probabilistic, so exact row-by-row ordering can vary between runs. | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This one as well.
Suggested change
|
||||||
| The sample is designed to validate these patterns, not a deterministic sequence. | ||||||
|
|
||||||
| ## Run the backlog sample | ||||||
|
|
||||||
| ### 1. Start Temporal Server with Fairness enabled | ||||||
|
|
||||||
| The following dev server config is known to work well for this sample: | ||||||
|
|
||||||
| ```bash | ||||||
| temporal server start-dev \ | ||||||
| --dynamic-config-value matching.useNewMatcher=true \ | ||||||
| --dynamic-config-value matching.enableFairness=true \ | ||||||
| --dynamic-config-value matching.numTaskqueueReadPartitions=1 \ | ||||||
| --dynamic-config-value matching.numTaskqueueWritePartitions=1 | ||||||
| ``` | ||||||
|
|
||||||
| Why these options: | ||||||
|
|
||||||
| - `matching.useNewMatcher=true`: enables the matcher implementation that supports task queue priority/fairness behavior used by this sample. | ||||||
| - `matching.enableFairness=true`: turns on fairness-aware dispatch so `FairnessKey` and `FairnessWeight` affect ordering within a priority level. | ||||||
| - `matching.numTaskqueueReadPartitions=1` and `matching.numTaskqueueWritePartitions=1`: forces a single partition for this sample, which makes observed ordering easier to interpret and less noisy than multi-partition dispatch. | ||||||
|
|
||||||
| ### 2. Start only the Workflow Worker | ||||||
|
|
||||||
| ```bash | ||||||
| go run task-queue-priority-fairness/worker/main.go -mode workflow | ||||||
| ``` | ||||||
|
|
||||||
| ### 3. Start the Workflow | ||||||
|
|
||||||
| In another terminal: | ||||||
|
|
||||||
| ```bash | ||||||
| go run task-queue-priority-fairness/starter/main.go | ||||||
| ``` | ||||||
|
|
||||||
| The starter prints a reminder to start the Activity Worker. At this point, the Workflow has scheduled many Activity Tasks, but no Activity Worker is polling yet, so the Activity Task Queue has backlog. | ||||||
|
|
||||||
| ### 4. Start the constrained Activity Worker | ||||||
|
|
||||||
| In another terminal: | ||||||
|
|
||||||
| ```bash | ||||||
| go run task-queue-priority-fairness/worker/main.go -mode activity | ||||||
| ``` | ||||||
|
|
||||||
| The Activity Worker uses `MaxConcurrentActivityExecutionSize: 1`, making the dispatch/start order easy to observe in logs and in the starter output. | ||||||
|
|
||||||
| ## What to look for | ||||||
|
|
||||||
| The starter prints a table of observed Activity start order and a summary. In the backlog-focused flow, urgent preview jobs submitted last should be dispatched before lower-priority queued work. Within normal render jobs, small tenants should appear before the large tenant drains its backlog, and `premium-media` should receive repeated dispatches while it remains backlogged. | ||||||
|
|
||||||
| ## Example output | ||||||
|
|
||||||
| A successful run should look similar to this. | ||||||
| > [!NOTE] | ||||||
| > Timestamps, Run IDs, and Worker IDs will differ on your machine. | ||||||
|
|
||||||
| ```text | ||||||
| 2026/05/05 00:45:09 Started workflow. WorkflowID task-queue-priority-fairness-<example-id> RunID <example-run-id> | ||||||
| If you are running the full backlog sample, start the Activity Worker now: | ||||||
| go run task-queue-priority-fairness/worker/main.go -mode activity | ||||||
| Activity start order: | ||||||
|
|
||||||
| 01 started_at=<ts> priority=1 fairness_key=premium-media weight=3.0 kind=urgent-preview job=premium-media-urgent-preview-00 | ||||||
| 02 started_at=<ts> priority=1 fairness_key=premium-media weight=3.0 kind=urgent-preview job=premium-media-urgent-preview-01 | ||||||
| 03 started_at=<ts> priority=1 fairness_key=small-studio-a weight=1.0 kind=urgent-preview job=small-studio-a-urgent-preview-01 | ||||||
| 04 started_at=<ts> priority=1 fairness_key=small-studio-a weight=1.0 kind=urgent-preview job=small-studio-a-urgent-preview-00 | ||||||
| 05 started_at=<ts> priority=3 fairness_key=premium-media weight=3.0 kind=normal-render job=premium-media-normal-render-06 | ||||||
| 06 started_at=<ts> priority=3 fairness_key=small-studio-b weight=1.0 kind=normal-render job=small-studio-b-normal-render-00 | ||||||
| 07 started_at=<ts> priority=3 fairness_key=small-studio-a weight=1.0 kind=normal-render job=small-studio-a-normal-render-02 | ||||||
| ... | ||||||
| 37 started_at=<ts> priority=3 fairness_key=large-studio weight=1.0 kind=normal-render job=large-studio-normal-render-17 | ||||||
| 38 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-03 | ||||||
| 39 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-02 | ||||||
| 40 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-01 | ||||||
| 41 started_at=<ts> priority=5 fairness_key=large-studio weight=1.0 kind=background-archive job=large-studio-background-archive-00 | ||||||
|
|
||||||
| Summary: | ||||||
|
|
||||||
| Priority: | ||||||
| Urgent jobs use PriorityKey=1. | ||||||
| Normal jobs use PriorityKey=3. | ||||||
| Background jobs use PriorityKey=5. | ||||||
| Urgent work was dispatched before lower-priority queued work: OBSERVED | ||||||
|
|
||||||
| Fairness: | ||||||
| Tenant IDs are used as FairnessKey values. | ||||||
| Small tenants appeared before the large tenant drained all normal jobs: OBSERVED | ||||||
|
|
||||||
| Weighted fairness: | ||||||
| premium-media uses FairnessWeight=3.0. | ||||||
| Other tenants use FairnessWeight=1.0. | ||||||
| Premium work received repeated dispatches while backlogged: OBSERVED | ||||||
|
|
||||||
| Note: | ||||||
| Fairness ordering is probabilistic and can vary between runs. | ||||||
| ``` | ||||||
|
|
||||||
| How to read this output: | ||||||
|
|
||||||
| - Rows `01` to `04` are all urgent (`priority=1`) even though urgent jobs were scheduled last in `BuildJobs()`. This shows priority overtaking queued lower-priority work. | ||||||
| - Normal jobs (`priority=3`) include multiple fairness keys near the front (`premium-media`, `small-studio-a`, `small-studio-b`, `large-studio`) instead of draining one tenant first. | ||||||
| - `premium-media` appears more often in the early normal-priority dispatches while it has queued work, reflecting its `FairnessWeight=3.0` compared with `1.0` for the other tenants. | ||||||
| - Background jobs (`priority=5`) start only after normal-priority work in this run, matching the expected priority behavior. | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| package task_queue_priority_fairness | ||
|
|
||
| import ( | ||
| "context" | ||
| "time" | ||
|
|
||
| "go.temporal.io/sdk/activity" | ||
| ) | ||
|
|
||
| func ProcessRenderJob(ctx context.Context, job RenderJob) (RenderResult, error) { | ||
| startedAt := time.Now().UTC() | ||
|
|
||
| logger := activity.GetLogger(ctx) | ||
| logger.Info( | ||
| "Started render job", | ||
| "started_at", startedAt, | ||
| "priority", job.PriorityKey, | ||
| "tenant", job.Tenant, | ||
| "weight", job.FairnessWeight, | ||
| "kind", job.Kind, | ||
| "job_id", job.JobID, | ||
| ) | ||
|
|
||
| time.Sleep(150 * time.Millisecond) | ||
|
|
||
| return RenderResult{ | ||
| StartedAt: startedAt, | ||
| JobID: job.JobID, | ||
| Tenant: job.Tenant, | ||
| Kind: job.Kind, | ||
| PriorityKey: job.PriorityKey, | ||
| FairnessKey: job.FairnessKey, | ||
| FairnessWeight: job.FairnessWeight, | ||
| }, nil | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "log" | ||
| "time" | ||
|
|
||
| "go.temporal.io/sdk/client" | ||
| "go.temporal.io/sdk/contrib/envconfig" | ||
|
|
||
| task_queue_priority_fairness "github.com/temporalio/samples-go/task-queue-priority-fairness" | ||
| ) | ||
|
|
||
| func main() { | ||
| c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) | ||
| if err != nil { | ||
| log.Fatalln("Unable to create client", err) | ||
| } | ||
| defer c.Close() | ||
|
|
||
| jobs := task_queue_priority_fairness.BuildJobs() | ||
| workflowID := fmt.Sprintf("task-queue-priority-fairness-%d", time.Now().UnixNano()) | ||
| workflowOptions := client.StartWorkflowOptions{ | ||
| ID: workflowID, | ||
| TaskQueue: task_queue_priority_fairness.WorkflowTaskQueue, | ||
| } | ||
|
|
||
| we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, task_queue_priority_fairness.RenderWorkflow, jobs) | ||
| if err != nil { | ||
| log.Fatalln("Unable to execute workflow", err) | ||
| } | ||
|
|
||
| log.Println("Started workflow.", "WorkflowID", we.GetID(), "RunID", we.GetRunID()) | ||
| fmt.Println("If you are running the full backlog sample, start the Activity Worker now:") | ||
| fmt.Println("go run task-queue-priority-fairness/worker/main.go -mode activity") | ||
|
|
||
| var results []task_queue_priority_fairness.RenderResult | ||
| if err := we.Get(context.Background(), &results); err != nil { | ||
| log.Fatalln("Unable to get workflow result", err) | ||
| } | ||
|
|
||
| fmt.Println(task_queue_priority_fairness.FormatResults(results)) | ||
| summary := task_queue_priority_fairness.SummarizeResults(results) | ||
| fmt.Print(task_queue_priority_fairness.FormatSummary(summary)) | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I need to change up this wording as it is misleading