Allow per-cluster max_runners overrides for arc-runners-X scale sets#630
Open
huydhn wants to merge 1 commit into
Open
Allow per-cluster max_runners overrides for arc-runners-X scale sets#630huydhn wants to merge 1 commit into
huydhn wants to merge 1 commit into
Conversation
huydhn
force-pushed
the
cleanup-h100-pet-instances-uw1
branch
from
019d838 to
46f1210
Compare
huydhn
changed the title
huydhn
force-pushed
the
cleanup-h100-pet-instances-uw1
branch
from
46f1210 to
81c3e0d
Compare
huydhn
changed the title
jeanschmidt
reviewed
May 26, 2026
| # arc-runners-h100/defs/*.yaml are sized for 1 reserved 8-GPU node, but | ||
| # this cluster holds 6 (see nodepools-h100.capacity_reservation_ids above). | ||
| # Override each scale set's max_runners to def×6. | ||
| max_runners: |
Contributor
There was a problem hiding this comment.
IMO max runners should be defined in the nodepool,
Can't we do this in the nodepool and duplicate them? Maybe it makes more sense
Contributor
Author
There was a problem hiding this comment.
Let me try that, and see how it looks like. I think I know what you mean
huydhn
force-pushed
the
cleanup-h100-pet-instances-uw1
branch
from
81c3e0d to
94b46e1
Compare
huydhn
force-pushed
the
cleanup-h100-pet-instances-uw1
branch
2 times, most recently
from
03172e6 to
c92ac31
Compare
max_runners now accepts either an int (baseline applied to every cluster,
unchanged for the ~30 existing CPU/GPU defs and the B200 defs) or a
mapping with a required `default` baseline plus per-cluster overrides:
max_runners:
default: 8
arc-cbr-production-uw1: 48
Each arc-runners-h100 def adopts the mapping form. arc-cbr-production-uw1
inherits the 6 H100 nodes freed by pytorch-gha-infra#1180, so each scale
set is sized to def×6 (48 / 24 / 12 / 6). us-east-2 keeps the def baselines.
Capacity info stays co-located with the runner shape rather than spread
between clusters.yaml and the def — addresses review feedback on the
earlier clusters.yaml-based approach.
huydhn
force-pushed
the
cleanup-h100-pet-instances-uw1
branch
from
c92ac31 to
45c2b6e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
arc-runners-X/defs/*.yamlnow own per-cluster capacity directly.max_runnersaccepts either an int (baseline applied to every cluster — backwards-compatible with the ~30 existing CPU/GPU defs and the B200 defs) or a mapping:Each
arc-runners-h100def adopts the mapping form.arc-cbr-production-uw1inherits the 6 H100 nodes freed by pytorch-gha-infra#1180, so its scale sets render at def×6 (48 / 24 / 12 / 6). us-east-2 keeps the def baselines (unchanged).Capacity stays co-located with the runner shape rather than split between
clusters.yamland the def — addresses @jeanschmidt's review feedback on the earlierclusters.yaml-based approach.Test plan
pytest modules/arc-runners/scripts/python/test_generate_runners.py— 5 new tests covering the mapping form, plus all existing tests green.maxRunners: 48 / 24 / 12 / 6 (uw1) and 8 / 4 / 2 / 1 (us-east-2 baseline).arc-runners-h100toarc-cbr-production-uw1; confirmmaxRunners= 48 / 24 / 12 / 6.arc-runners-h100/-b200toarc-cbr-production; confirmmaxRunnersis unchanged.