Need a way for actors to select workers

[thockin]

Consider a service which has a free-tier and a paid-tier. They might reasonably provision "good" machines for their paid-tier users and "cheap" machines for their free-tier users. When a free-tier user does something which causes an actor to be instantiated, we need to offer a way to designate that this actor must use only workers on the "cheap" machines.

This could be done as a k8s-style selector (the actor has a selector, the workers have labels like "tier=free"), but that's not the only way to do it, so before we lock it down, we should at least consider other approaches.

Perhaps it is a per-user concept rather than per-actor? Or per-namespaces (see #21)?

Is it something that the person who creates the actor is allowed to set? k8s pods having a node selector gives them a LOT of control over themselves, so much so that people impose policies to restrict those. If we are trying to keep this as simple as possible, we should seek to avoid things like webhook policy implementations. :)

Maybe it's better expressed on the worker: "this worker can handle actors from namespaces where tier=free".

We need to consider how the control plane will track and query for matches. K8s labels started simple (exactly equal) but added in, not-in, etc semantics. Ideally we can offload some of the matching to the optimized storage layer(s)?

[maxsmythe]

Are different classes of workers just entirely different worker pools?

[thockin]

I assumed that different pools could have different metadata, but within a single pool they would all be identical

[EItanya]

I agree this seems like exactly the use-case for different worker pools. I wonder if this is actually a limitation of actor templates mapping onto worker pools. Rather the ability to map onto multiple depending on some conditions. Does that track with what you're thinking?

[maxsmythe]

That was my thought, that this was less about the worker and more about the pool.

You could have a 1:1 mapping of actor:pool (like we have now), which can support multiple tiers via multiple actors. Problems there include redundant config and no native way of handling the upgrade use-case (or a downgrade).

Tying it to the user could make sense, but that would make upgrades all-or-nothing.

Is it only a binary (free/paid), or is it something like resource requests where different users can run actors on larger or smaller workers? Something akin to autoscaling. Do we expect an actor to be permanently bound to a worker pool once created?

What if we allowed a QoS signal for each resume?

That starts to get into set logic (workerpool allowed for ActorTemplate AND user)