Support ConfigMap and Secret env sources in ActorTemplate containers

[EItanya]

Summary

ActorTemplate container env currently accepts Kubernetes corev1.EnvVar in the CRD schema, so users can write valueFrom.configMapKeyRef and valueFrom.secretKeyRef in manifests. At runtime, Substrate flattens env vars into its atelet workload protobuf, which historically carried only literal name/value pairs. That means valueFrom entries are accepted by the API but are not usable by actors.

Add support for resolving valueFrom.configMapKeyRef and valueFrom.secretKeyRef for ActorTemplate container env before handing the workload spec to atelet.

Why this matters

• Secret-backed env vars are the standard Kubernetes pattern for API keys and credentials. Without this, demos and real workloads either inline secrets as plain env values in rendered manifests or need custom workaround scripts.
• ConfigMap-backed env vars are the standard Kubernetes pattern for non-secret runtime configuration.
• The current behavior is surprising because the CRD schema exposes Kubernetes EnvVar, including valueFrom, but Substrate silently loses the source and sends an empty literal value to atelet.
• Supporting these two common valueFrom sources makes ActorTemplate behavior closer to what Kubernetes users expect while keeping unsupported sources explicit.

Proposed scope

• Support literal env.value as before.
• Support env.valueFrom.configMapKeyRef from the ActorTemplate namespace.
• Support env.valueFrom.secretKeyRef from the ActorTemplate namespace.
• Preserve Kubernetes optional: true behavior by skipping the env var if the referenced object or key is missing.
• Return a clear error for missing required objects/keys.
• Return a clear error for unsupported sources such as fieldRef, resourceFieldRef, and fileKeyRef.

Security / design note

Resolving Secret and ConfigMap references in ate-api means ate-api needs read access to those resources. The straightforward implementation grants the ate-api service account get access to ConfigMaps and Secrets so ActorTemplates work without custom per-namespace RBAC. This is a conscious tradeoff for now, but it should remain a security TODO/design choice.

Possible future designs include per-namespace delegated RBAC, admission-time validation/materialization, or another scoped credential model that avoids broad Secret read access in the control plane.

Acceptance criteria

• ActorTemplate env with valueFrom.secretKeyRef reaches the actor process with the referenced Secret value.
• ActorTemplate env with valueFrom.configMapKeyRef reaches the actor process with the referenced ConfigMap value.
• Missing required refs fail clearly instead of becoming empty env vars.
• Optional missing refs are skipped.
• Unsupported valueFrom sources fail clearly.
• Docs call out supported sources and the security/design tradeoff.

[thockin]

Thank you for citing the security concern. It's pretty significant, IMO

[EItanya]

Thank you for citing the security concern. It's pretty significant, IMO

I also addressed this in the PR I created. I think both approaches have potential downsides. Either we send these over the wire, or we allow all runners to read all secrets. This design could also have overlap with your namespace issue as you could force the runner to coexist with the secrets.

Has there been prior thinking done on this from your side, I definitely want to get this right, but I'll also say that this is an incredibly important thing to add ASAP

[thockin]

Taahir Ahmed (@ahmedtd) and Michael Taufen (@mtaufen) probably done the most thinking on this

[mtaufen]

The industry is broadly converging on a pattern of injecting credentials on the wire to prevent them from ending up in the agent context/being leaked by agents who have access to the env and filesystem. I'd like substrate to natively support this, or be easy to plug in the proxy of your choice.

We may still want some kind of secret support in the future for actors that run agent loops but aren't giving LLMs direct access to the local filesystem or env vars, for example to make it easier for those actors to connect to LLM APIs. But we need to be careful that our approach doesn't just encourage plumbing secrets to where agents also see them.

Finally, NOT supporting secrets is guaranteed to lead to ugly things like putting them in plaintext because "there was no other way to do it" (and agents especially will do this if they feel they have to get a job done), so +1 it's VERY important we address this gap.

[ahmedtd]

Embedding the Kubernetes EnvVar type directly was something I did in the rush for launch --- I will send a PR to replace it with an equivalent substrate-specific type that only has the fields for the things we actively support.

I agree, though, that we need this feature.

We can either have ate-api-server read the ConfigMap / Secret, or atelet. Ideally, if we have atelet read the ConfigMap, we can find some way to piggyback on the actor identity (similar to node isolation in Kubernetes). I can't immediately think of a way to make it work by building on the node restriction plugin / node authorizer because the ActorTemplate is not necessarily in the same namespace as the worker pod (if it were, then atelet could impersonate kubelet and get access to the ConfigMap / Secret via node authorizer).

Probably the design that makes the most sense for now is to make sure that ate-api-server has permission to read all relevant ConfigMaps and Secrets, and interpolate the values into the RPC call to atelet. ate-api-server is already highly privileged, and all of the RPC calls are going to be protected with mTLS.

[ahmedtd]

One open question I have is: if the environment variable values change on resume (because we read them again from the configmap, and the configmap was updated), does that break the gVisor snapshot?

They definitely won't be updated in the memory of the running snapshot, but I don't know what is safe to change in the OCI bundle vs what is not when restoring from a snapshot.

[ahmedtd]

nybidari or Fabricio Voznika (@fvoznika) Do you know the answer to the above?

[EItanya]

Probably the design that makes the most sense for now is to make sure that ate-api-server has permission to read all relevant ConfigMaps and Secrets, and interpolate the values into the RPC call to atelet. ate-api-server is already highly privileged, and all of the RPC calls are going to be protected with mTLS.

This is the design I went with for #20, I left it draft only because I wanted to continue the discussion but the implementation should be ok.

Finally, NOT supporting secrets is guaranteed to lead to ugly things like putting them in plaintext because "there was no other way to do it" (and agents especially will do this if they feel they have to get a job done), so +1 it's VERY important we address this gap.

I couldn't agree with this more.

I think in the long-term saying that secrets can never be injected into actors directly may be overly restrictive, but for agents of course injecting on the wire is much more secure. I wonder if an API that allows for either/or could be the most flexible and allow for the most security in the long-term.

[fvoznika]

We decided to allow changes in environment variables between save/restore sessions, especially to support golden images and fork cases where parts of the environment changes. However, environment variables are stored in a memory block in the user mode processes and can't be changed by the kernel. So upon restore, the new environment variables are placed in /proc/gvisor/spec_environ and the application must refresh them to get the new values. More details here.

[mtaufen]

Probably the design that makes the most sense for now is to make sure that ate-api-server has permission to read all relevant ConfigMaps and Secrets, and interpolate the values into the RPC call to atelet. ate-api-server is already highly privileged, and all of the RPC calls are going to be protected with mTLS.

We need to think about scalability here. Is the intent that these sources be per-template or per-actor?

[mtaufen]

Not to mention security - ate-api-server still runs in the cluster, so this is giving a handful of K8s nodes access to many/most secrets. Now I wish we'd done a real authorizer for ReferenceGrant. https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/3766-referencegrant

[mtaufen]

interpolate the values into the RPC call to atelet. ate-api-server is already highly privileged, and all of the RPC calls are going to be protected with mTLS

Do we log RPC arguments today? Can we tag this data as sensitive for redaction before any RPC logs are printed?