Provisioning the control VM failed in the Kubernetes cluster

[anirudh09041]

Getting the below error when provisioning the cluster.

Error
Provisioning the control VM failed in the Kubernetes cluster : test-cluster-2

in Error description i got this :
Description
Error while creating Kubernetes cluster. domain(s): [test-cluster-2-control-19db5014c45, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local] addresses: [10.96.33.163]

What could be the underlying issue ?? How can i resolve this.

[laxmansingh18200]

Facing the same issue, any help will be appreciated

[DaanHoogland]

Any exceptions in logs, @anirudh09041 ?

[anirudh09041]

I can see now the control vm is getting started but when i login through console i see the below message and after 30 mins the cluster provisioning fails.

[anirudh09041]

i am using the below configuration

compute offering

Template

[DaanHoogland]

please study the logs for any related errors.

[anirudh09041]

HI @DaanHoogland , i found out that iso is not getting attached

[root@cloudstack secondary]# virsh domblklist i-2-120-VM --details
Type Device Target Source

file disk vda /var/lib/libvirt/images/38d592d7-dc21-47b7-84ee-c6489687959f
file cdrom hdc -

[root@cloudstack secondary]#

[anirudh09041]

i have set up the iso in the below way

[Pearl1594]

Few questions @anirudh09041

1. In the management server logs, you do see the attachIso command being sent?
2. Can you ssh into the control node ssh -i -p 2222 cloud@<control_ip>
3. If you can SSH into the control node, can you check the /var/log/cloud-init-output.log or /var/log/daemon.log files to see if there are any issues reported there.
4. Does the node provisioning time out waiting on the binaries? or it starts after sometime ?
5. Is this consistent or a one-off occurrence?
6. Can you try force stopping and starting the cluster just to see if that works.

[kiranchavala]

@anirudh09041 what is the ubuntu template are you using the templates fromt this link

https://download.cloudstack.org/testing/custom_templates/ubuntu/22.04/

[anirudh09041]

@kiranchavala , i have tried both the below templates but same issue

https://download.cloudstack.org/testing/custom_templates/ubuntu/22.04/cks-ubuntu-2204-kvm.qcow2.bz2

https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img

[anirudh09041]

HI @Pearl1594 , below are the answers to the previously mentioned questions. Can you please have a look.

Yes, the attachIso command is being sent and it even succeeds. Here's the exact sequence from my management server log:

06:50:55.765 - AttachCommand sent for i-2-120-VM
06:50:55.955 - "Attached binaries ISO for VM" (SUCCESS)
06:50:56.043 - DettachCommand sent for the SAME ISO on the SAME VM
The ISO attaches successfully but is immediately detached ~278ms later by a DettachCommand (note the typo "Dettach" instead of "Detach"). This happens for both control and worker nodes.

I cannot SSH into the control node because port forwarding doesn't appear to be set up. The VM is on an isolated network (192.168.100.x).

The cloud-init script times out waiting for /mnt/k8sdisk/ after 100 attempts (25 minutes), which is expected since the ISO is detached.

This is 100% consistent and happens every time.

When I manually attach the ISO via virsh change-media, the ISO stays attached but the VM is still stuck at the login prompt because the patch.sh deployment script is hung waiting for qemu-guest-agent (which is not installed in my Ubuntu cloud image).

[Pearl1594]

since you are using the public templates and it's not completed provisioning, can you try logging in via console use username: cloud passowrd: cloud. If you can do that, attach the iso again and run the /opt/bin/setup-kube-system script and then the deploy-kube-system script. I'm not sure if you'd be able to log in with those, after a specific point of cluster setup, it revokes access.

[anirudh09041]

I tried doing login but not able to login.

According to you what should be the ideal templates which i should use and from where.
Kubernetes ISOs and Templates in order to provision a cks cluster.

[rishabhjain1997]

Hi @anirudh09041 and @Pearl1594. I'm a teammate of @anirudh09041 and here is what I've found so far. @Pearl1594 can you please help us with a solution based on the RCA below?

1. The actual error seems to be the async job error:

 $ grep "Complete async job-774" /var/log/cloudstack/management/management-server.log
                                                                                                                                                        
  2026-05-05 06:50:56,274 DEBUG ... Complete async job-774, jobStatus: FAILED, resultCode: 530,                                                         
  result: ... "errortext":"Failed to setup Kubernetes cluster : test-cluster-23 is not in                                                               
  usable state as the system is unable to access control node VMs of the cluster"   

KubernetesClusterStartWorker logs show a similar error:

2026-05-05 06:50:56,035 ERROR [c.c.k.c.a.KubernetesClusterStartWorker]                                                                                
  Failed to setup Kubernetes cluster : test-cluster-23 is not in usable state                                                                           
  as the system is unable to access control node VMs of the cluster

So the real failure seems to be: CKS finishes provisioning the VMs and FW/PF/LB rules, then calls KubernetesClusterUtil.isKubernetesClusterServerRunning(), which fails because the API server never came up.

The kube-apiserver never came up because each VM was stuck "Starting" for 50 minutes

$ grep -E "i-2-120-VM.*StartCommand|i-2-120-VM.*Start completed" \                                                                                    
      /var/log/cloudstack/management/management-server.log                                                                                              
                                                                                                                                                        
  05:10:40,293 ... Sending { Cmd ... StartCommand ... id=120 ...     <- StartCommand sent                                                               
  06:00:43,090 ... Start completed for VM ... i-2-120-VM             <- StartAnswer arrived  

So cluster provisioning sat for ~100 min (50 min × 2 VMs sequentially) just waiting for the VMs to be marked Started.

The 50-minute stall is the KVM agent repeatedly retrying patch.sh

$ grep "patch.sh -n i-2-120-VM" /var/log/cloudstack/agent/agent.log                                                                                   
                                                                                                                                                        
  05:15:43 WARN ... Process [2551078] for command [/usr/share/cloudstack-common/scripts/                                                                
     vm/hypervisor/kvm/patch.sh -n i-2-120-VM -c template=domP name=test-cluster-23-control                                                             
     ...] timed out. Output is [].                                                                                                                      
  05:20:43 WARN ... Process [2567946] ... timed out. Output is [].                                                                                      
  05:25:43 WARN ... Process [2588320] ... timed out. Output is [].                                                                                      
  05:30:43 WARN ... Process [2605401] ... timed out. Output is [].                                                                                      
  05:35:43 WARN ... Process [2621993] ... timed out. Output is [].                                                                                      
  05:40:43 WARN ... Process [2638725] ... timed out. Output is [].                                                                                      
  05:45:43 WARN ... Process [2655334] ... timed out. Output is [].                                                                                      
  05:50:43 WARN ... Process [2672092] ... timed out. Output is [].
  05:55:43 WARN ... Process [2688808] ... timed out. Output is [].                                                                                      
  06:00:43 WARN ... Process [2705703] ... timed out. Output is [].

patch.sh times out because qemu-guest-agent isn’t responding in the guest

[root@cloudstack ~]# virsh qemu-agent-command i-2-120-VM '{"execute":"guest-ping"}' --timeout 5
error: Guest agent is not responding: QEMU guest agent is not connected

Here is the custom template that we’re using:

mysql> SELECT name, format, url FROM vm_template WHERE id = 217;
  +------------------+--------+-------------------------------------------+                                                                             
  | name             | format | url                                       |                                                                             
  +------------------+--------+-------------------------------------------+                                                                             
  | Ubuntu-CKS-Fresh | QCOW2  | http://10.96.32.32/ubuntu-cks-fresh.qcow2 |                                                                             
  +------------------+--------+-------------------------------------------+   

That template doesn't seem to have qemu-guest-agent installed (or running at boot), so patch.sh has nothing to talk to — hence the timeout in the previous block.

And patch.sh hangs trying to inject /var/lib/cloud/data/cmdline

 mysql> SELECT name, value FROM configuration
         WHERE name IN ('cloud.kubernetes.control.node.install.attempt.wait.duration',
                        'cloud.kubernetes.control.node.install.reattempt.count');                                                                       
  +-------------------------------------------------------------+-------+                                                                               
  | cloud.kubernetes.control.node.install.attempt.wait.duration |    15 |                                                                               
  | cloud.kubernetes.control.node.install.reattempt.count       |   100 |                                                                               
  +-------------------------------------------------------------+-------+   

Even after patch.sh finally gives up, cloud-init inside the guest can't make progress: it sits in its own retry loop waiting for the binaries ISO at /mnt/k8sdisk/ to appear. The retry length is configured globally as 100 × 15s = 25 min:

CKS only actually attaches the binaries ISO at the very end of the workflow (06:50:55 below) — by then cloud-init has been dead for over an hour.

$ grep -E "Attached binaries ISO|Detached Kubernetes binaries|isKubernetesClusterServerRunning" \
      /var/log/cloudstack/management/management-server.log                                                                                              
   
  06:50:55,955 INFO  ... Attached binaries ISO for VM: ... i-2-120-VM                                                                                   
  06:50:56,034 INFO  ... Attached binaries ISO for VM: ... i-2-122-VM
  06:50:56,035 ERROR ... Failed to setup Kubernetes cluster ...                                                                                         
  06:50:56,145 INFO  ... Detached Kubernetes binaries from VM: ... i-2-120-VM
  06:50:56,271 INFO  ... Detached Kubernetes binaries from VM: ... i-2-122-VM     

And the ISO attach is mmediately followed by a detach 1s later.

Summary:

qemu-guest-agent is missing or disabled in the Ubuntu-CKS-Fresh template, so the host's patch.sh has nothing to talk to and times out for 50 minutes per VM (~100 min total for both). Without patch.sh, the cmdline metadata is never injected, the cks-init module inside cloud-init never runs, and kubeadm-init never executes. When CKS eventually calls isKubernetesClusterServerRunning(), the API server isn't up, the readiness check fails, and the cluster goes to Error.

@Pearl1594 please let us know how to remediate this issue. Any help would be greatly appreciated!

[rishabhjain1997]

Update — qemu-guest-agent autostart issue persists

Hi @Pearl1594 and @anirudh09041 — quick update on the patch.sh / qga investigation.

What we did

1. Disabled the two new hosts (cloudstack-node2.local, kvm2.cloudstack.local) so VMs are forced onto cloudstack.local.

2. Patched the Ubuntu cloud image (/var/www/html/ubuntu-cks-fresh-qga.qcow2) to install qemu-guest-agent, containerd, and CKS deps via virt-customize --install.

3. Found that systemctl enable qemu-guest-agent.service was a no-op because the unit shipped by Ubuntu has an empty [Install] section — it's "static" and relies on udev/device activation:

   [Unit]       
   Description=QEMU Guest Agent
   BindsTo=dev-virtio\x2dports-org.qemu.guest_agent.0.device
   After=dev-virtio\x2dports-org.qemu.guest_agent.0.device                                                                          
   
   [Service]                                                                                                                        
   ExecStart=-/usr/sbin/qemu-ga
   Restart=always                                                                                                                   
   RestartSec=0
                                                                                                                                    
   [Install]    
   

4. As a workaround, manually created the symlink inside the qcow2:

/etc/systemd/system/multi-user.target.wants/qemu-guest-agent.service
-> /lib/systemd/system/qemu-guest-agent.service
5. Discovered the existing Ubuntu-CKS-Fresh-qga-v2 template was registered against the wrong URL (pointed at the v1 file). Re-registered as Ubuntu-CKS-Fresh-qga-v3 against the patched file and provisioned test-cluster-29.

Current state

• VM i-2-143-VM for test-cluster-29 is on cloudstack.local and has been in Starting for ~28 minutes.
• patch.sh still times out after 5 min:

WARN ... Process [...] for command [.../patch.sh -n i-2-143-VM ...] timed out. Output is [].
ERROR ... Passing cmdline failed:timeout

• virsh qemu-agent-command i-2-143-VM '{"execute":"guest-ping"}' →

error: Guest agent is not responding: QEMU guest agent is not connected

Hypothesis

The wants/ symlink alone isn't enough. BindsTo= blocks systemd from starting qemu-guest-agent.service until the virtio-port device unit (dev-virtio\x2dports-org.qemu.guest_agent.0.device) activates — and either the udev rule isn't firing in our boot, or the device unit never resolves.

Checking further on this

[rishabhjain1997]

Would it be easier to build with a RHEL image?

[mehulved]

The template is missing qemu-guest-agent.
I have been able to get it up and running by running the following steps:

• Start a VM with the Ubuntu 22.04 CKS template
• SSH into the VM
• Install qemu-guest-agent
• Stop the VM
• Save the disk as a new template
• Create a kubernetes cluster using the new template.

The cluster will be up and running but the status will be stuck at Starting as the CSI driver isn't able to pick up the correct disk offering. I'm looking into this issue and will update as and when resolved.
I have tested my cluster otherwise and can confirm that basics work:

• Running Pods
• Scaling deployments
• Ingress using nodeport
• Internet Reachability
• DNS
• Storage(once CSI is manually configured)
• GPU Operator(where GPUs are present)