Skip to content

[multiple] Accept CA certificate if expired when CRC is used#3720

Merged
danpawlik merged 1 commit intoopenstack-k8s-operators:mainfrom
danpawlik:fix-molecule-reproducer
Mar 3, 2026
Merged

[multiple] Accept CA certificate if expired when CRC is used#3720
danpawlik merged 1 commit intoopenstack-k8s-operators:mainfrom
danpawlik:fix-molecule-reproducer

Conversation

@danpawlik
Copy link
Copy Markdown
Contributor

@danpawlik danpawlik commented Feb 26, 2026

It can happen that the image used by the CRC has expired certificate,
so it require to approve new generated certificate before making test.
The CRC log shows a log message:

level=info msg="Kubelet serving certificate has expired, waiting for automatic renewal... [will take up to 5 minutes]"
Failed to renew TLS certificates: please check if a newer CRC release is available:
Temporary error: certificate /var/lib/kubelet/pki/kubelet-server-current.pem still expired (x59)

Wait for the cluster to be stable when CA cert is expired.
Also move all crc command to single playbook, for easier
maintenance.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Feb 26, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign brjackma for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@danpawlik danpawlik force-pushed the fix-molecule-reproducer branch from 6e845ff to 0ed017c Compare February 26, 2026 19:11
@danpawlik danpawlik changed the title [reproducer] Accept CA certificate if expired when CRC is used [multiple] Accept CA certificate if expired when CRC is used Feb 26, 2026
@danpawlik danpawlik force-pushed the fix-molecule-reproducer branch from 0ed017c to b68f767 Compare February 26, 2026 19:45
@danpawlik danpawlik marked this pull request as draft February 26, 2026 19:57
@danpawlik
Copy link
Copy Markdown
Contributor Author

It seems all images are outdated. Waiting for infra team to replace images.

@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/549ca1031aa64fd88a1ffbc83a6e03df

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 13m 00s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 34s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 30m 25s
cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 12m 19s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 10s
✔️ adoption-standalone-to-crc-ceph-provider SUCCESS in 2h 56m 50s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 10s
cifmw-pod-pre-commit FAILURE in 9m 10s
✔️ cifmw-molecule-cert_manager SUCCESS in 22m 15s
✔️ cifmw-molecule-ci_local_storage SUCCESS in 28m 02s
✔️ cifmw-molecule-cifmw_helpers SUCCESS in 4m 42s
cifmw-molecule-env_op_images TIMED_OUT in 30m 55s
cifmw-molecule-install_openstack_ca FAILURE in 24m 33s
✔️ cifmw-molecule-manage_secrets SUCCESS in 30m 07s
cifmw-molecule-openshift_login TIMED_OUT in 30m 55s
✔️ cifmw-molecule-openshift_obs SUCCESS in 26m 48s
✔️ cifmw-molecule-openshift_setup SUCCESS in 21m 40s
✔️ cifmw-molecule-operator_deploy SUCCESS in 12m 10s
✔️ cifmw-molecule-os_must_gather SUCCESS in 24m 40s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 47s
✔️ cifmw-molecule-set_openstack_containers SUCCESS in 26m 12s
cifmw-molecule-shiftstack FAILURE in 22m 33s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 28m 13s
✔️ cifmw-molecule-tofu SUCCESS in 13m 48s

@danpawlik
Copy link
Copy Markdown
Contributor Author

recheck

@danpawlik danpawlik requested a review from a team February 27, 2026 11:12
@danpawlik danpawlik force-pushed the fix-molecule-reproducer branch 2 times, most recently from 8631fae to d0c55c1 Compare February 27, 2026 11:43
@danpawlik danpawlik marked this pull request as ready for review February 27, 2026 11:57
@danpawlik danpawlik requested a review from holser February 27, 2026 11:58
Comment thread roles/cifmw_helpers/tasks/crc_start.yml Outdated
Comment thread roles/cifmw_helpers/tasks/crc_start.yml
@danpawlik danpawlik force-pushed the fix-molecule-reproducer branch 2 times, most recently from 466ec7e to 289154a Compare February 27, 2026 13:51
@holser holser self-requested a review February 27, 2026 13:56
holser
holser previously approved these changes Feb 27, 2026
Copy link
Copy Markdown
Contributor

@holser holser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

Comment thread roles/cifmw_helpers/tasks/crc_start.yml Outdated
Comment thread roles/cifmw_helpers/tasks/crc_start.yml Outdated
ignore_errors: true # noqa: ignore-errors

- name: Login to the OpenShift when certificate is expired
when: "'Kubelet serving certificate has expired' in _crc_output.stderr"
Copy link
Copy Markdown
Contributor

@evallesp evallesp Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) suggestion: I'd go by creating an above task that checks the previous _crc_output to fail with message:

msg: >-
      CRC start failed with an unexpected error:
      {{ _crc_output.stderr }}

if: _crc_output.rc != 0 AND "'Kubelet serving certificate has expired' not in _crc_output.stderr"
So if there's an error unrelated to certificates, this would fail with a clear message.
If the error is related to certificates the fail task wouldn't execute and then we go to the certificate recovery path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its printing already

Copy link
Copy Markdown
Contributor

@michburk michburk Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the crc start fails for some reason other than the cert issue we're expecting here, we still ignore whatever error that was because of ignore_errors in the above task, right? Is it safe to assume we can continue on to other tasks without manually failing like Enrique is suggesting? Or are we certain that this task (Login to the OpenShift when certificate is expired) will fail? If so, it could still save some time to manually fail early and avoid all of the retries, right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this printing if the error is not related with expired certificate?

Comment thread roles/reproducer/tasks/ocp_layout.yml Outdated
Comment thread roles/cifmw_helpers/tasks/crc_start.yml
Comment thread roles/cifmw_helpers/tasks/crc_start.yml
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Feb 27, 2026

New changes are detected. LGTM label has been removed.

@danpawlik danpawlik requested a review from holser February 27, 2026 16:18
@danpawlik danpawlik requested review from a team, evallesp and michburk February 27, 2026 16:18
Comment thread roles/cifmw_helpers/tasks/crc_start.yml Outdated
It can happen that the image used by the CRC has expired certificate,
so it require to approve new generated certificate before making test.
The CRC log shows a log message:

    level=info msg="Kubelet serving certificate has expired, waiting for automatic renewal... [will take up to 5 minutes]"
    Failed to renew TLS certificates: please check if a newer CRC release is available:
    Temporary error: certificate /var/lib/kubelet/pki/kubelet-server-current.pem still expired (x59)

Wait for the cluster to be stable when CA cert is expired.
Also move all crc command to single playbook, for easier
maintenance.

Signed-off-by: Daniel Pawlik <dpawlik@redhat.com>
@danpawlik danpawlik force-pushed the fix-molecule-reproducer branch from 4a595fb to 252e5f5 Compare February 27, 2026 16:45
@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/551e59eab468480fa8193bba79402152

openstack-k8s-operators-content-provider FAILURE in 4m 56s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 5m 37s
⚠️ adoption-standalone-to-crc-ceph-provider SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 11s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 13s
cifmw-molecule-cert_manager FAILURE in 14m 06s
✔️ cifmw-molecule-ci_local_storage SUCCESS in 12m 48s
✔️ cifmw-molecule-cifmw_helpers SUCCESS in 4m 40s
✔️ cifmw-molecule-env_op_images SUCCESS in 23m 41s
✔️ cifmw-molecule-install_openstack_ca SUCCESS in 25m 58s
✔️ cifmw-molecule-manage_secrets SUCCESS in 21m 04s
✔️ cifmw-molecule-openshift_login SUCCESS in 24m 39s
✔️ cifmw-molecule-openshift_obs SUCCESS in 12m 42s
✔️ cifmw-molecule-openshift_setup SUCCESS in 13m 28s
✔️ cifmw-molecule-operator_deploy SUCCESS in 14m 35s
✔️ cifmw-molecule-os_must_gather SUCCESS in 25m 35s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 10s
✔️ cifmw-molecule-set_openstack_containers SUCCESS in 23m 34s
✔️ cifmw-molecule-shiftstack SUCCESS in 16m 21s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 17m 15s
✔️ cifmw-molecule-tofu SUCCESS in 11m 56s

@danpawlik
Copy link
Copy Markdown
Contributor Author

recheck

1 similar comment
@danpawlik
Copy link
Copy Markdown
Contributor Author

recheck

@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d63b44ff8fe84106950a1d386732790d

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 29m 52s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 32m 06s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 28m 55s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 15m 45s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 35s
adoption-standalone-to-crc-ceph-provider FAILURE in 1h 13m 23s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 41s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 38s
✔️ cifmw-molecule-cert_manager SUCCESS in 12m 34s
✔️ cifmw-molecule-ci_local_storage SUCCESS in 12m 20s
✔️ cifmw-molecule-cifmw_helpers SUCCESS in 4m 34s
✔️ cifmw-molecule-env_op_images SUCCESS in 21m 25s
✔️ cifmw-molecule-install_openstack_ca SUCCESS in 29m 20s
✔️ cifmw-molecule-manage_secrets SUCCESS in 19m 23s
✔️ cifmw-molecule-openshift_login SUCCESS in 25m 50s
✔️ cifmw-molecule-openshift_obs SUCCESS in 13m 02s
✔️ cifmw-molecule-openshift_setup SUCCESS in 11m 42s
✔️ cifmw-molecule-operator_deploy SUCCESS in 17m 58s
✔️ cifmw-molecule-os_must_gather SUCCESS in 25m 01s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 14s
✔️ cifmw-molecule-set_openstack_containers SUCCESS in 24m 29s
✔️ cifmw-molecule-shiftstack SUCCESS in 13m 12s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 16m 54s
✔️ cifmw-molecule-tofu SUCCESS in 15m 20s

@danpawlik
Copy link
Copy Markdown
Contributor Author

recheck

@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d70505b1aebe43d8a9bf9e4a0b1d0247

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 11m 45s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 02s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 28m 04s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 59m 38s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 40s
✔️ adoption-standalone-to-crc-ceph-provider SUCCESS in 2h 58m 35s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 59s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 33s
✔️ cifmw-molecule-cert_manager SUCCESS in 12m 28s
✔️ cifmw-molecule-ci_local_storage SUCCESS in 12m 26s
✔️ cifmw-molecule-cifmw_helpers SUCCESS in 4m 35s
cifmw-molecule-env_op_images FAILURE in 18m 05s
✔️ cifmw-molecule-install_openstack_ca SUCCESS in 25m 34s
✔️ cifmw-molecule-manage_secrets SUCCESS in 21m 54s
✔️ cifmw-molecule-openshift_login SUCCESS in 22m 53s
✔️ cifmw-molecule-openshift_obs SUCCESS in 12m 52s
✔️ cifmw-molecule-openshift_setup SUCCESS in 11m 48s
✔️ cifmw-molecule-operator_deploy SUCCESS in 16m 46s
✔️ cifmw-molecule-os_must_gather SUCCESS in 25m 44s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 19s
✔️ cifmw-molecule-set_openstack_containers SUCCESS in 25m 11s
✔️ cifmw-molecule-shiftstack SUCCESS in 13m 26s
✔️ cifmw-molecule-sushy_emulator SUCCESS in 18m 57s
✔️ cifmw-molecule-tofu SUCCESS in 15m 10s

@michburk
Copy link
Copy Markdown
Contributor

michburk commented Mar 2, 2026

recheck

@danpawlik danpawlik merged commit d93ea23 into openstack-k8s-operators:main Mar 3, 2026
7 of 8 checks passed
@danpawlik danpawlik deleted the fix-molecule-reproducer branch March 3, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants