Allow for scripts in CKS ISO to fully control CKS deployment

[JSpon]

Description

This PR...

allows for the CKS cluster to be installed and configured with configuration files in the ISO, thus allowing for more flexibility in the cluster creation. This will allow for the cluster to be built with a custom configuration for kubeadm, as well as packages to be installed using tools like Helm.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

This has been tested in our test environment with custom ISOs that install Cilium instead of Weave, install/upgrade custom Helm charts as part of the deployment, and configure CSI drivers so that CloudStack volumes can be used for persistent volumes.

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 15.28%. Comparing base (21af134) to head (0d7cc4f).
Report is 52 commits behind head on main.

❗ Current head 0d7cc4f differs from pull request most recent head d1f2c6c

Please upload reports for the commit d1f2c6c to get more accurate results.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9087      +/-   ##
============================================
- Coverage     15.28%   15.28%   -0.01%     
- Complexity    11535    11538       +3     
============================================
  Files          5425     5424       -1     
  Lines        474138   474334     +196     
  Branches      58984    59493     +509     
============================================
+ Hits          72486    72508      +22     
- Misses       393584   393768     +184     
+ Partials       8068     8058      -10     

Flag	Coverage Δ
uitests	4.25% <ø> (-0.02%)	⬇️
unittests	16.02% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shwstppr]

@JSpon does this also need a change in create-kubernetes-iso.sh script to include the custom script while preparing the ISO?

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9638

[weizhouapache]

I like the idea to setup cks cluster in another way than user-data.
however, as @shwstppr said, there should be some changes with create-kubernetes-iso.sh

can you please also share the setup-kube-system files, for example , to install cilium ? @JSpon
the setup process might be very complicated for end users, to be honest.

[JSpon]

@shwstppr, @weizhouapache - sure. I am testing examples of setup-kube-system (and deploy-kube-system) and will update my PR with the changes to the create-kubernetes-binaries-iso.sh and the examples (and a README.md) after testing is complete

[JSpon]

I made the update to add examples. I had an issue with the first commit, so fixed it and force pushed with the updates.

[weizhouapache]

I made the update to add examples. I had an issue with the first commit, so fixed it and force pushed with the updates.

thanks @JSpon

it looks like refactoring, almost all codes are copied from the k8s-*.yaml
can you point out the main changes ?

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9716

[JSpon]

@weizhouapache, the code in basic/scripts is a direct copy, meant as a starting point. The code in cilium has changes only in control-node. The changes are lines 29-35 and 51-63 in setup-kube-system and 31 and 56-57 in deploy-kube-system.tmpl. I can remove the other directories (control-node-add and node) as well as setup-containerd.tmpl in cilium since they aren't needed.

[DaanHoogland]

@blueorangutan test alma9 kvm-alma9

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-10329)
Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9
Total time taken: 50281 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9087-t10329-kvm-alma9.zip
Smoke tests completed. 131 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@JSpon , sorry to bother you with the apache bylaws, but all files need a license header, except the very trivial ones.

[JSpon]

@JSpon , sorry to bother you with the apache bylaws, but all files need a license header, except the very trivial ones.

Sorry about that. Thanks for pointing those out. I have made the updates.

[weizhouapache]

@JSpon
To be frank, I think we need to solve two problems

• lots of code are duplicated.
• lots of parameters are passed to setup-kube-system

I do not have suggestion for now.

I have some simple changes to install cilium by command lines. I will create a draft PR. Maybe we can have more discussions and get a better solution.

I like the idea to run some customized scripts before or after k8s setup.

[JSpon]

@weizhouapache, understood. I can also come up with a shell script to extract the information from the existing yml files instead of including copies, and figure a way of easily add cilium into that. This would remove the 18 sample files with all of the duplication in favor of the script, if it is worth it.

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13630

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

@JSpon @shwstppr @Pearl1594 , is this still relevant?

[weizhouapache]

@JSpon
I am closing this PR,

there are lots of changes in CloudStack 4.21.0 (see #9102), please have a look