Datapact demo with emulation by aleenathomas · Pull Request #192 · DataCloud-project/SIM-PIPE

aleenathomas · 2026-03-10T15:50:50Z

Main features added

integration with Moose API
added virtualization module which runs in Debian and WSL
edited install scripts
refactoring

- storing k3s server api and node token in a secret

- added new clusterrole and clusterrole binding for controller

- added dependencies - added script

works: deployment on worker node wip: metrics collection from worker node

works: metrics collection from worker node wip: automation

trimmed create kube node script tested argo workflows with output files on worker node removed host network true in cadvisor added frontend page for vmnodes added crd added get all vmnodes wip create new vmnode

- added nodes section - added node selection in create dry run modal wip - Edit create dry run mutation and resolver to insert new parameter Edit argo workflows to insert the name of the vmnodes crd as node selector

- added status in nodes crd

- Insert selected node into argo workflows automatically

fungiboletus · 2026-03-17T14:33:21Z

charts/simpipe/templates/postgresql-statefulset.yaml

@@ -0,0 +1,69 @@
+{{- if .Values.postgresql.enabled -}}


We could consider switching to CloudNativePG one day. Not this time.

fungiboletus · 2026-03-17T14:33:50Z

charts/simpipe/Chart.lock

I don't know if we should remove the .lock to be honest.

charts/simpipe/values.yaml

fungiboletus · 2026-03-17T14:37:34Z

charts/simpipe/values.yaml

+  resources:
+    requests:
+      memory: 512Mi
+      cpu: 250m
+    limits:
+      memory: 1Gi
+      cpu: 1000m


I'm usually not a fan of having this set by default because it's more annoying than useful in my opinion.

charts/simpipe/values.yaml

fungiboletus · 2026-03-17T14:38:37Z

charts/simpipe/values.yaml

+    requests:
+      memory: 256Mi
+      cpu: 250m
+    limits:
+      memory: 1Gi
+      cpu: 1000m


Also defaults that could be left unset.

fungiboletus · 2026-03-17T14:38:58Z

charts/simpipe/values.yaml

-  simpipeRefreshInterval: 3s
-  simpipeScrapeTimeout: 2s
+  simpipeRefreshInterval: 7s
+  simpipeScrapeTimeout: 7s


Are those the new values that are HP compatible ?

Yes actually these values can be changed back since we are doing virtualization (must faster) instead of emulation.

fungiboletus · 2026-03-17T14:39:49Z

controller/src/argo/emulation-utils.ts

+  if (osInfo.ID === 'debian') {
+    return 'debian13';
+  }


I'm not convinced about this code snippet long term.

fungiboletus · 2026-03-17T14:40:27Z

controller/src/argo/test

Should be removed.

fungiboletus · 2026-03-17T14:41:38Z

controller/src/emulation/scripts/create-kube-node_qemu.sh

+  - iptables -A INPUT -p udp --dport 8472 -j ACCEPT
+
+  # Start K3s agent only after Docker is ready
+  - curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.4+k3s1 INSTALL_K3S_EXEC="agent --docker" K3S_URL=${K3S_SERVER_URL} K3S_TOKEN=${K3S_TOKEN} sh -


Here we are hardcoding k3s version and also installing it without ansible. I guess it's fine but we may have some mismatches long term.

Moving the version number to a config variable for now.

fungiboletus · 2026-03-17T14:42:35Z

controller/src/emulation/scripts/create-kube-node_wsl.sh

+  - bash -c 'ip route del default via 172.30.0.1 dev lo || true'
+
+  # Start K3s agent (let k3s/flannel pick the interface/IP based on the
+  - curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.4+k3s1 INSTALL_K3S_EXEC="agent --docker" K3S_URL=${K3S_SERVER_URL} K3S_TOKEN=${K3S_TOKEN} sh - 


fungiboletus · 2026-03-17T14:43:06Z

controller/src/emulation/scripts/create-kube-node_wsl.sh

+  DNS1=$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf)
+fi
+if [ -z "$DNS1" ]; then
+  DNS1="10.43.0.10"


The last time we were using Google DNS, and this time a local DNS resolver. Is this specific to WSL ?

fungiboletus · 2026-03-17T14:45:10Z

controller/src/k8s/api-tokens.ts

+function maskSecret(value: string): string | undefined {
+  const trimmed = value.trim();
+  if (!trimmed) return undefined;
+  if (trimmed.length <= 4) return '*'.repeat(trimmed.length);
+
+  const prefix = trimmed.slice(0, 2);
+  const suffix = trimmed.slice(-2);
+  const maskLength = Math.min(Math.max(trimmed.length - 4, 4), 12);
+  return `${prefix}${'*'.repeat(maskLength)}${suffix}`;
+}


Yep, this isn't safe.

Suggested change

function maskSecret(value: string): string | undefined {

const trimmed = value.trim();

if (!trimmed) return undefined;

if (trimmed.length <= 4) return '*'.repeat(trimmed.length);

const prefix = trimmed.slice(0, 2);

const suffix = trimmed.slice(-2);

const maskLength = Math.min(Math.max(trimmed.length - 4, 4), 12);

return `${prefix}${'*'.repeat(maskLength)}${suffix}`;

}

function maskSecret(value: string): string {

return "************";

}

fungiboletus · 2026-03-17T14:45:40Z

controller/src/k8s/api-tokens.ts

+
+  return {
+    hasValue: true,
+    maskedPreview: maskSecret(trimmed),


Or even better:

Suggested change

maskedPreview: maskSecret(trimmed),

maskedPreview: "************"

fungiboletus · 2026-03-17T14:46:53Z

controller/src/k8s/resources.ts

+  const processTable = await getProcessTable();
+  if (!processTable) return false;


So now we assume that simpipe's controller run on the qemu host, which is a significant thing in the architecture.

controller/src/moose/moose.ts

controller/src/config.ts

controller/Dockerfile

fungiboletus · 2026-03-17T14:52:42Z

controller/Dockerfile

+wget -O /app/images/ubuntu-18.qcow2 https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img && \
+wget -O /app/images/ubuntu-20.qcow2 https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img && \
+wget -O /app/images/ubuntu-22.qcow2 https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img && \
+wget -O /app/images/ubuntu-24.qcow2 https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img


That creates a huge container. Shouldn't we download that at runtime in a volume.

fungiboletus · 2026-03-17T14:55:12Z

cloud.iso

should be removed :)

fungiboletus · 2026-03-17T14:55:49Z

install.py

-    except subprocess.CalledProcessError as e:
-        print(f"❌ Error while installing helm-diff plugin: {e}")
-        sys.exit(1)
+    print("ℹ️ Skipping helm-diff plugin (not required for SIM-PIPE).")


We could probably just remove all mention to helm diff.

fungiboletus · 2026-03-17T14:56:13Z

meta-data

not sure this should stay

fungiboletus · 2026-03-17T14:56:21Z

network-config

this one may have to go as well?

fungiboletus · 2026-03-17T14:56:51Z

user-data

Should this be removed?

fungiboletus · 2026-03-17T15:00:32Z

Alright, I made some comments. Overall I think it's mostly the secret thing that should be fixed. We shouldn't disclose the size of the secret nor the beginning and the end in my opinion.
Some files should be removed.

I'm not a svelte developer so I couldn't asses the frontend code very well but it did sound fair.

Overall, for the record, I'm not sure where this is going long term. We have moose now, not sure why, I have said it in person already, but this sounds like out of scope.

The project is now very much about virtualisation with qemu, which is fine. Perhaps one could have used kubevirt instead of making qemu from scratch in kubernetes, but i don't know the pros and cons.

I think someone didn't like helm diff :D

We are overall running quite old dependencies at this point, but for a low TRL demo this is fine.

…aCloud-project/SIM-PIPE into datapact-demo-with-emulation

aleenathomas and others added 30 commits January 20, 2025 13:07

Merge branch 'main' into feature/emulation

b9638c2

Parametrized script

f3cb830

WIP: adding graphql resolver/dockerfile

ed5c7c1

Added kubernetes secret for cluster join details

8be0de6

- storing k3s server api and node token in a secret

Edited helm to add permissions for controller to create new cluster node

8298798

- added new clusterrole and clusterrole binding for controller

Fix: undefined workflow templates

e12dd99

Updated controller dockerfile for emulation

8a8f6ab

- added dependencies - added script

Added emulation resources section in frontend

23ffa6e

added graphql mutation to create a new resource

ec13401

Edited graphlql schema to add a new resource

255483f

Added create new resource modal

f81cf0b

WIP: connecting frontend and backend for emulation

c7bd65a

Added listing and deleting of resources in frontend

32db6ef

Added CRD for persisting resources

0f0d4c8

Added create and delete resource in schema and resolvers

205c90b

Preloaded os images in dockerfile

6f7aa69

Updated permissions to manage the new resource CRDs

d2a86de

Added shutdown script for deleting emulation resources

767d2ee

WIP: Temp files

06a531d

Partial update

9152630

git clean

949a35a

WIP: working version

c36bb92

Merge remote-tracking branch 'origin/main' into feature/emulation

8725916

solve metrics collection issue

1faae3c

works: deployment on worker node wip: metrics collection from worker node

automation

a60252c

works: metrics collection from worker node wip: automation

new:

b31c9d9

trimmed create kube node script tested argo workflows with output files on worker node removed host network true in cadvisor added frontend page for vmnodes added crd added get all vmnodes wip create new vmnode

Commented out print statements

e0e2683

Frontend

0e18170

- added nodes section - added node selection in create dry run modal wip - Edit create dry run mutation and resolver to insert new parameter Edit argo workflows to insert the name of the vmnodes crd as node selector

Controller

1c279f1

- added status in nodes crd

Frontend:

72adec9

- Insert selected node into argo workflows automatically