Support of kustomize remotebase / git submodule

[dwkop]

Issue

Currently kargo-render uses a flag --load-restrictor LoadRestrictionsRootOnly in case of kustomize, which actually blocks the usage of kustomize remote bases or kustomize remote components.

#kargo-render.yaml
configVersion: v1alpha1
branchConfigs:
- name: stage/dev
  appConfigs:
    demo:
      configManagement:
        kustomize:
          path: stages/dev
      outputPath: demo

#stages/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

## Remote Components
resources:
  - https://gh-enterprise.com/org/kargo-demo-kustomize.git//stages/dev-remote?ref=HEAD
#components:
#  - https://gh-enterprise.com/org/kargo-demo-kustomize.git//stages/dev-remote?ref=HEAD

Error

time="2024-01-03T11:35:09Z" level=error msg="error executing Promotion: error executing Git-based promotion mechanisms: error executing Kargo Render promotion mechanisms: error rendering manifests for git repo \"https://gh-enterprise.com/org/kargo-demo-service\" via Kargo Render: error rendering manifests: error executing cmd [/usr/local/bin/kargo-render render --repo https://gh-enterprise.com/org/kargo-demo-service --ref  --target-branch stage/dev --repo-username cortex-bot --output json --image artifacts.rbi.tech/docker/org/cortex-template-kustomize-application/nginx:2023.52.13]: Error: error pre-rendering manifests: error generating manifests using Argo CD repo server: `kustomize build /tmp/1395234519/repo/stages/dev --load-restrictor LoadRestrictionsRootOnly` failed exit status 1: Error: accumulating resources: accumulating resources from 'https://gh-enterprise.com/org/kargo-demo-kustomize.git//stages/dev-remote?ref=HEAD': MalformedYAMLError: yaml: line 175: mapping values are not allowed in this context in File: https://gh-enterprise.com/org/kargo-demo-kustomize.git//stages/dev-remote?ref=HEAD\n" freight=4fadca4b6a0c9f671f143df43e50272e9201083c namespace=kargo-demo-service promotion=dev.01hk7hp6sf8hpq1q7ektcrfjsf.4fadca4 stage=dev

Proposal

• Support the usage of kustomize remote base or remote components by using kustomize flag --load-restrictor LoadRestrictionsNone
• Support git submodules as an equivalent way to render manifests into another repo.

[krancour]

Hi @dan1el-k. I think #201 may have addressed this by, among other things, allowing users to pass whatever flags they want to the Argo CD repo server (which is actually invoked more as a library here).

Although merged, this just has not been released yet because it's a large breaking change that is not yet documented.

[dwkop]

Hi @dan1el-k. I think #201 may have addressed this by, among other things, allowing users to pass whatever flags they want to the Argo CD repo server (which is actually invoked more as a library here).

Although merged, this just has not been released yet because it's a large breaking change that is not yet documented.

Ahh, now it makes sense.
In our tests we were wondering that the CRDs are not really matching the docs as well as the argocd-schema.json + schema.json.

Looking forward to test the new release then :). Thanks !!

[krancour]

@dan1el-k I just cut v0.1.0-rc.34, but it's not integrated into "Kargo proper" yet.

[dwkop]

@dan1el-k I just cut v0.1.0-rc.34, but it's not integrated into "Kargo proper" yet.

@krancour, just saw two weeks ago kargo 0.3.2 were release. And also begin of January v0.1.0-rc.34 were merged to main.
akuity/kargo@de1afba
When do you plan including it into the release ?

[dwkop]

Status update on this issue

I did few more tests an found out that the actual functionality of using kustomize with remote bases partially works.

Example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://my-company-github/company-org/kargo-demo-service.git//deployment/base?timeout=120&ref=HEAD

Running it from local using the kargo-render docker image (ghcr.io/akuity/kargo-render:v0.1.0-rc.40)

This prompts me for another time for the Github credentials, even though I have them stored in my local keychain and don't need to specify them when using the image with a simple kustomize.yaml without remote base.

After putting in the credentials this 2nd time, kargo-render successfully commits to the stage branch

Running it via Kargo UI, so deploying a freight to this stage.

This fails with following error:

executing Git-based promotion mechanisms: error executing Kargo Render 
promotion mechanism: error rendering manifests via Kargo Render: error 
rendering manifests: error executing cmd [/usr/local/bin/kargo-render 
--target-branch stage/shared-dev1 --local-in-path 
/tmp/repo-3238425418/repo --local-out-path 
/tmp/repo-scrap-3164092133/rendered-manifests --repo-username cortex-bot
 --output json]: Error: error pre-rendering manifests: error generating 
manifests using Argo CD repo server: `kustomize build 
/tmp/repo-1119831774/repo/applications/cortex-okd-day2configuration/stages/dev1`
 failed exit status 1: Error: accumulating resources: accumulating 
resources from 
'https://my-company-github/my-org/kargo-demo-service.git//deployment/base?timeout=120&ref=HEAD':
 MalformedYAMLError: yaml: line 184: mapping values are not allowed in 
this context in File: 
https://my-company-github/my-org/kargo-demo-service.git//deployment/base?timeout=120&ref=HEAD
</div></div></div></div></div></div>Verifications
Live Manifest
	Date	Name	Approved by	Freight
Aug 9th 2024 08:56:25	shared-dev1.01j4tys7g502dzs1sp0esz0x5n.4be6d03	N/A	[4be6d03](https://kargo-cortex-kargo.apps.cortex-dev01.cortex.my-company-domain/project/mercury/freight/4be6d03a55be3a0120b47e9e1bcba4170dc0ad96)
Aug 7th 2024 15:11:52	shared-dev1.01j4pff9b2fbzh6x0qg18ksa6n.15192bb		[15192bb](https://kargo-cortex-kargo.apps.cortex-dev01.cortex.my-company-domain/project/mercury/freight/15192bb91a24b9b9282aa8f66eaf3dd6649c75c5)
Aug 7th 2024 15:11:49	shared-dev1.01j4pff660weaq6ygxrykj0jk0.15192bb	N/A	[15192bb](https://kargo-cortex-kargo.apps.cortex-dev01.cortex.my-company-domain/project/mercury/freight/15192bb91a24b9b9282aa8f66eaf3dd6649c75c5)
Aug 7th 2024 15:09:51	shared-dev1.01j4pfbjke0hrwd20ep4e37fwh.b6c69a8	N/A	[b6c69a8](https://kargo-cortex-kargo.apps.cortex-dev01.cortex.my-company-domain/project/mercury/freight/b6c69a85963492ccba8f54d70b6185c83bc8bdec)
Promotion Errored
error executing Git-based promotion mechanisms: error executing Kargo Render promotion mechanism: error rendering manifests via Kargo Render: error rendering manifests: error executing cmd [/usr/local/bin/kargo-render --target-branch stage/shared-dev1 --local-in-path /tmp/repo-3238425418/repo --local-out-path /tmp/repo-scrap-3164092133/rendered-manifests --repo-username cortex-bot --output json]: Error: error pre-rendering manifests: error generating manifests using Argo CD repo server: `kustomize build /tmp/repo-1119831774/repo/applications/cortex-okd-day2configuration/stages/dev1` failed exit status 1: Error: accumulating resources: accumulating resources from 'https://my-company-github/my-org/kargo-demo-service.git//deployment/base?timeout=120&ref=HEAD': MalformedYAMLError: yaml: line 184: mapping values are not allowed in this context in File: https://my-company-github/my-org/kargo-demo-service.git//deployment/base?timeout=120&ref=HEAD

Conclusion

So my assumption here is that the GH credentials are not properly handed through in case of a remote base scenario, which would explain why locally it would prompt for credentials but within the Kargo UI it just fails

[mihaigalos]

I got kustomize remote bases working by patching the kargo Deployments.
This is only a temporary solution until support lands in kargo render. That's because kargo queries the apiserver for the credentials instead of mounting it.

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: netrc-template
  namespace: my-kargo
data:
  .netrc: |
    machine my.domain
      login GITHUB_USERNAME
      password ${GITHUB_TOKEN}

deployment.yaml

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
 - cm.yaml

patches:

- target:
    kind: Deployment
    name: kargo-controller
    namespace: mercury-kargo
  patch: |
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: not-used
    spec:
      template:
        spec:
          volumes:
          - name: netrc-template
            configMap:
              name: netrc-template
              items:
                - key: .netrc
                  path: .netrc
          - name: netrc
            emptyDir: {}
          initContainers:
            - name: init-netrc
              image: busybox
              resources:
                limits:
                  cpu: 1
                  memory: 768Mi
                requests:
                  cpu: 100m
                  memory: 512Mi

              volumeMounts:
                - name: netrc-template
                  mountPath: /mnt/template
                - name: netrc
                  mountPath: /mnt/netrc
              env:
                - name: GITHUB_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: github-token
                      key: password
              command:
                - sh
                - -c
                - |
                  cp /mnt/template/.netrc /mnt/netrc/.netrc &&
                  sed -i 's/\${GITHUB_TOKEN}/'$GITHUB_TOKEN'/g' /mnt/netrc/.netrc
          containers:
            - name: controller
              volumeMounts:
                - name: netrc
                  readOnly: true
                  mountPath: /.netrc
                  subPath: .netrc
              env:
                - name: GITHUB_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: github-token
                      key: password
              command:
                - /usr/local/bin/kargo
                - controller