feat: download kubelet with oras in network isolated windows cluster by jiashun0011 · Pull Request #8042 · Azure/AgentBaker

jiashun0011 · 2026-03-09T07:33:19Z

What this PR does / why we need it:

/kind feature

Which issue(s) this PR fixes:

Download kubelet with oras in network isolated windows cluster.

Copilot

Pull request overview

Adds support for downloading Windows kubelet binaries via ORAS when running in a network-isolated cluster context (using the bootstrap profile container registry), and extends test coverage to exercise the new path.

Changes:

Update Get-KubePackage to download kubelet binaries via ORAS when BootstrapProfileContainerRegistryServer is set; otherwise keep HTTP download behavior.
Add/extend Pester tests for ORAS vs HTTP selection behavior in Get-KubePackage.
Add a new Windows network-isolated e2e scenario intended to validate ORAS-based kubelet download behavior.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 6 comments.

File	Description
staging/cse/windows/kubeletfunc.ps1	Adds ORAS-based kubelet package download branch when bootstrap profile registry is configured.
staging/cse/windows/kubeletfunc.tests.ps1	Adds Pester contexts to cover ORAS vs HTTP download paths for kubelet package retrieval.
e2e/scenario_win_test.go	Adds a network-isolated Windows e2e test and alters Windows 2025 bootstrap mutator to use a custom CSE package URL.

You can also share your feedback on Copilot code review. Take the survey.

staging/cse/windows/kubeletfunc.tests.ps1

e2e/scenario_win_test.go

staging/cse/windows/kubeletfunc.ps1

Copilot · 2026-03-09T07:38:00Z

staging/cse/windows/kubeletfunc.tests.ps1

+            # The ORAS invocation line in Get-KubePackage evaluates $global:OrasPath
+            # and then invokes 'pull' as a separate command with the remaining arguments.
+            # Define and mock 'pull' so the invocation succeeds silently in tests.
+            function pull { }
+            Mock pull -MockWith {}


These tests are currently structured around the broken ORAS invocation semantics (mocking a pull function so $global:OrasPath pull ... doesn't error). That means the test suite would still pass even if the script never invokes oras.exe. After fixing Get-KubePackage to correctly run ORAS (e.g., via & $global:OrasPath pull ...), update the tests to validate the ORAS invocation through a mockable wrapper function (e.g., Invoke-OrasPull) instead of mocking pull.

staging/cse/windows/kubeletfunc.ps1

fseldow · 2026-03-09T09:06:58Z

staging-cse-windows.zip

forget to delete?

e2e/scenario_win_test.go

staging/cse/windows/kubeletfunc.ps1

fseldow · 2026-03-09T09:14:55Z

staging/cse/windows/kubeletfunc.ps1

+    # download kubelet binraries based on whether BootstrapProfileContainerRegistryServer is set.
+    if ($global:BootstrapProfileContainerRegistryServer) {
+        Logs-To-Event -TaskName "AKS.WindowsCSE.DownloadKubletBinariesWithOras" -TaskMessage "Start to download kubelet binaries with oras. KubeBinariesVersion: $global:KubeBinariesVersion, BootstrapProfileContainerRegistryServer: $global:BootstrapProfileContainerRegistryServer"
+        $global:OrasPath pull $global:BootstrapProfileContainerRegistryServer/aks/packages/kubernetes/windowszip:$global:KubeBinariesVersion --platform="windows/amd64" --registry-config=$global:OrasRegistryConfigFile --output $zipfile


& $global:ORAS_PATH

fseldow · 2026-03-09T09:19:23Z

staging/cse/windows/kubeletfunc.ps1

+    # download kubelet binraries based on whether BootstrapProfileContainerRegistryServer is set.
+    if ($global:BootstrapProfileContainerRegistryServer) {
+        Logs-To-Event -TaskName "AKS.WindowsCSE.DownloadKubletBinariesWithOras" -TaskMessage "Start to download kubelet binaries with oras. KubeBinariesVersion: $global:KubeBinariesVersion, BootstrapProfileContainerRegistryServer: $global:BootstrapProfileContainerRegistryServer"
+        $global:OrasPath pull $global:BootstrapProfileContainerRegistryServer/aks/packages/kubernetes/windowszip:$global:KubeBinariesVersion --platform="windows/amd64" --registry-config=$global:OrasRegistryConfigFile --output $zipfile


is it possible that windowszip version is different from k8sversion?
for example, k8s version is v1.30.100, package version might be v1.30.100-lts (with suffix)

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

You can also share your feedback on Copilot code review. Take the survey.

staging/cse/windows/networkisolatedclusterfunc.tests.ps1

+    Mock Set-ExitCode -MockWith { throw "Set-ExitCode:$($ExitCode):$ErrorMessage" }
+    Mock Get-Item -MockWith { return New-Object -TypeName PSObject -Property @{ FullName = $DestinationPath } }


staging/cse/windows/networkisolatedclusterfunc.tests.ps1

+    { DownloadFileWithOras -Reference $reference -DestinationPath $destPath -ExitCode 80 } | Should -Not -Throw
+
+    Assert-MockCalled -CommandName 'Write-Log' -ParameterFilter {
+      $message -like "*platform=windows/amd64*"
+    }


staging/cse/windows/kubeletfunc.ps1

+            Set-ExitCode -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL -ErrorMessage "DownloadFileWithOras function is not available. networkisolatedclusterfunc.ps1 may not be sourced."
+        }
+        Logs-To-Event -TaskName "AKS.WindowsCSE.DownloadKubletBinariesWithOras" -TaskMessage "Start to download kubelet binaries with oras. KubeBinariesVersion: $global:KubeBinariesVersion, BootstrapProfileContainerRegistryServer: $global:BootstrapProfileContainerRegistryServer"
+        $orasReference = "$($global:BootstrapProfileContainerRegistryServer)/aks/packages/kubernetes/windowszip:v$($global:KubeBinariesVersion)"


staging/cse/windows/kubeletfunc.ps1

+        for ($i = 0; $i -le $maxRetries; $i++) {
+            try {
+                DownloadFileWithOras -Reference $orasReference -DestinationPath $zipfile -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL
+                break
+            } catch {
+                Write-Log "Attempt ${i}: oras pull failed for $orasReference. Error: $_"
+                if ($i -eq $maxRetries) {
+                    Set-ExitCode -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL -ErrorMessage "Exhausted retries for oras pull $orasReference. Error: $_"
+                }


staging/cse/windows/kubeletfunc.tests.ps1

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

You can also share your feedback on Copilot code review. Take the survey.

staging/cse/windows/networkisolatedclusterfunc.ps1

+    $downloadTimer = [System.Diagnostics.Stopwatch]::StartNew()
+    $orasArgs = @(
+        "pull",
+        $Reference,
+        "--platform=$Platform",
+        "--registry-config=$($global:OrasRegistryConfigFile)",
+        "--output", $tempDir
+    )
+    & $global:OrasPath @orasArgs
+    if ($LASTEXITCODE -ne 0) {
+        $downloadTimer.Stop()
+        Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue
+        Set-ExitCode -ExitCode $ExitCode -ErrorMessage "oras pull failed with exit code $LASTEXITCODE for $Reference"
+    }
+    $downloadTimer.Stop()
+    $elapsedMs = $downloadTimer.ElapsedMilliseconds
+
+    # Find the downloaded file in the temp directory and move it to the desired path
+    $downloadedFile = Get-ChildItem -Path $tempDir -File | Select-Object -First 1
+    if (-not $downloadedFile) {
+        Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue
+        Set-ExitCode -ExitCode $ExitCode -ErrorMessage "oras pull succeeded but no file found in temp directory for $Reference"
+    }
+
+    Write-Log "Downloaded file name: $($downloadedFile.Name) (size: $($downloadedFile.Length) bytes) in temp directory $tempDir"
+
+    # Ensure the destination parent directory exists
+    $destDir = [System.IO.Path]::GetDirectoryName($DestinationPath)
+    if ($destDir -and -not (Test-Path $destDir)) {
+        New-Item -ItemType Directory -Path $destDir -Force | Out-Null
+    }
+
+    # Remove existing destination if present, then move the downloaded file
+    if (Test-Path $DestinationPath) {
+        Remove-Item -Path $DestinationPath -Force
+    }
+    Write-Log "Moving $($downloadedFile.FullName) to $DestinationPath"
+    Move-Item -Path $downloadedFile.FullName -Destination $DestinationPath -Force
+    Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue
+
+    if ($global:AppInsightsClient -ne $null) {
+        $event = New-Object "Microsoft.ApplicationInsights.DataContracts.EventTelemetry"
+        $event.Name = "FileDownload"
+        $event.Properties["FileName"] = $Reference
+        $event.Properties["Method"] = "oras"
+        $event.Metrics["DurationMs"] = $elapsedMs
+        $global:AppInsightsClient.TrackEvent($event)
+    }
+
+    Write-Log "Downloaded $Reference to $DestinationPath via oras in $elapsedMs ms"
+    Get-Item $DestinationPath -ErrorAction Continue | Format-List | Out-String | Write-Log


e2e/scenario_win_test.go

+				nbc.ContainerService.Properties.SecurityProfile = &datamodel.SecurityProfile{
+					PrivateEgress: &datamodel.PrivateEgress{
+						Enabled:                 true,
+						ContainerRegistryServer: fmt.Sprintf("%s.azurecr.io/aks-managed-repository", config.PrivateACRName(config.Config.DefaultLocation)),


staging/cse/windows/kubeletfunc.tests.ps1

+        It "Should call DownloadFileWithOras with correct reference when BootstrapProfileContainerRegistryServer is set" {
+            Get-KubePackage -KubeBinariesSASURL 'https://xxx.blob.core.windows.net/kubernetes/v1.29.2/windowszip/v1.29.2-1int.zip'
+            Assert-MockCalled -CommandName 'DownloadFileWithOras' -Exactly -Times 1 -ParameterFilter {
+                $Reference -eq 'myregistry.azurecr.io/aks/packages/kubernetes/windowszip:v1.29.2' -and
+                $DestinationPath -eq 'c:\k.zip' -and
+                $ExitCode -eq $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL
+            }


e2e/scenario_win_test.go

staging/cse/windows/kubeletfunc.ps1

fseldow · 2026-03-12T03:35:30Z

staging/cse/windows/kubeletfunc.ps1

+        }
+        Logs-To-Event -TaskName "AKS.WindowsCSE.DownloadKubletBinariesWithOras" -TaskMessage "Start to download kubelet binaries with oras. KubeBinariesVersion: $global:KubeBinariesVersion, BootstrapProfileContainerRegistryServer: $global:BootstrapProfileContainerRegistryServer"
+        $orasReference = "$($global:BootstrapProfileContainerRegistryServer)/aks/packages/kubernetes/windowszip:v$($global:KubeBinariesVersion)"
+        $maxRetries = 5


can we share same retry with DownloadFileOverHttp to minimize the code change?

for example put if bootstrapProfileRegistry at line 225

I feel keeping them separate makes the logic more straightforward.

Although both paths implement retries, the actual download mechanisms are different (oras vs HTTP), and the error handling is slightly different as well. Combining them into a shared retry loop may make the flow harder to read.

Given this code runs in CSE during node bootstrap, I’d prefer keeping it explicit and easy to follow.

Per discussion, I agree keeping the PR change smaller, especially avoiding too much change in existing logic function structure, is a good point.

I’m okay with this direction.

Using two separate if blocks also seems fine to me. It keeps the original HTTP path mostly untouched, while isolating the new oras path into its own helper, so the diff is smaller and easier to review. I also agree not using else gives a bit more flexibility if we need to extend the condition handling later.

I’ll update it in this way.

staging/cse/windows/networkisolatedclusterfunc.ps1

jiashun0011 · 2026-03-16T02:43:46Z

staging-cse-windows.zip

forget to delete?

deleted

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

You can also share your feedback on Copilot code review. Take the survey.

staging/cse/windows/kubeletfunc.ps1

+            try {
+                DownloadFileWithOras -Reference $orasReference -DestinationPath $zipfile -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL
+                break
+            } catch {
+                Write-Log "Attempt ${i}: oras pull failed for $orasReference. Error: $_"
+                if ($i -eq $maxRetries) {
+                    Set-ExitCode -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL -ErrorMessage "Exhausted retries for oras pull $orasReference. Error: $_"
+                }
+                Start-Sleep -Seconds $retryDelaySeconds


staging/cse/windows/networkisolatedclusterfunc.ps1

+    $downloadTimer = [System.Diagnostics.Stopwatch]::StartNew()
+    $orasArgs = @(
+        "pull",
+        $Reference,
+        "--platform=$Platform",
+        "--registry-config=$($global:OrasRegistryConfigFile)",
+        "--output", $tempDir
+    )
+    & $global:OrasPath @orasArgs
+    if ($LASTEXITCODE -ne 0) {
+        $downloadTimer.Stop()
+        Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue
+        Set-ExitCode -ExitCode $ExitCode -ErrorMessage "oras pull failed with exit code $LASTEXITCODE for $Reference"
+    }
+    $downloadTimer.Stop()
+    $elapsedMs = $downloadTimer.ElapsedMilliseconds
+
+    # Find the downloaded file in the temp directory and move it to the desired path
+    $downloadedFile = Get-ChildItem -Path $tempDir -File | Select-Object -First 1
+    if (-not $downloadedFile) {
+        Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue
+        Set-ExitCode -ExitCode $ExitCode -ErrorMessage "oras pull succeeded but no file found in temp directory for $Reference"
+    }
+
+    Write-Log "Downloaded file name: $($downloadedFile.Name) (size: $($downloadedFile.Length) bytes) in temp directory $tempDir"
+
+    # Ensure the destination parent directory exists
+    $destDir = [System.IO.Path]::GetDirectoryName($DestinationPath)
+    if ($destDir -and -not (Test-Path $destDir)) {
+        New-Item -ItemType Directory -Path $destDir -Force | Out-Null
+    }
+
+    # Remove existing destination if present, then move the downloaded file
+    if (Test-Path $DestinationPath) {
+        Remove-Item -Path $DestinationPath -Force
+    }
+    Write-Log "Moving $($downloadedFile.FullName) to $DestinationPath"
+    Move-Item -Path $downloadedFile.FullName -Destination $DestinationPath -Force
+    Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue


staging/cse/windows/networkisolatedclusterfunc.tests.ps1

+    { DownloadFileWithOras -Reference $reference -DestinationPath $destPath -ExitCode 80 } | Should -Not -Throw
+
+    Assert-MockCalled -CommandName 'Write-Log' -ParameterFilter {
+      $message -like "*platform=windows/amd64*"
+    }


staging/cse/windows/networkisolatedclusterfunc.tests.ps1

+  It "should call oras with correct arguments on success" {
+    $reference = "myregistry.azurecr.io/aks/packages/kubernetes/windowszip:1.29.2"
+    $destPath = "c:\k.zip"
+
+    $global:OrasPath = "Write-Output"
+    { DownloadFileWithOras -Reference $reference -DestinationPath $destPath -ExitCode 80 } | Should -Not -Throw
+  }


azure-pipelines · 2026-03-16T08:57:46Z

There was an error handling pipeline event fa0590be-b5f0-479e-812d-33890e8827ec.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

You can also share your feedback on Copilot code review. Take the survey.

staging/cse/windows/kubeletfunc.ps1

+        $maxRetries = 5
+        $retryDelaySeconds = 10
+        for ($i = 0; $i -le $maxRetries; $i++) {
+            try {
+                DownloadFileWithOras -Reference $orasReference -DestinationPath $zipfile -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL
+                break
+            } catch {
+                Write-Log "Attempt ${i}: oras pull failed for $orasReference. Error: $_"
+                if ($i -eq $maxRetries) {
+                    Set-ExitCode -ExitCode $global:WINDOWS_CSE_ERROR_ORAS_PULL_WINDOWSZIP_FAIL -ErrorMessage "Exhausted retries for oras pull $orasReference. Error: $_"
+                }
+                Start-Sleep -Seconds $retryDelaySeconds
+            }
        }


staging/cse/windows/kubeletfunc.ps1

-        if ($?) {
-            break
+
+    # download kubelet binraries via http if BootstrapProfileContainerRegistryServer is not set


e2e/scenario_win_test.go

+		Tags: Tags{
+			NetworkIsolated: true,
+			NonAnonymousACR: false,
+		},


staging/cse/windows/networkisolatedclusterfunc.tests.ps1

  }
 }

+<<<<<<< HEAD


staging/cse/windows/networkisolatedclusterfunc.tests.ps1

+  BeforeEach {
+    $global:OrasPath = "C:\aks-tools\oras\oras.exe"
+    $global:OrasRegistryConfigFile = "C:\oras-config.json"
+    $global:AppInsightsClient = $null


Copilot AI review requested due to automatic review settings March 9, 2026 07:33

jiashun0011 requested review from AbelHu, Devinwong, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, juan-lee, junjiezhang1997, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, sulixu, surajssd, timmy-wright and zachary-bailey as code owners March 9, 2026 07:33

jiashun0011 temporarily deployed to test March 9, 2026 07:33 — with GitHub Actions Inactive

Copilot started reviewing on behalf of jiashun0011 March 9, 2026 07:33 View session

Copilot AI reviewed Mar 9, 2026

View reviewed changes

fseldow reviewed Mar 9, 2026

View reviewed changes

jiashun0011 force-pushed the jiashunliu/ni-oras-cache-1 branch from 76afb46 to 4450dfb Compare March 12, 2026 00:25

jiashun0011 temporarily deployed to test March 12, 2026 00:25 — with GitHub Actions Inactive

jiashun0011 force-pushed the jiashunliu/ni-oras-cache-1 branch from 4450dfb to bac8e29 Compare March 12, 2026 00:40

Copilot AI review requested due to automatic review settings March 12, 2026 00:40

jiashun0011 temporarily deployed to test March 12, 2026 00:41 — with GitHub Actions Inactive

Copilot started reviewing on behalf of jiashun0011 March 12, 2026 00:41 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 12, 2026 00:45

jiashun0011 temporarily deployed to test March 12, 2026 00:45 — with GitHub Actions Inactive

Copilot started reviewing on behalf of jiashun0011 March 12, 2026 00:46 View session

jiashun0011 temporarily deployed to test March 12, 2026 00:48 — with GitHub Actions Inactive

Copilot AI reviewed Mar 12, 2026

View reviewed changes

fseldow reviewed Mar 12, 2026

View reviewed changes

jiashun0011 force-pushed the jiashunliu/ni-oras-cache-1 branch from 87651bd to bf626ee Compare March 16, 2026 03:48

jiashun0011 temporarily deployed to test March 16, 2026 03:48 — with GitHub Actions Inactive

jiashun0011 force-pushed the jiashunliu/ni-oras-cache-1 branch from bf626ee to 663af34 Compare March 16, 2026 03:58

Copilot AI review requested due to automatic review settings March 16, 2026 03:58

jiashun0011 temporarily deployed to test March 16, 2026 03:59 — with GitHub Actions Inactive

Copilot started reviewing on behalf of jiashun0011 March 16, 2026 03:59 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

jiashun0011 force-pushed the jiashunliu/ni-oras-cache-1 branch from 663af34 to 411dd66 Compare March 16, 2026 08:57

jiashun0011 temporarily deployed to test March 16, 2026 08:57 — with GitHub Actions Inactive

feat: download kubelet with oras in network isolated windows cluster

af24e0a

Copilot AI review requested due to automatic review settings March 16, 2026 12:12

jiashun0011 force-pushed the jiashunliu/ni-oras-cache-1 branch from 411dd66 to af24e0a Compare March 16, 2026 12:12

jiashun0011 deployed to test March 16, 2026 12:12 — with GitHub Actions Active

Copilot started reviewing on behalf of jiashun0011 March 16, 2026 12:13 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

		Mock Set-ExitCode -MockWith { throw "Set-ExitCode:$($ExitCode):$ErrorMessage" }
		Mock Get-Item -MockWith { return New-Object -TypeName PSObject -Property @{ FullName = $DestinationPath } }

Conversation

jiashun0011 commented Mar 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fseldow commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fseldow Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

fseldow Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

Uh oh!

Uh oh!

fseldow Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

fseldow Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

jiashun0011 Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

jiashun0011 Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jiashun0011 commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

azure-pipelines bot commented Mar 16, 2026

Uh oh!

Copilot AI left a comment