Skip to content

workflows: fix release process #11637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
edsiper wants to merge 4 commits into
base: master
Choose a base branch
from
+53 −11
Open

workflows: fix release process #11637

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cd875a9
dockerfiles: windows: implement retry logic
edsiper Mar 31, 2026
fc7069b
workflows: staging: sync repo changes in order
edsiper Mar 31, 2026
855ac26
dockerfiles: windows: add -PassThru
edsiper Mar 31, 2026
ef57d8d
workflows: add packaging script
edsiper Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/staging-release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,9 @@ jobs:

- name: Sync to release bucket on S3
run: |
aws s3 sync "packaging/releases/$CODENAME" "s3://${{ secrets.AWS_S3_BUCKET_RELEASE }}/$CODENAME" --delete --follow-symlinks --no-progress
./packaging/sync-apt-repo-to-s3.sh \
"packaging/releases/$CODENAME" \
"s3://${{ secrets.AWS_S3_BUCKET_RELEASE }}/$CODENAME"
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
Expand Down
26 changes: 16 additions & 10 deletions dockerfiles/Dockerfile.windows
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,35 +32,41 @@ ENV MSVS_BUILD_TOOLS_VERSION="$MSVS_VERSION" `

RUN $msvs_build_tools_dist_name=\"vs_buildtools.exe\"; `
$msvs_build_tools_dist=\"${env:TMP}\${msvs_build_tools_dist_name}\"; `
$msvs_build_tools_channel=\"C:\local\VisualStudio.chman\"; `
$msvs_build_tools_dist_url=\"${env:MSVS_BUILD_TOOLS_DOWNLOAD_URL}/${env:MSVS_BUILD_TOOLS_VERSION}/release/${msvs_build_tools_dist_name}\"; `
$msvs_build_tools_channel_url=\"${env:MSVS_BUILD_TOOLS_DOWNLOAD_URL}/${env:MSVS_BUILD_TOOLS_VERSION}/release/channel\"; `
Write-Host \"Downloading Visual Studio Build Tools...\"; `
Write-Host \"${msvs_build_tools_dist_url} -> ${msvs_build_tools_dist}\"; `
Write-Host \"${msvs_build_tools_channel_url} -> ${msvs_build_tools_channel}\"; `
Invoke-WebRequest -OutFile \"${msvs_build_tools_dist}\" \"${msvs_build_tools_dist_url}\"; `
Invoke-WebRequest -OutFile \"${msvs_build_tools_channel}\" \"${msvs_build_tools_channel_url}\"; `
$max_retries = 5; `
for ($retry = 1; $retry -le $max_retries; $retry++) { `
try { `
Invoke-WebRequest -OutFile \"${msvs_build_tools_dist}\" \"${msvs_build_tools_dist_url}\"; `
break; `
} catch { `
if ($retry -eq $max_retries) { `
throw `
}; `
Write-Host \"Download attempt $retry failed: $($_.Exception.Message)\"; `
Start-Sleep -Seconds (10 * $retry); `
} `
}; `
Write-Host \"Installing Visual Studio Build Tools into ${env:MSVS_HOME}...\"; `
$p = Start-Process \"${msvs_build_tools_dist}\" `
-ArgumentList '--quiet ', '--wait ', '--norestart ', '--nocache', `
\"--installPath ${env:MSVS_HOME}\", `
\"--channelUri ${msvs_build_tools_channel}\", `
\"--installChannelUri ${msvs_build_tools_channel}\", `
'--add Microsoft.VisualStudio.Workload.VCTools', `
'--add Microsoft.VisualStudio.Workload.MSBuildTools', `
'--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64', `
'--add Microsoft.VisualStudio.Component.VC.CoreBuildTools', `
'--add Microsoft.VisualStudio.Component.VC.MSVC.143', `
'--add Microsoft.VisualStudio.Component.Windows10SDK.19041' `
-NoNewWindow -Wait; `
-NoNewWindow -Wait -PassThru; `
Write-Host \"Visual Studio Build Tools installer exit code: $($p.ExitCode)\"; `
if ($p.ExitCode -ne 0 -and $p.ExitCode -ne 3010) { `
throw \"Visual Studio Build Tools installer failed with exit code $($p.ExitCode)\" `
}; `
if (-not (Test-Path \"${env:MSVS_HOME}\VC\Auxiliary\Build\vcvars64.bat\")) { `
throw \"Visual Studio Build Tools installation is incomplete: ${env:MSVS_HOME}\VC\Auxiliary\Build\vcvars64.bat not found\" `
}; `
Remove-Item -Force \"${msvs_build_tools_dist}\"; `
Remove-Item -Path \"${msvs_build_tools_channel}\" -Force;
Remove-Item -Force \"${msvs_build_tools_dist}\";

ENV CMAKE_HOME="C:\cmake"
ARG CMAKE_VERSION="3.31.6"
Expand Down
34 changes: 34 additions & 0 deletions packaging/sync-apt-repo-to-s3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash
set -euo pipefail

# Upload an APT repo in two phases so Release metadata only becomes visible
# after the referenced package indexes and pool files are already uploaded.

SOURCE_DIR=${1:?Usage: sync-apt-repo-to-s3.sh <source-dir> <s3-destination>}
DESTINATION=${2:?Usage: sync-apt-repo-to-s3.sh <source-dir> <s3-destination>}

if [[ ! -d "$SOURCE_DIR" ]]; then
echo "ERROR: missing source dir: $SOURCE_DIR"
exit 1
fi

SOURCE_DIR=$(realpath "$SOURCE_DIR")

aws s3 sync "$SOURCE_DIR" "$DESTINATION" \
--delete \
--follow-symlinks \
--no-progress \
--exclude "dists/*/InRelease" \
--exclude "dists/*/Release" \
--exclude "dists/*/Release.gpg"
Comment on lines +21 to +23
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

In AWS CLI s3 sync, when --deleteis used with--exclude, are excluded objects skipped from deletion?

💡 Result:

Yes, excluded objects are skipped from deletion when using aws s3 sync with both --delete and --exclude.

Citations:

🏁 Script executed:

cat -n packaging/sync-apt-repo-to-s3.sh

Repository: fluent/fluent-bit

Length of output: 1387

Excluded metadata won't be deleted; removed suites leave stale Release* objects in S3.

Phase 1 excludes Release* files from deletion, and Phase 2 only uploads existing metadata files without deleting orphaned ones. If a distribution suite is removed from the source, its metadata remains in S3 indefinitely.

Suggested fix (metadata sync with include filters + delete) 
-while IFS= read -r metadata_file; do
-    relative_path=${metadata_file#"$SOURCE_DIR"/}
-    aws s3 cp "$metadata_file" "$DESTINATION/$relative_path" --no-progress
-done < <(find "$DIST_DIR" -type f \( -name "InRelease" -o -name "Release" -o -name "Release.gpg" \) | sort)
+aws s3 sync "$DIST_DIR" "$DESTINATION/dists" \
+    --delete \
+    --no-progress \
+    --exclude "*" \
+    --include "*/InRelease" \
+    --include "*/Release" \
+    --include "*/Release.gpg"
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@packaging/sync-apt-repo-to-s3.sh` around lines 21 - 23, The script currently
excludes "dists/*/InRelease", "dists/*/Release", and "dists/*/Release.gpg" from
deletion which leaves stale Release* objects in S3; update the sync logic so
metadata is synced with include filters and deleted when missing: remove or stop
excluding those three patterns from the deletion pass, and add a cleanup
sync/pass that uses include patterns like "dists/*/Release*" (and the specific
file names) with --delete (or equivalent delete-enabled command) so orphaned
Release*, InRelease and Release.gpg objects are removed; search for the
rsync/aws s3 sync invocation(s) around the existing exclude lines and modify
them to perform an include+--delete cleanup for the dists/*/Release* metadata.


DIST_DIR="$SOURCE_DIR/dists"
if [[ ! -d "$DIST_DIR" ]]; then
echo "ERROR: missing dists dir in source: $DIST_DIR"
exit 1
fi
Comment on lines +17 to +29
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Move repository preflight checks before the destructive sync step.

Line 17 runs a --delete sync before Line 26 validates that dists exists. If that validation fails, the destination may already be partially mutated.

Suggested fix 
 SOURCE_DIR=$(realpath "$SOURCE_DIR")
 
+DIST_DIR="$SOURCE_DIR/dists"
+if [[ ! -d "$DIST_DIR" ]]; then
+    echo "ERROR: missing dists dir in source: $DIST_DIR"
+    exit 1
+fi
+
+mapfile -t METADATA_FILES < <(
+    find "$DIST_DIR" -type f \( -name "InRelease" -o -name "Release" -o -name "Release.gpg" \) | sort
+)
+if [[ ${`#METADATA_FILES`[@]} -eq 0 ]]; then
+    echo "ERROR: no APT metadata files found under: $DIST_DIR"
+    exit 1
+fi
+
 aws s3 sync "$SOURCE_DIR" "$DESTINATION" \
     --delete \
     --follow-symlinks \
     --no-progress \
     --exclude "dists/*/InRelease" \
     --exclude "dists/*/Release" \
     --exclude "dists/*/Release.gpg"
-
-DIST_DIR="$SOURCE_DIR/dists"
-if [[ ! -d "$DIST_DIR" ]]; then
-    echo "ERROR: missing dists dir in source: $DIST_DIR"
-    exit 1
-fi
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@packaging/sync-apt-repo-to-s3.sh` around lines 17 - 29, Move the repository
preflight checks to run before the destructive aws s3 sync call: validate that
DIST_DIR="$SOURCE_DIR/dists" exists and any other necessary preconditions (using
the existing DIST_DIR and SOURCE_DIR variables) and exit non-zero if they fail,
then run the aws s3 sync command (which currently uses --delete,
--follow-symlinks, --no-progress and excludes) only after those checks pass;
update the script so the check block that echoes "ERROR: missing dists dir in
source: $DIST_DIR" and exits is located before the aws s3 sync invocation.


while IFS= read -r metadata_file; do
relative_path=${metadata_file#"$SOURCE_DIR"/}
aws s3 cp "$metadata_file" "$DESTINATION/$relative_path" --no-progress
done < <(find "$DIST_DIR" -type f \( -name "InRelease" -o -name "Release" -o -name "Release.gpg" \) | sort)
Loading