Add caching mechanism for CI tests in Jenkinsfile#4682
Open
Add caching mechanism for CI tests in Jenkinsfile#4682
Conversation
- Introduced optional caching for CI test results to skip stages that have already passed for the same commit and Docker image tag. - Added functions to manage cache paths and success markers. - Updated the `rocmtest` function to utilize caching logic, allowing tests to be skipped if previous results are available. - Enhanced Docker setup to handle cached builds and dependencies more effectively.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an optional filesystem-backed “success cache” to the Jenkins pipeline so reruns for the same commit (and same Docker image tag) can skip CI stages that already passed, reducing wasted CI time after transient failures.
Changes:
- Introduces cache helper functions and stage-level cache keys/marker paths in
Jenkinsfile. - Updates
rocmtestto optionally skip docker build/test when a prior success marker exists, with special handling to restore/stash cached.debartifacts for ONNX Runtime flows. - Adds
docs/Jenkins.mddocumenting environment variables, cache keying, and caveats.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
docs/Jenkins.md |
Documents how to enable/force-disable the CI success cache and how keys/markers are laid out. |
Jenkinsfile |
Implements success-marker caching, skip logic, and .deb restore/stash behavior for restart scenarios. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Jenkinsfile
Outdated
Comment on lines
+156
to
+160
| if (stageCacheId && ciTestCacheEnabled()) { | ||
| markerPath = successMarkerPath(gitCommit, cacheImageTag, stageCacheId) | ||
| env.MIGRAPHX_CI_MARKER_PATH = markerPath | ||
| env.MIGRAPHX_CI_DEB_CACHE = debCachePath(gitCommit, env.IMAGE_TAG) | ||
| skipTests = sh(returnStatus: true, script: 'test -f "$MIGRAPHX_CI_MARKER_PATH"') == 0 |
| if (skipTests) { | ||
| echo "Skipping tests for ${stageCacheId} (cached success, commit ${gitCommit})" | ||
| if (stageCacheId == 'hip_clang_release') { | ||
| sh 'mkdir -p build && cp "$MIGRAPHX_CI_DEB_CACHE"/*.deb build/' |
| env.MIGRAPHX_CI_DEB_CACHE = debCachePath(gitCommit, env.IMAGE_TAG) | ||
| sh 'mkdir -p "$MIGRAPHX_CI_DEB_CACHE" && cp build/*.deb "$MIGRAPHX_CI_DEB_CACHE"/' | ||
| } | ||
| sh 'mkdir -p "$(dirname "$MIGRAPHX_CI_MARKER_PATH")" && touch "$MIGRAPHX_CI_MARKER_PATH" && echo "BUILD_URL=${BUILD_URL}" >> "$MIGRAPHX_CI_MARKER_PATH"' |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #4682 +/- ##
===========================================
+ Coverage 92.24% 92.24% +0.01%
===========================================
Files 578 578
Lines 28309 28328 +19
===========================================
+ Hits 26112 26131 +19
Misses 2197 2197 🚀 New features to boost your workflow:
|
pfultz2
reviewed
Mar 18, 2026
| def variant = conf.get("variant", env.STAGE_NAME) | ||
| def setup = conf.get("setup", {}) | ||
| def stageCacheId = conf.get("stageCacheId", null) | ||
| def cacheImageTag = conf.get("cacheImageTag", env.IMAGE_TAG) |
Collaborator
There was a problem hiding this comment.
imageTag is already passed in. This is redundant.
pfultz2
reviewed
Mar 18, 2026
| def rocmtest = { Map conf = [:], Closure body -> | ||
| def variant = conf.get("variant", env.STAGE_NAME) | ||
| def setup = conf.get("setup", {}) | ||
| def stageCacheId = conf.get("stageCacheId", null) |
Collaborator
There was a problem hiding this comment.
This should come from variant variable.
pfultz2
reviewed
Mar 18, 2026
| echo "[MIGRAPHX_CI_SKIP_TRACE] setup ${variant}: cache is ON — checking marker on agent filesystem" | ||
| // #endregion | ||
| markerPath = successMarkerPath(gitCommit, cacheImageTag, stageCacheId) | ||
| env.MIGRAPHX_CI_MARKER_PATH = markerPath |
Collaborator
There was a problem hiding this comment.
These dont need to be environment variables.
pfultz2
reviewed
Mar 18, 2026
| echo "[MIGRAPHX_CI_SKIP_TRACE] build ${variant}: branch=${skipTests && stageCacheId == 'hip_clang_release' ? 'hip_skip_stash' : skipTests ? 'skip_no_docker' : 'full_run'}" | ||
| // #endregion | ||
| if (skipTests && stageCacheId == 'hip_clang_release') { | ||
| stash includes: 'build/*.deb', name: 'migraphx-package' |
Collaborator
There was a problem hiding this comment.
This is really bad, it relies on name of the stages and re-add in the code. If we need to replay the stashing then it would be better to have a flag we pass to rocmtest that does the stashing, something like rocmtest(stash: "build/*.deb")
pfultz2
reviewed
Mar 18, 2026
| } | ||
| } | ||
| if (stageCacheId && ciTestCacheEnabled()) { | ||
| env.MIGRAPHX_CI_MARKER_PATH = successMarkerPath(gitCommit, cacheImageTag, stageCacheId) |
Collaborator
There was a problem hiding this comment.
This doesnt need to be an environment variable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Each task in the job can take over 30 minutes to run. If a task fails for a random reason such as network timeout the entire the Jenkins job is restarted. This adds to the burden on the CI servers. The goal here is to skip a task if the Jenkins job was simply restarted using the same PR commit hash.
Technical Details
rocmtestfunction to utilize caching logic, allowing tests to be skipped if previous results are available.Changelog Category
Add a
CHANGELOG.mdentry for any option other thanNot Applicable