Skip to content

Update container infrastructure#99

Open
lispandfound wants to merge 9 commits intopegasusfrom
container_redux
Open

Update container infrastructure#99
lispandfound wants to merge 9 commits intopegasusfrom
container_redux

Conversation

@lispandfound
Copy link
Copy Markdown
Contributor

Updates the container build pipeline to reflect my recent usage of the infrastructure. The idea is to separate the two containers into:

  1. A bootstrap container that contains a pre-baked install of everything except EMOD3D which we can't publish.
  2. A minimal runner.def that takes the last step of installing EMOD3D.

Then a simple flow.cylc workflow stage can actually produce the runner.sif file immediately like so

[[build_container]]
script = """
git clone --depth 1 git@github.com:ucgmsim/EMOD3D.git
cp ${CYLC_WORKFLOW_RUN_DIR}/runner.def .
apptainer build runner.sif runner.def
mv runner.sif ${CYLC_WORKFLOW_RUN_DIR}
"""

the main advantage of this is that the entire workflow is self-contained and we do not have to shuttle around large runner.sif files.

Copilot AI review requested due to automatic review settings March 17, 2026 04:14
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the container infrastructure to optimize the build process for EMOD3D. By introducing a two-stage approach with a pre-built bootstrap container and a minimal runner definition, the changes aim to create a more efficient and self-contained workflow for generating the final runner.sif image, reducing dependencies and simplifying deployment.

Highlights

  • Container Infrastructure Separation: The container build process has been refactored into two distinct stages: a bootstrap container and a minimal runner definition, enhancing modularity and efficiency.
  • New Bootstrap Dockerfile: A new container/Dockerfile was added to create a base image containing pre-installed common dependencies like build tools, Python 3.13.1, and Rust, excluding EMOD3D.
  • Streamlined Runner Definition: The container/runner.def was updated to leverage the new bootstrap image, simplifying its role to primarily focus on the final EMOD3D installation and compilation.
  • Self-Contained Workflow: This new two-stage approach enables a more self-contained workflow for building runner.sif files directly within a Cylc workflow stage, eliminating the need to transfer large intermediate files.
Changelog
  • container/Dockerfile
    • Added a new Dockerfile to create a base image with core dependencies.
    • Configured Python 3.13.1 installation from source.
    • Included Rust installation and virtual environment setup.
  • container/runner.def
    • Modified to use the new earthquakesuc/workflow-bootstrap:latest image as its base.
    • Simplified the %post section to only compile EMOD3D, removing redundant dependency installations and data downloads.
    • Updated environment variables to reflect the new software paths from the bootstrap image.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/container.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the container build process by splitting it into a bootstrap Docker image and a minimal Apptainer runner definition. This is a good approach to streamline the workflow. My review focuses on improving the reproducibility of the builds and fixing a critical issue where compiled binaries were being deleted.

Key feedback points:

  • Pinning base image versions in both the Dockerfile and runner.def to ensure reproducible builds.
  • Correcting a critical bug in runner.def where compiled binaries were removed during a cleanup step.
  • A minor cleanup in an environment variable definition.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the container build approach to split “bootstrap” dependencies into a publishable Docker image, and keep the private EMOD3D install/build step in a lightweight Apptainer runner.def, enabling self-contained workflow runs without distributing large .sif artifacts.

Changes:

  • Refactors container/runner.def to build EMOD3D on top of a prebuilt bootstrap image.
  • Adds a new container/Dockerfile to build the publishable bootstrap Docker image (toolchain, Python, Rust, workflow install).
  • Replaces the GitHub Actions workflow to build/push the Docker image and run tests via docker run instead of building/uploading an Apptainer image.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

File Description
container/runner.def Switches Apptainer base to a prebuilt bootstrap image and builds EMOD3D from a provided source directory.
container/Dockerfile Introduces a bootstrap Docker image build (system deps, Python from source, Rust, venv, installs workflow).
.github/workflows/container.yml Updates CI to build/push the bootstrap Docker image to Docker Hub and run pytest in a container.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +19 to +22
export RUSTUP_HOME=/sw/rust
export CARGO_HOME=/sw/cargo
export PATH="/sw/venv/bin:/EMOD3D/tools:/sw/cargo/bin:/sw/python/bin:$PATH"
export VIRTUAL_ENV="/sw/venv"
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

# Setup Virtual Environment and Install via pip
ARG WORKFLOW_BRANCH=pegasus
docker run --rm \
-v ${{ github.workspace }}:/mnt \
${{ env.IMAGE_NAME }}:sha-${{ github.sha }} \
bash -c "pytest /mnt/tests"
lispandfound and others added 4 commits March 23, 2026 09:37
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants