Skip to content

DEVELOPER.md

Latest commit

 

History

History
71 lines (37 loc) · 6.77 KB

DEVELOPER.md

File metadata and controls

71 lines (37 loc) · 6.77 KB

Developer notes

Mirroring to GitHub

The Rapthor repository is mirrored to GitHub for historical reasons. This section contains information on how to properly set up repository mirroring. The "standard" way (Settings | Repository | Mirroring repositories) doesn't work if you want to use an SSH key pair generated by GitLab. For some stupid reason GitLab uses deprecated SHA-1 for the generated SSH key pair, which is refused by GitHub for security reasons. And there is no way to tell GitLab to generate a different type of SSH key pair. The recommended way, described below, uses a Personal Access Token.

On the GitHub side

On GitHub, go to your account (click your avatar in the top right) and select Settings. In the left-pane of the newly opened page select Developer settings (at the bottom), and then Personal access tokens | Fine-grained tokens. Or use this direct link: https://github.com/settings/personal-access-tokens

Click on Generate new token. This will open a new page. A number of fields are mandatory, like the Token name, and Expiration (choose wisely). Under Repository access, click on Only select repositories and then select the desired repository from the pull-down menu Select repositories. Under Permissions, click on Repository permissions, scroll down to Contents, click the Access button on the right and select Read and write access level. Scroll all the way to the bottom of the page and click the green Generate token button. Copy the personal access token now, because you won't be able to see it again!

On the GitLab side

On GitLab, go to your project page (e.g. https://git.astron.nl/RD/rapthor), and then select Settings | Repository. Select Mirroring repostories and click on the Add new button. This will open a Add new mirror repository pane. In the Git repository URL field, fill in the name of the mirror repository on GitHub (e.g. https://github.com/darafferty/rapthor.git). As Mirror direction select Push. As Authentication method select Username and Password. For Username, fill in your GitHub username; for Password, paste the copied personal access token. You probably want to mirror all branches, so there's no need to change anything here. Scroll down a bit until you see a blue Mirror repository button, and click on it.

Try if mirroring works by clicking the Update now button (the two curved arrows). The page should now show a banner telling you that the remote repository is being updated. This will take a while.

References

Creating a release

Creating a Rapthor release is a bit more involved than simply pressing a button in the GitLab GUI. The main reason is that all the CWL files that contain a dockerPull line need to be updated on-the-fly to contain the proper reference to the release. This is taken care of by the CI/CD pipeline, but it is good to understand how this pipeline works under the hood. First, let's properly configure the GitLab project.

Configure the project in the GitLab GUI

The CI/CD pipelines needs to have push rights to the Git repository. We use an access token to grant these rights. The following preparations need to be done only once (though they have to be repeated when the access token expires):

  • Create a project access token (Settings > Access tokens), and grant it the Maintainer role and the write_repository scope. Copy the token directly after you've created it, because there is no way to retrieve it later on.
  • Create a CI/CD variable (Settings > CI/CD > Variables) named GIT_PUSH_TOKEN, and set its value to the access token that you just created (and copied!). Mark the variable as masked and protected. This variable will be used when a release is created (see .gitlab-ci.yml file).
  • Create CI/CD variables (Settings > CI/CD > Variables) named DH_REGISTRY_USER and DH_REGISTRY_PASSWORD, containing a valid username/password for a DockerHub account. Mark both variables as masked and mark the password variable as protected as well. These variables are used when publishing the generated docker image to DockerHub.
  • Mark release branches as protected (see below what the CI/CD pipeline considers a release). Go to Settings > Repository > Protected branches and click on the button Add protected branch. In the Branches drop-down menu enter releases/* (wildcard release); in the Allowed to merge drop-down menu select No one; and in the Allowed to push and merge drop-down menu select Developers + Maintainers. Disable Require approval from code owners. Next click the blue Protect button.

NOTE: The name of the access token will be used in the Created by column in the Pipelines view for pipelines that were triggered as a result of an automatic git push action.

What does the CI/CD pipeline consider a release?

The CI/CD pipeline will consider every branch whose name starts with releases/ a release branch. To create a release named <my-release>, you need to manually create a branch named releases/<my-release>.

NOTE:<my-release> must be a valid name according to the PEP-440 versioning scheme. For example, v5.0rc1 is a valid name, whereas ldv_404 is not. However, you can add a local identifier to the name, e.g. v5.0rc1+ldv_404.

What does the CI/CD pipeline do with a release branch?

When the CI/CD pipeline is run on a release branch, it will, in addition to a normal CI/CD pipeline run, do the following:

  • update all the CWL files containing a dockerPull line with the correct image tag, and commit the changes
  • tag the release using <my-release> as tag
  • deploy the docker image to Docker Hub
  • if <my-release> is a versioned release:1
    • undo the changes to the CWL files, and commit
    • merge the changes into the default branch

The rationale behind the two extra steps for a versioned release is that we want the git tag to be visible on the default branch, so that git describe --tags will return the desired result.

NOTE: If a tag <my-release> already exists, the pipeline will bail out with an error. In order to let it run successfully, the tag <my-release> needs to be removed manually in the GitLab GUI. This is to avoid that an existing release will be clobbered!

Creating the release in the GitLab GUI

Once the CI/CD pipeline has run, and the new release has been tagged, a new release can be created in the GitLab GUI. Navigate to Deploy > Releases, click on the Create new release button, and proceed as you would normally do when creating a new release.

Footnotes

  1. A versioned release is a release whose name starts with a V or v. ↩