diff --git a/README.md b/README.md index b055799..c4bf040 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ -# BioImage +# BioShell -BioImage is a pre‑configured, cloud‑ready bioinformatics environment designed to make command‑line–based biological data analysis easy to deploy, use, and scale. The primary goal of BioImage is to provide researchers and training providers with a flexible and cost‑effective platform for running bioinformatics workflows without the overhead of complex system setup or infrastructure management. +BioShell is a pre‑configured, cloud‑ready bioinformatics environment designed to make command‑line–based biological data analysis easy to deploy, use, and scale. The primary goal of BioShell is to provide researchers and training providers with a flexible and cost‑effective platform for running bioinformatics workflows without the overhead of complex system setup or infrastructure management. -BioImage enables users to focus on analysis and training, rather than on system configuration and resource provisioning. +BioShell enables users to focus on analysis and training, rather than on system configuration and resource provisioning. -### How BioImage Is Built -BioImage is built using Packer and Ansible, and is designed to run on OpenStack‑based cloud infrastructure. This approach enables reproducible image builds, consistent configuration, and rapid deployment. The result is a ready‑to‑use virtual machine (VM) image that includes commonly used bioinformatics tools, dependencies, and sensible defaults for CLI‑based analysis. +### How BioShell Is Built +BioShell is built using Packer and Ansible, and is designed to run on OpenStack‑based cloud infrastructure. This approach enables reproducible image builds, consistent configuration, and rapid deployment. The result is a ready‑to‑use virtual machine (VM) image that includes commonly used bioinformatics tools, dependencies, and sensible defaults for CLI‑based analysis. -This repository contains configuration and automation for building a custom Ubuntu‑based BioImage and provisioning instances on OpenStack‑compatible cloud environments. +This repository contains configuration and automation for building a custom Ubuntu‑based BioShell and provisioning instances on OpenStack‑compatible cloud environments. ---------------------------- ## Table of Contents @@ -19,7 +19,7 @@ This repository contains configuration and automation for building a custom Ubun * [Setup](#setup) * [Activation](#activation) * [Build Image](#build-image) -* [Using the BioImage](#Using-the-BioImage) +* [Using BioShell](#Using-BioShell) ## Spinning up a VM in OpenStack @@ -30,7 +30,7 @@ The control host (or head VM) is the machine from which you manage and deploy re - Holding your OpenStack credentials (RC file) - Running OpenStack CLI commands - Orchestrating or launching additional compute resources -- Acting as the control point for bioimage installation and management +- Acting as the control point for BioShell installation and management The head VM does not need to run workloads itself. It can be a VM running inside your OpenStack project (recommended) or a local machine (laptop or workstation) with network access to OpenStack APIs @@ -50,13 +50,13 @@ The following steps describe how to create a head VM using the generic OpenStack 1. Log in to your OpenStack dashboard and navigate to `Compute → Instance` 2. Select `launch instance` 3. The Launch Instance dialog will appear, open on the Details tab -4. Enter an instance name, choose a something descriptive (e.g. bioimage-head, control-host), and add a description +4. Enter an instance name, choose a something descriptive (e.g. bioshell-head, control-host), and add a description ### Select the Source Image (Operating System) 1. In the source tab, Select Boot Source: Image 2. Choose a supported Ubuntu image (eg. Ubuntu 22.04) -3. If the bioimage has been previously built you should be able to select bioimage at this step. +3. If the BioShell image has been previously built you should be able to select BioShell at this step. ### Choose an Instance Flavor @@ -119,8 +119,8 @@ ssh -i /path/to/your/key @ then clone this repository: ``` -git clone https://github.com/AustralianBioCommons/bioimage -cd bioimage +git clone https://github.com/AustralianBioCommons/BioShell +cd BioShell ``` ## Environment @@ -148,7 +148,7 @@ Example (template): ```Shell scp -i ~/.ssh/your_ssh_key \ ~/Downloads/my-project-openrc.sh \ - ubuntu@:/home/ubuntu/bioimage + ubuntu@:/home/ubuntu/BioShell ``` Where: @@ -193,15 +193,15 @@ If a project name is returned, your OpenStack environment is configured correctl ### Step 1: Initialize Packer Navigate to the `build` directory and initialize the Packer plugins: ``` -cd bioimage/build +cd BioShell/build packer init . ``` ### Step 2: Prepare Packer build configuration -Before running the build, review and update `openstack-bioimage.pkr.hcl` to ensure the values match your OpenStack environment. If using a prepared config skip to step 3. +Before running the build, review and update `openstack-bioshell.pkr.hcl` to ensure the values match your OpenStack environment. If using a prepared config skip to step 3. -**Note: Example working configurations for [Nectar](build/examples/openstack-bioimage-nectar.pkr.hcl) and [Nirin](build/examples/openstack-bioimage-nirin.pkr.hcl) are included and were last successfully tested on 2 February 2026. The Nirin configuration requires you to add your project [network](#network-cloud-dependant).** +**Note: Example working configurations for [Nectar](build/examples/openstack-bioshell-nectar.pkr.hcl) and [Nirin](build/examples/openstack-bioshell-nirin.pkr.hcl) are included and were last successfully tested on 2 February 2026. The Nirin configuration requires you to add your project [network](#network-cloud-dependant).** At a minimum, check the following fields in the `source "openstack"` block: #### Source Image (Base OS) @@ -284,7 +284,7 @@ If your cloud has a default network, this field may be omitted. #### CVMFS Configuration -The [`build-bioimage.yml`](build/build-bioimage.yml) playbook configures CVMFS for the image. +The [`build-bioshell.yml`](build/build-bioshell.yml) playbook configures CVMFS for the image. - By default, the CVMFS HTTP proxy is set to **DIRECT** to make the build more portable across environments. - If a infrastructure specific proxy is available (eg. `http://cvmfs-proxy-1.nci.org.au:3128;http://cvmfs-proxy-2.nci.org.au:3128` on Nirin), update the `CVMFS_HTTP_PROXY` line in the playbook. @@ -301,40 +301,40 @@ The relevant task in the playbook: CVMFS_QUOTA_LIMIT=4096 CVMFS_USE_GEOAPI=yes ``` -### Step 3: Build the BioImage +### Step 3: Build BioShell Once the configuration has been updated, run the build: ``` -packer build openstack-bioimage.pkr.hcl +packer build openstack-bioshell.pkr.hcl ``` ### Step 4: Verify Image After the build process is complete, verify the newly created image by running: ``` -openstack image list | grep bioimage +openstack image list | grep bioshell ``` If successful, you should see output similar to the following (showing the image ID (UUID), Name, and Status): ``` -| | bioimage | active | +| | BioShell | active | ``` -## Using the BioImage +## Using BioShell -This section describes how to launch a virtual machine using the BioImage and what functionality is available once the instance is running. +This section describes how to launch a virtual machine using BioShell and what functionality is available once the instance is running. -### Launching a BioImage VM +### Launching a BioShell VM Launch a new instance by following the steps in [Launch an Instance](#launch-an-instance). -When you reach the Source section, select Image and choose bioimage instead of a standard Ubuntu image. If the image was built successfully, no other changes are required +When you reach the Source section, select Image and choose BioShell instead of a standard Ubuntu image. If the image was built successfully, no other changes are required Once the instance is active, connect to it via SSH using the selected key pair. ``` -ssh -i /path/to/your/key @ +ssh -i /path/to/your/key @ ``` -### Tools Available in BioImage +### Tools Available in BioShell -BioImage includes a curated set of commonly used bioinformatics and workflow tools, exposed through the environment modules system. +BioShell includes a curated set of commonly used bioinformatics and workflow tools, exposed through the environment modules system. #### modules @@ -369,7 +369,7 @@ ls /cvmfs/data.galaxyproject.org ls /cvmfs/singularity.galaxyproject.org ``` -For an explanation of what CVMFS is, how it works, and how it is used in BioImage, see [CVMFS documentation](docs/cvmfs.md). +For an explanation of what CVMFS is, how it works, and how it is used in BioShell, see [CVMFS documentation](docs/cvmfs.md). #### R and RStudio `R` is pre-installed system-wide on the VM and is available without loading a module. @@ -411,4 +411,38 @@ Login using: If you have not set a password yet: ``` sudo passwd $USER +``` + +#### Shelley-Bio + +Shelley-Bio is a command-line assistant built into BioShell to help you find, explore, and install bioinformatics tools. It answers questions in plain language and can locate software available through the image's installed package managers and module system. + +Shelley-Bio is available system-wide and requires no module loading: +``` +shelley-bio --help +``` + +**Key commands:** + +| Command | Description | +|---|---| +| `shelley-bio search ""` | Find tools by describing what you want to do | +| `shelley-bio find ` | Look up a specific tool by name | +| `shelley-bio versions ` | List available versions of a tool | +| `shelley-bio build ` | Install or build a tool | +| `shelley-bio interactive` | Start an interactive session for guided assistance | + +**Examples:** +``` +# Find tools for quality control +shelley-bio search "quality control" + +# Look up samtools +shelley-bio find samtools + +# Check available versions of STAR +shelley-bio versions STAR + +# Launch interactive mode for guided help +shelley-bio interactive ``` \ No newline at end of file diff --git a/build/build-bioimage.yml b/build/build-bioshell.yml similarity index 85% rename from build/build-bioimage.yml rename to build/build-bioshell.yml index 0de98e1..c3d0dd8 100644 --- a/build/build-bioimage.yml +++ b/build/build-bioshell.yml @@ -1,5 +1,5 @@ --- -- name: Build a bioimage +- name: Build BioShell hosts: all become: true @@ -51,7 +51,9 @@ - jupyter - snakemake - r-base - - pip + - python3 + - python3-pip + - python3-venv - ansible - tree state: present @@ -479,7 +481,14 @@ module-whatis "RStudio Server 2023.12.1" prepend-path PATH "/usr/lib/rstudio-server/bin:/usr/lib/rstudio-server/bin" - set ip [exec /bin/bash -c "hostname -I 2>/dev/null | awk '{print \$1}'"] + + set ip [exec /bin/bash -c " + curl -fsS --max-time 0.5 http://169.254.169.254/latest/meta-data/public-ipv4 2>/dev/null || true + "] + + if { $ip eq "" } { + set ip [exec /bin/bash -c "hostname -I 2>/dev/null | awk '{print \$1}'"] + } puts stderr "" puts stderr "RStudio Server" @@ -642,3 +651,94 @@ - '/apps/Modules/modulefiles' - '/opt/Modules/modulefiles' + # ============================== + # Install Shelley Bio + # ============================== + - name: Ensure Python 3 and pip are installed + apt: + name: + - python3 + - python3-pip + state: present + update_cache: yes + + - name: Ensure Git is installed + apt: + name: git + state: present + + - name: Create installation directory for Shelley Bio + file: + path: /opt/shelley-bio + state: directory + owner: root + group: root + mode: '0755' + + - name: Clone Shelley Bio repository + git: + repo: https://github.com/Sydney-Informatics-Hub/shelley-bio.git + dest: /opt/shelley-bio + version: main + force: yes + depth: 1 + + - name: Install Shelley Bio dependencies globally + pip: + requirements: /opt/shelley-bio/requirements.txt + state: present + executable: pip3 + + - name: Install Shelley Bio package globally + pip: + name: /opt/shelley-bio + editable: yes + state: present + executable: pip3 + + + - name: Create system-wide launcher script + copy: + dest: /usr/local/bin/shelley-bio + owner: root + group: root + mode: '0755' + content: | + #!/usr/bin/env bash + exec python3 -m shelley_bio.client.cli "$@" + + - name: Test Shelley Bio CLI + command: shelley-bio search "quality control" + register: shelley_cli_test + ignore_errors: yes + + - name: Show Shelley Bio CLI test output + debug: + var: shelley_cli_test.stdout_lines + + - name: Add BioShell welcome message with turtle + copy: + dest: /etc/motd + owner: root + group: root + mode: '0644' + content: | + ############################################################ + # # + # Welcome to BioShell! # + # # + # A range of bioinformatics software is available in this # + # # + # # + # If you need help finding and installing bioinformatics # + # tools, ask Shelley-Bio # + # _ .----. # + # (_ \/ \_, # + # `uu----uu' # + # Basic commands: # + # shelley-bio find # + # shelley-bio search "" # + # shelley-bio versions # + # shelley-bio build # + # shelley-bio interactive # + ############################################################ \ No newline at end of file diff --git a/build/examples/openstack-bioimage-nectar.pkr.hcl b/build/examples/openstack-bioshell-nectar.pkr.hcl similarity index 90% rename from build/examples/openstack-bioimage-nectar.pkr.hcl rename to build/examples/openstack-bioshell-nectar.pkr.hcl index 8b9f02e..6a5cf59 100644 --- a/build/examples/openstack-bioimage-nectar.pkr.hcl +++ b/build/examples/openstack-bioshell-nectar.pkr.hcl @@ -13,7 +13,7 @@ packer { } source "openstack" "ubuntu" { - image_name = "bioimage" + image_name = "bioshell" flavor = "r3.small" source_image = "c0250c96-98a4-4bfa-b67c-51874808337f" ssh_username = "ubuntu" @@ -24,7 +24,7 @@ build { sources = ["source.openstack.ubuntu"] provisioner "ansible" { - playbook_file = "./build-bioimage.yml" + playbook_file = "./build-bioshell.yml" extra_arguments = [ "--extra-vars", "ansible_user=ubuntu" diff --git a/build/examples/openstack-bioimage-nirin.pkr.hcl b/build/examples/openstack-bioshell-nirin.pkr.hcl similarity index 91% rename from build/examples/openstack-bioimage-nirin.pkr.hcl rename to build/examples/openstack-bioshell-nirin.pkr.hcl index db23d00..083f29f 100644 --- a/build/examples/openstack-bioimage-nirin.pkr.hcl +++ b/build/examples/openstack-bioshell-nirin.pkr.hcl @@ -13,7 +13,7 @@ packer { } source "openstack" "ubuntu" { - image_name = "bioimage" + image_name = "bioshell" flavor = "c3.1c2m10d" networks = [""] # update via: openstack network list - select network associated with your project availability_zone = "CloudV3" @@ -26,7 +26,7 @@ build { sources = ["source.openstack.ubuntu"] provisioner "ansible" { - playbook_file = "./build-bioimage.yml" + playbook_file = "./build-bioshell.yml" extra_arguments = [ "--extra-vars", "ansible_user=ubuntu" diff --git a/build/openstack-bioimage.pkr.hcl b/build/openstack-bioshell.pkr.hcl similarity index 96% rename from build/openstack-bioimage.pkr.hcl rename to build/openstack-bioshell.pkr.hcl index 8232a36..986375e 100644 --- a/build/openstack-bioimage.pkr.hcl +++ b/build/openstack-bioshell.pkr.hcl @@ -13,7 +13,7 @@ packer { } source "openstack" "ubuntu" { - image_name = "bioimage" + image_name = "bioshell" flavor = "" # update via: openstack flavor list networks = [""] # update via: openstack network list availability_zone = "" # update via: openstack availability zone list diff --git a/docs/DEPRECATED_old_guid_nirin.md b/docs/DEPRECATED_old_guid_nirin.md index 8f7bc8c..34190cf 100644 --- a/docs/DEPRECATED_old_guid_nirin.md +++ b/docs/DEPRECATED_old_guid_nirin.md @@ -38,7 +38,7 @@ Stop instances when they are not in use and restart them as needed. ./openstack/instances-stop.sh ``` -### Tools available in Bioimage +### Tools available in BioShell #### modules @@ -73,7 +73,7 @@ ls /cvmfs/data.galaxyproject.org ls /cvmfs/singularity.galaxyproject.org ``` -For an explanation of what CVMFS is, how it works, and how it is used in BioImage, see [CVMFS documentation](docs/cvmfs.md). +For an explanation of what CVMFS is, how it works, and how it is used in BioShell, see [CVMFS documentation](docs/cvmfs.md). ## User Access diff --git a/docs/cvmfs.md b/docs/cvmfs.md index e355815..91d77e5 100644 --- a/docs/cvmfs.md +++ b/docs/cvmfs.md @@ -1,9 +1,9 @@ # CernVM-FS (CVMFS) ## What is CVMFS? -CernVM-FS (CVMFS) is a read-only, network-distributed filesystem originally developed to deliver software and data efficiently at scale. In BioImage, CVMFS is used to provide access to shared resources—such as container images, reference datasets, and other commonly used scientific files—without requiring them to be installed or copied onto each VM. +CernVM-FS (CVMFS) is a read-only, network-distributed filesystem originally developed to deliver software and data efficiently at scale. In BioShell, CVMFS is used to provide access to shared resources—such as container images, reference datasets, and other commonly used scientific files—without requiring them to be installed or copied onto each VM. -## Why BioImage uses CVMFS +## Why BioShell uses CVMFS - Lightweight (smaller VM images) - Consistent (everyone sees the same tools/data) - Easier to maintain (updates happen centrally) diff --git a/manage/openstack/create-bootable-volumes.sh b/manage/openstack/create-bootable-volumes.sh index d5ee5ad..ac10b28 100755 --- a/manage/openstack/create-bootable-volumes.sh +++ b/manage/openstack/create-bootable-volumes.sh @@ -1,6 +1,6 @@ #!/bin/bash -BASE_IMAGE_NAME="bioimage" +BASE_IMAGE_NAME="bioshell" VOLUME_PREFIX="training-VM" VOLUME_SIZE=20