This repository is the third part of the FAIRification pipeline and is responsible for uploading metadata to data.bbmri.cz catalogue.

Supported sequencing types

Miseq, New Miseq, MammaPrint

How to run the scripts

Dev environment

Using main.py

1. Install requirements

pip install -r requiremenents.txt

2. Run main.py

python main.py -r path/to/pseudonymized/runs/folder  -o /path/to/root/organisation/folder -p /path/to/patients/folder

Using docker-compose

docker compose up -d --build

Test environment

Folder structure

/muni-sc/test/
├── Libraries/ # Required libraries
├── pseudonymized_runs/ # Input runs from pseudonymisation
├── Patients/ # Patient metadata
├── organized_runs/ # Organised runs output
│ └── logs/ # Logs from organiser/uploader runs
├── wsi/ # Optional WSI files for testing

Running a Test

1. Have a run that can be uploaded in the organized_runs/ dir

2. Start the uploader service:

docker compose -f compose.test.yml up --build

Viewing Logs

Logs for each run are in /muni-sc/test/organized_runs/logs/uploader.

To view all container logs:

docker compose -f compose.test.yml logs

In production

Production is running on Kubernetes cluster SensitiveCloud

Using kubernetes (kubectl)

Deploy dependent secrets

## Supported sequencing types
Miseq, New Miseq, MammaPrint

## How to run the scripts
### Locally - Development
#### Using main.py
1. Install requirements
```bash
pip install -r requiremenents.txt

2. Run main.py

python main.py -r path/to/pseudonymized/runs/folder  -o /path/to/root/organisation/folder -p /path/to/patients/folder

Using docker-compose

docker compose up -f compose.yml -d --build

In production

Production is running on Kubernetes cluster SensitiveCloud

Using kubernetes (kubectl)

Deploy dependent secrets

kubectl apply -f kubernetes/catalog-secret.yaml -n bbmri-mou-ns

kubectl apply -f kubernetes/uploader-job.yaml -n bbmri-mou-ns

Deploying new version in production

Build new docker image

docker build --no-cache <public-dockerhub-repository>/data-catalogue-organiser:<version> .
docker push <public-dockerhub-repository>/data-catalogue-organiser:<version> 
# change version in kubernetes/organiser-job.yaml

Debigging

You can visit kubernetes UI Rancher find the failing pod and investigate in logs. On how to use Rancher and SensitiveCloud visit Docs

Other option is running a testing job and investigation inside the cluster filemanager (to check user permissions etc.)

kubectl apply kubectl apply -f kubernetes/testing-job.yaml -n bbmri-mou-ns

Then connect to terminal of this job/pod on Rancher

kubectl apply -f kubernetes/organiser-job.yaml -n bbmri-mou-ns

Deploying new version in production

Build new docker image

docker build --no-cache <public-dockerhub-repository>/data-catalogue-uploader:<version> .
docker push <public-dockerhub-repository>/data-catalogue-uploader:<version> 
# change version in kubernetes/organiser-job.yaml

Debigging

You can visit kubernetes UI Rancher find the failing pod and investigate in logs. On how to use Rancher and SensitiveCloud visit Docs

Other option is running a testing job and investigation inside the cluster filemanager (to check user permissions etc.)

kubectl apply kubectl apply -f kubernetes/testing-job.yaml -n bbmri-mou-ns

Then connect to terminal of this job/pod on Rancher