In construction
This technical guide explains the components of Pitschi and how they work together.
Pitschi employs two methods of moving data from instrument computer (camera computer to be exact) to the RDM.
If Pitschi can be installed directly into the camera computer, it will use the direct transfer method.
In for some reasons, the camera computer cannot be modified, or the RDM cannot be mounted directly to the camera computer, then the indirect method is used. In this mode, the data is synced via a third party computer (support machine, preprocessing computer).
Pitschi relies on the MeDICI for data movement and UQ RDM for data storage. The booking information is queried from PPMS.
Pitschi consists of the following components:
- Clowder: a modified version of the Clowder framework for data management and automatic metadata extraction
- XAPI: an API to manage data importation from RDM storage collections. The booking information from PPMS is cached in this XAPI as well.
- Pitschi-cli: A desktop client residing in the instrument computers for syncing datasets and files to the project's storage collection.
- DataMover: a Web service to handle 1) mounting of correct collection to the third party computer 2) sync the data and 3) ingest the data to Clowder. This Web service provides a client, which can be accessible from the camera machine.
Pitschi is deployed on a Docker Swarm Cluster on top of the QRISCloud (OpenStack) research cloud. Ansible 2.9 is used provision the stack. The deployment code can be found here
In Clowder, space is the highest level in hierarchy of data organization. Space consists of datasets, and within datasets, there are files and folders. Collection is a group of datasets. Users can generally create spaces by themselves and within each space and dataset, what users can do is limited by their roles. A more detailed description of Clowder can be found here
In Pitschi, while the data hierarchy is the same, we have modified the access control to fit our data flow. This section of the guide details how data management is done is Pitschi.
In Pitschi, manual creation of accounts and spaces is disabled. Instead, projects and its accounts are synced from PPMS.
UQ users will need to use UQ SSO to log into Pitschi, while external accounts are created using email address supplied from PPMS.
