Welcome!

Installation

Install GIT Windows

Install GIT LFS

Nice intro video to LFS

Install Anaconda

Install Github Desktop, if you do not like to use the terminal so much.

• Setup

Download Visual Studio Build Tools, and install C++ package manager (You might need admin rights for this)

Install Atom

Cloning the CA2-MEC repository

This can either be done with Github Desktop or with a terminal. When we say terminal, in Windows we mean Anaconda prompt. However, all code that starts with git can be done with Github Desktop.

Make a folder where you want to have it, e.g. c:\apps\

With the terminal run

cd c:\apps
git clone https://github.com/sarahthon/CA2_MEC.git

cd CA2_MEC
git switch vemundss

Anaconda

Windows: Search for anaconda and open Anaconda prompt.

Mac: open a terminal

Navigate to into CA2_MEC, and create anaconda environment locally by installing the environment file environment.yml:

conda env create -f environment.yml 

Then, enter the environment using its name expipe

activate expipe

Exiting the environment can be done by

deactivate

Installing expipe for CINPLA

Navigate to where you have cloned CA2_MEC, then install the CA2_MEC requirements. This clones and installs specific commit of the github repos.

activate expipe
pip install -r requirements.txt

Two repos have been edited and are currently cloned into src-folder when cloning this repo. For these two you need to navigate into each folder and install by pip install .:

cd src
cd spatial-maps
pip install .
cd..

cd expipe-plugin-cinpla
pip install . 
cd..

Using pip install . will later let you uninstall the package simply by entering the environment and: pip uninstall "package-name"

Manually install the following in the src folder and switch to the given commit:

cd src
git clone https://github.com/CINPLA/time-frequency.git
cd time-frequency
git checkout 84336a9c546ba81691388ea6a51a0668639058c5
cd..

git clone https://github.com/CINPLA/spike-waveform.git
cd spike-waveform
git checkout 2ebb3ea8eb96d185517554ce1dd845c9d36e54b1
cd..

git clone https://github.com/CINPLA/visual-stimulation.git
cd visual-stimulation
git checkout 83363a45553dca5d46ea182c83ec8f6d95259f94
cd..

Note: after cloning all repos into src, you can also set a new python path in the environment to this folder so you don't have to enter each folder and install. Alternatively add this to the notebooks before importing the packages:

import sys
sys.path.append("/path/to/src")`

Adding expipe_plugin_cinpla plugin

In order to add lab-specific expipe commands, run:

expipe config global -a plugin expipe_plugin_cinpla

Installing spike sorting tools (optional)

In order to run spike sorting with the machine you are using, you have to install them. Navigate to where you have cloned cinpla-base, then install the cinpla-base requirements-spiketools. Choose the appropriate file (windows-linux)

cd CA2-MEC
pip install -r requirements-spiketools-windows.txt (or requirements-spiketools-linux.txt)

This will install the Python-based sorters(klusta, spyking-circus, mountainsort, herding spikes, tridesclous)

Installing Matlab-based spike sorters

Kilosort2, Ironclust, and WaveClus are matlab-based spike sorters. To install them navigate to C:\apps and run:

git clone https://github.com/MouseLand/kilosort2.git
git clone https://github.com/jamesjun/ironclust.git
git clone https://github.com/csn-le/wave_clus.git

In order to let the system know where these packages are installed we have to set environment variables. If you have admin access, in Windows, select start-->Computer-->right-click-->Properties. Then click on Advanced settings-->Environment variables and add this three New variables:

name: KILOSORT2_PATH     variable: C:\apps\kilosort2
name: IRONCLUST_PATH    variable: C:\apps\ironclust
name: WAVECLUS_PATH    variable: C:\apps\wave_clus

In case you don't have admin access you can set temporary environment variables from the anaconda prompt by running:

set KILOSORT2_PATH=C:\apps\kilosort2
set IRONCLUST_PATH=C:\apps\ironclust
set WAVECLUS_PATH=C:\apps\wave_clus

Installing curationtools [phy]

Dependencies for curation tools are alredy installed in environment

Phy is already installed in environment, but you can test out a newer one with:

pip uninstall phy
pip uninstall phylib

pip install --pre --upgrade phy

Adding a remote server

To add a remote server for spike sorting, run:

expipe add-server -n name-of-the-server -d IP-address -un username 

PC at IBV has name = torkel-beist

You will be prompted a password for the server. From time to time you need to update the IP-address.

Notebooks:

You should now be able to open jupyter notebook in folder CA2_MEC/notebooks:

cd CA2_MEC/notebooks
jupyter notebook

To have access to you data saved in expipe, you must let the system know where this project is stored by setting its path as an environment variable. If you have admin access, in Windows, select start-->Computer-->right-click-->Properties. Then click on Advanced settings-->Environment variables and add this New variable:

name: PROJECTNAME_PATH     variable: C:\path\to\projects\projectname

Then you must change the path to corresponding project in \apps\CA2_MEC\ca2-mec\dataloader.py under the function def project_path(): so that it reads:

def project_path():
    path = os.environ.get("PROJECTNAME_PATH")

If you do not have the packages in src as environmental variables in your environment (see note under "installing requirements" above), you must refer to the packages' location before importing them into notebooks by adding in the notebook:

import sys
sys.path.append("/path/to/src")`

Getting started with git LFS and gitea@nird

Register a Gitea account

• Contact Mikkel, Alessio or Svenn-Arne
• Navigate to Gitea and expipe on NIRD

Create an expipe repository

• Initialize with .gitignore, licence and README

Clone repository

• from notebook (see example below)
• with git desktop

If you want to clone a LFS repository and only get the pointer files (not the large files) do

git lfs install --skip-smudge

And then clone as usual

git clone https://gitea.expipe.sigma2.no/user_name/my_project_name.git

Open jupyter notebook inside repository

Navigate to my_project_name and write in the terminal

jupyter notebook

In stead of using the terminal from now on, commands can be run from within the notebook if it is begun with an exclamation mark. That is, if you would write expipe init in the terminal, you would write in the notebook

expipe init

This command adds actions, entities and templates folders together with an expipe.yaml, this is necesarry so that my_project_name will be recognized as a expipe project.

Git LFS

Intro from atlassian Official docs

There are many good tutorials for LFS, use those or look at the brief intro below.

Normal git is not "wired" to handle large files, this is why we use git LFS which is helping us handling large files (LFS stands for Large File Storage)

Be sure to initialize LFS properly before adding and committing files. It is possible to rewrite history later (with git lfs migrate import), but this can be a pain..

git lfs install

Next we need to tell LFS which files should be tracked as large files, if you are not sure, you can add all data which is put in the action/data folder by

git lfs track "actions/*/data/**/*"

Open the .gitattributes files and make sure it reads: (you may want to add the last line which tells LFS to skip .yaml files)

actions/*/data/**/* filter=lfs diff=lfs merge=lfs -text
*.yaml !filter !diff !merge

The first line says that all the contents in all data directories whithin every action should be handeled by git LFS. The second line says that even though all the data directories should be handeled by LFS, all files ending with .yaml should not be handeled by LFS.

In short, these lines ensures that all file except .yaml files will be downloaded as LFS files when the repository is cloned or pulled if nothing else is specified. This means that all files in data except .yaml files will be text files pointing to the real data files on NIRD.

If you only want to track npy and dat files

git lfs track "*.npy"
git lfs track "*.dat"

Add the .gitattributes file

git add .gitattributes

To avoid downloading the original files when doing a git pull the following command adds .lfsconfig with a line specifying to exclude all LFS tracked files.

git config -f .lfsconfig lfs.fetchexclude "*"

git add .lfsconfig
git commit -am "init expipe and LFS"

Now try to add and commit some files which should be tracked by LFS, and make sure they are correct by looking at

git lfs ls-files

Add templates

Navigate to cinpla-base/src/expipe-templates-cinpla/templates, where you'll find a bunch of example templates, copy some you want to use or write your own. If you want to write your own template the filename must on the form filename.yaml and at a minimum contain

name: filename
identifier: filename

It is probably wise to commit after you add the templates.

git add -A
git commit -am "added some templates"

And now push your changes:

git push

Working with expipe

Now you are ready to start using expipe

Open cinpla browser

from expipe_plugin_cinpla.widgets import browser
browser.display('workshop')

You can register actions (surgery, adjustments, recordings, perfusions, etc.) and process data using the jupyter GUI. Once you have worked on the repo, you should git add the changed files, git commit and git push.

If you want to physically download the files of an action to perform analysis, for example, first you need to pull the required files:

git pull
git lfs fetch -I path-to-action
git lfs checkout

or equivalently

git -c lfs.fetchexclude="" lfs pull -I path-to-action

If you want to download all files:

git lfs fetch --all
git lfs checkout

Once you have pushed everything, if you want to free some space you can delete your actions and run (do it with caution!!!):

git reset --hard

Open expipe browser

You can check the status of your expipe project using the expipe Browser:

import expipe
expipe.Browser('workshop').display()

Troubleshooting

gitea

If you want to store your credentials do

git config credential.helper store
git pull

Type in your credentials and it is then stored.

LFS

If you get lfs-timeout errors when pushing (i/o timeout, error: failed to push some refs), consider changing your lfs settings to with

This might be a sign that LFS is not tracking your files properly, if so try git lfs migrate import, although we have experienced this not to properly add files to LFS, see which files are tracked by git lfs ls-files

git config lfs.tlstimeout 300
git config lfs.activitytimeout 60
git config lfs.dialtimeout 600
git config lfs.concurrenttransfers 1

git config --global https.postBuffer 2097152000
git config --global http.postBuffer 2097152000

Sometimes when cloning, pulling, or pushing to gitea you might get this error:

fatal: unable to access 'https://gitea.expipe.sigma2.no/username/project.git/': The requested URL returned error: 403

This can be solved by changing the repo name copied from gitea:

https://gitea.expipe.sigma2.no/username/project.git/

to

https://username@gitea.expipe.sigma2.no/username/project.git/

If your .git folder gets huge, you can delete old LFS files from local storage with (make sure you are sync with remote (all files are pushed))

git lfs prune

python

• if you get a multiarray error when running expipe init run:

pip uninstall numpy
pip install numpy

No module named repository

pip install gitpython

jupyter notebook

Stale connection, unable to connect to kernel

pip install "tornado<6"