Are RIAs necessary?

[asmacdo]

Problem

goal of this issue: collect RIA discussion in 1 place, freeing up other related issues to be more focused.

babs currently creates input_ria/ and output_ria/ as part of every project. These are local RIA stores used to shuttle data between the project and job scratch directories. I do not understand the benefit, and we end up with more clones of the same dataset.

• input_ria is a clone of the input dataset that jobs clone from, but the actual data is never pulled into it. Jobs datalad get through it and the data comes from the upstream source. It's an empty intermediary adding an extra hop in the clone chain.
• output_ria collects job results before the octopus merge. Could we use a regular bare git repo for the same benefits?

The RIA pattern was motivated by the FAIRly big paper, where condor jobs needed to work across systems without a shared filesystem. When a shared filesystem exists, RIAs add indirection (extra clones, extra remotes) without solving a real problem.

Proposal

Replace RIA stores with direct git operations on the shared filesystem:

• Input: jobs clone directly from the analysis dataset (or a bare repo alongside it)
• Output: jobs push result branches directly to the analysis dataset (or a bare repo), with a lock file to serialize pushes as Yarik suggested in #327

This simplifies the project layout, removes a conceptual layer users have to understand, and eliminates the post-merge clone-from-output-RIA step.

Related discussion

• Feature Request: Make zipping optional #327 — Yarik raised whether RIAs are necessary on shared filesystems (comment)
• Support git worktree #338 — ephemeral worktree clones (further reduces need for RIA indirection)
• Use/promote use of datalad's add-archive-content #343 — related project layout discussion
• git-annex parallel push issue — may need serialized push workaround

Questions

• Are there babs users running without a shared filesystem (e.g., HTCondor across sites)? If so, RIA removal would break their workflow.
• Is the octopus merge step tightly coupled to RIA, or does it just need branches from any git remote?

[satra]

@asmacdo - at mit there are effectively two filesystems (a slow one and a fast one). the fast one is where compute happens and data needs to be pulled into that and the jobs cannot be all pulling from the slow system. so just having a shared filesystem does not resolve the issues with staged processes when you are operating at the scales of hbn/abcd/ukbiobank.

does it need rias? that can be decided, but as far as we knew that was a way of making this happen without a more expensive clone operation.

[yarikoptic]

Replace RIA stores with direct git operations on the shared filesystem:

bad. as @satra pointed out -- there are use cases warranting use of RIAs. We need not to switch but make them optional!

edit: clarification -- and "optional" separately, as it could be a setup with input_ria only and could be setup with output_ria only, none (shared, might need locking + serial pushing to overcome git/git-annex shortcoming), or both (current).

[yarikoptic]

overall -- there should be nothing special in "RIA"s besides that they might need to be created. Otherwise -- any git (or git-annex special) remote (datalad sibling) should work while original study/project directory being the ultimate location where things converge!