Split runcards and runner?

[alecandido]

@felixhekhorn was proposing to split runcards and runner repositories. To maintain the data and the program separated.

On my side I see:

• pro: the development is decoupled, you can clone/download your data repository with nothing there but runcards, and we can version it (tagging) for various works and so on
• con: sometimes you really need to implement a feature (like a cut in MadGraph) for a specific runcard, and it would be nice to develop and release them together, instead of having to take action on different repositories (mostly not needed for anything but MadGraph, at the moment, but things might change)

Since we're getting closer and closer to our pineline, and we should at some point even change names (as agreed at the in-person code meeting, runcards are vp ones), I guess that people involved can start discussing @felixhekhorn @scarlehoff @cschwan

[felixhekhorn]

• I'm still a big fan of this, because the two things are really decoupled
• as also briefly discussed with @alecandido: I would even do a further step and split eventually the data repo along the paper repo - since typically runcards have a clear association to a paper: e.g. afb, mw, 4.0 (the question whether the different papers are hosted in the very same repo or not is a separate question - one might even be fine)
• for the case of simultaneous development: it is true, you have to work in two separate repos, but I would say mostly in the code repo: for the data repo you set the parameter and that's it - whether the thing might be ignored or misinterpreted is a pure code issue

the use case I'm currently having in mind is positivity: @tgiani asked me to produce some grids - so what I would like to do is:

• create a subfolder in the positivity repo called fks
• create create several subfolders inside fks called Q2_5, Q2_10, etc.
• copy inside each of this folders the NNPDF_POS_* folders from the main repo and adjust them inside
• pip install what-ever-rr (or similar)
• create a config file in each of the Q2 folders
• run all grids in all Q2 folders
• pip install pineko
• create a config files in each of the Q2 folders
• create all FK tables in all Q2 folders
• send them to Tommaso (@tgiani you see it is not sooo trivial ;-) )

[scarlehoff]

This reminds me of the buildmaster discussion and at the end it was decided to have everything together.
Since for now runcard runners (the people) and runner (the program) developers are the same people I think it makes sense to keep it together at least for the time being. Once the program is so stable that the runcards really have a life on its own then we can easily split the folder of runcards out.

[alecandido]

• for the case of simultaneous development: it is true, you have to work in two separate repos, but I would say mostly in the code repo: for the data repo you set the parameter and that's it - whether the thing might be ignored or misinterpreted is a pure code issue

Consider that MadGraph runcards contains analyses files (Fortran), patches (still Fortran, but in the patch format), and postrun scripts (shell). E.g.:
https://github.com/NNPDF/runcards/tree/924078d60beabd579e98e782d7ee306ec85a490e/nnpdf31_proc/ATLAS_1JET_8TEV_R06

[alecandido]

• pip install what-ever-rr (or similar)

The moment we decide a name, I can set the releasing workflow (reusing the usual one, so it is really a few lines of YAML).

[cschwan]

The moment we decide a name, I can set the releasing workflow (reusing the usual one, so it is really a few lines of YAML).

After seeing a bunch of videos on how to grow pineapples - apparently it's quite easy 😉 - I'm thinking about pineapplfarm or a variation of that:

pineapplfarm grow ATLAS_WP_7TEV theories/theory_200_1.yaml

This will grow a nice PineAPPL grid (luckily you don't have to wait for several months/years):

ATLAS_WP_7TEV.pineappl.lz4

That's another idea for a name; I think the analogy is quite nice!

[alecandido]

Then what about just farm, for simplicity? (and ease of typing)

We can add a pineapple in a banner, or somewhere else, to stress the concept of the pineapple farm :D

P.S.: it seems like it's not a command provided by any Ubuntu package as well

Command 'farm' not found, did you mean:
  command 'ferm' from deb ferm (2.5.1-1)
  command 'faxrm' from deb hylafax-client (3:6.0.7-3.1)
  command 'faxrm' from deb mgetty-fax (1.2.1-1.1)
  command 'fasm' from deb fasm (1.73.27-1)
  command 'form' from deb form (4.2.1+git20200217-1)
  command 'frm' from deb mailutils (1:3.14-1)
Try: sudo apt install <deb name>

P.P.S.: actually the python package will be called pineappl-farm

[cschwan]

@alecandido I'm afraid farm might (or might not) be a bit too generic. There's already FARM (seems to be an API, no binary). A compromise could be pinefarm, which is similar to pineko.

[alecandido]

The domain already provides a natural namespace-like feature (it's unlikely to have a user of both), and repeated names are not so uncommon, you just have to avoid to overwrite something very common (from ls to systemctl, htop and so on).

However, I'm actually in favor of pinefarm.

Then maybe the package should be called pinefarm as well, it's the simplest choice.

[alecandido]

Ok, from now on, we finally have pinefarm and pinecards! 🎉

[alecandido]

And now it is also on PyPI 🐍

You can finally install with:

pip install pinefarm

[alecandido]

Now even workflows are in place, and we release by pushing tags.
The test workflows is not testing at the moment (because there is no test available, even though it would be a good idea to have a few). But it is linting, and installing dependencies at every push (otherwise CI runs only on tags).

[cschwan]

That's clever, I might copy it for PineAPPL.

[alecandido]

If you want to release tagging, I can provide something for PineAPPL.
Unfortunately, the workflow I'm using here is very much tuned for a Poetry-only project, but it is not so hard to cook one dedicated, it has only to:

• bump versions (but for this there is make_release.sh)
• run cargo publish for all relevant crates in the workspace (e.g. not pineappl_py)

pineappl_py is already released by the other workflow, running on releases. If you want to change to tags we only need to change the event.

[felixhekhorn]

For the sake of this issue: we keep them together for now (as suggested by @scarlehoff ) - so we can close this issue

PS: to continue the off-topic discussion

• @cschwan : I guess we can drop run.sh, Makefile, applcheck.cpp ?
• shall we also start publishing to rtd?

[alecandido]

We are already on rtd, just with the wrong name: https://runcars.readthedocs.io/en/latest/