Parameter file as input to workflow

[joergfunger]

We originally started having all tools and parameter files included in a single snakefile to simplify the execution of the workflows. In a second step thought about having a single ROCrate (and a single workflow) being executed per tool, e.g. there is one workflow for FEniCSx and another one for Kratos etc. However, our workflow now has actually no inputs (that @M-Jafarkhani could describe semantically) and we extract these using a specific function that is now user-dependent. One option might be that we interpret a single workflow only as a function that takes as input a couple of metadata/inputs - these are actually all defined in the benchmark definition (that we provide in the KG or in the benchmark ROCrate (depending on where the benchmark is now defined) or by the user (the tool and the version).

In the python package, the user says: "Run benchmark XYZ (URI or id of ROCrate) with tool FEnics (SoftwareApplication URI including version etc.) and a compute environment (docker image, conda, ..) with a script that takes as input a parameter.json file and outputs a metrics.json and a vtk.zip. In this setup, the loop over all parameter files is done by the user when creating the ROCrate and then a joint ROCrate is uploaded. Here are the details of these steps for a users that want to run an existing benchmark with their tool:

1. Define what the user supplies (benchmark + tool + environment)
   In the example, the inputs is a benchmark_rocrate_ref (ROHub URL/UUID or local ZIP/path), tool_uri (software_application, including metadata such as version number). For that we would have to provide an link how to add something in case that is not existing (has to be done a priory or in the package itself, let's see). The user would provide a compute environment (or can we provide that with the SoftwareApplication? @doigl )

2. Obtain the benchmark contents and discover parameter files
   If benchmark_rocrate_ref points to ROHub, use the ROHub client in ROhub to download the benchmark crate ZIP, unzip to a working directory, or similarly take the zip file, or extract the information from the NFDI4ING KG
   Discover parameter files (either unzip, or extract from KG)

3. Loop over all parameter files

   1. for each parameter file, generate a tool Snakefile per run from the benchmark-provided template
   2. Execute Snakemake once per parameter file with explicit input/output (each run creates a provenance file with clear input/output. The parameter file is passed as a parameter --config as a first-class workflow input to snakemake. Ensure each run writes into a distinct output subfolder (tool + configuration) so subsequent runs don’t overwrite artifacts.

4. Create a provenance RO-Crate for the complete loops and upload to ROHub
   This is maybe a tricky point, because I do not know if that is possible (including multiple runs in a single ROCrate). And maybe this is strongly related to how we link the benchmark definition to the provenance of a single execution. @doigl @M-Jafarkhani We currently create one ROCrate per tool/benchmark. If we do the approach outlined above, we get multiple provenance graphs (one per parameter file). They would have to be combined (in a similar way as we discussed today the question of @div-tyg of how to combine multiple ROCrates in a query). IMO this would be something we have to handle here (but in a similar way in the jupyter notebook for analyzing multiple ROCrates). At the end, we just have a a joint KG that comprises all information. @doigl and @M-Jafarkhani do you think there is an option to do so?

[div-tyg]

From the discussion with @M-Jafarkhani I understand that the semantic description can be done if the inputs to the workflow are provided directly inside the rule definition of a snakefile. At the moment, the inputs (parameter configurations) are read from the workflow_config.json file. This is requiring us to write extractor functions separately. Rewriting the Snakefile can solve the issue.

Further, I think we should ask ourselves the question, whether a snakemake workflow is absolutely necessary for us to be able to generate knoweldge graphs or RO-Crate objects. If that is the case, it is advisable to use the wildcards feature that snakemake provides. Running a single parameter configuration for each snakemake workflow run will be impractical. Instead while creating a benchmark, a benchmark creator should specify the parameter configuration filenames as wildcard in snakemake workflow template. This will remove the need to manually loop over parameter configurations.

Additionally, I think with the 1 ro-crate/tool setup, the number of wildcards in a snakemake workflow reduces to 1, so it shouldn't be complicated to write a workflow for it. Removal of the summary rule currently present in the snakefile will further simplify the task.

[joergfunger]

The question is, how we handle not being able to run a specific configuration. We currently assume that we run the workflow for all configurations, but we realized that certain options are not possible or implemented by a specific software. So the first option is that the user would have to select beforehand which configurations they are running, and then they have to loop over them from the userside (in our example, the user would make a list of the configurations and then do what you cal the external loop). Alternatively, the user can decide that for specific inputs (that are not implemented in a software tool), an error message is returned. This would somehow have to be implemented on the user side when reading in the json file with the parameter configurations and then decide, if they can use that (replace placeholders in the template inputs and run the tool), or provide an error message (that would have to be semantically described). In the first case, the loop has to be done anyway, in the second case we could just prepare e.g. individual folders and then let snakemake loop over all these folder with parameter files within a wildcard loop.

[M-Jafarkhani]

I created a visualization to clarify my understanding—please correct me if I’m wrong. I have two questions:

1. What is the difference between the reference RO (fetched in Step 1) and the final, jointly merged RO created in Step 4? Is the RO in Step 1 essentially a blueprint that guides the execution, while the RO in Step 4 is the final representation containing the actual results and values?

2. I think having a fixed structure for parameter specifications and the output file (i.e., summary.json) would make it much easier for the reporter plugin to extract parameters. A JSON Schema could enforce this structure. Should we expect such a fixed structure? Some information, such as units, cannot be easily specified in the Snakemake file.

[joergfunger]

1. These are two different ROCrates (even with a different structure). It is not even certain that the information from the first one is really stored in an ROCrate, but could e.g. be stored also in e.g. NFDI4ING knowledge graph - or both in an ROCrate and in the NFDI4INg KG. It should be a KG that merely stores the benchmark definition and links to files (such as the snakefile templates or some meshing files (or the mesh file directly) in case the mesh should be provided by the benchmark). In the figure below, essentially the left-hand part including the green benchmark and the sensor output. The second ROCrate, which stores the result of benchmark executions, could actually include the information from the benchmark definition (in theory that is not required since you can fetch both and just put them together, in practice I would add it due to having a complete KG that can be queried even locally with just the ResultROCrate being present). And it should store the right bottom part, essentially the tool implementation of the benchmark (so e.g. the fenics files and its compute environment), and then the actually simulation runs. There could be multiple runs (as sketched above), each run links to one configuration, one tool implementation (and the benchmark itself) and then links also to the outputs (the metrics and the vtk-files). And (this is not added yet), but also to the provenance of the execution that you are tracking with the reporter plugin. So each run should have a direct link to the complete execution provenance of that single run.
2. We assume that fixed structure, but IMO we just need that for the metrics.json file that we read in (and similar to the parameters store them directly in the KG). The parameters are directly extracted from the input KG (see above), and the summary is IMO no longer needed, since there is no additional information stored in there anymore. Originally, that was to summarize a single run of a configuration, but I would argue that this is no longer needed. It would be nice to have that structure for the VTK files as well, though I'm not sure if we find a unique standard for all tools.

What do we do if someone adds another parameter configuration (that was not originally in the benchmark). If that is just added to the general definition, then everyone querying the graph afterwards would run this additional configuration as well, potentially leading to challenges. @doigl Any idea, how we could avoid that, while still allowing users to define their own configurations? As such, I prefer running only those that are actually feasible, so the users loops over the configurations, and only creates results for those that are actually run). If they want to provide additional information, we could later think about a simulation run that provides as output some reason for not having run a certain configuration, but IMO that would be something we should think about in a later stage.