Eirene Coupling

[akashukla]

Overview

In this PR we enable the coupling of Gkeyll to Eirene for neutral evolution. The model for including plasma sources in Gkeyll based on eirene output is described in DR #923 . Gkeyll evolves the plasma and Eirene evolves the neutrals (recycling, reactions, etc.). They exchange data through I/O every time Gkeyll outputs data.

This PR contains several parts. The first is the automatic installation of eirene and grid generation parts of SOLPS-ITER as part of the gkeyll mkdeps process. (This only occurs if one uses --build-solps=yes). The second part is the installation of a python library written by @jRoeltgen and me which does post-processing and interpolation routines to pass plasma information from gkeyll to eirene and plasma sources from eirene to gkeyll. The third is the code within Gkeyll itself that applies the sources.

Automatic installation of Eirene + SOLPS-ITER Grid generation

Without modifying the eirene codebase itself, one has to use an existing interface for grid generation and coupling. We are using the SOLPS-ITER (https://github.com/iterorganization/SOLPS-ITER ) interface. The scrip install-deps/build-solps.sh automatically installs the necessary parts SOLPS_ITER into gkylsoft/. This is only tested on perlmutter, as the SOLPS-ITER installation process can be difficult.

Automatic Installation of GK-Neutral_coupling

To actually pass information between Gkeyll and Eirene, post-processing and interpolation routines are required. This is taken care of by the GK-Neutral_coupling repo (https://github.com/jRoeltgen/GK-Neutral_coupling/tree/main) which contains examples and instructions for how to run coupled simulations.

Code Within Gkeyll

With guidance from @manauref, the additions to the code have been written in such a way that they interfere minimally with the rest of the gyrokinetic app. Most of the new code is in gyrokinetic/apps/eirene.c and gyrokinetic/apps/gk_species_source_bgk.c. The second file, gyrokinetic/apps/gk_species_source_bgk.c, has now replaced gk_species_heating.c and retains the old functionality while also adding the ability to apply sources from an external model (EIRENE). I also had to modify array_ops.c with a new function gkyl_array_ceil_range to apply limits on the temperature of the sources come from eirene to improve the stability of simulations.

Input Files

The input file interface is quite simple. You provide

  struct gkyl_gyrokinetic_eirene  eirene = {
    .input_data_path = "./gkeyll_text_input/",          // Place where text files with eirene sources will appear
    .output_data_path = "./gkeyll_text_output/",    // Place where gkeyll will output the new data flag
    .num_coupling_species = 3,                                 // Number of species with eirene sources
    .coupling_species = {"elc", "ion", "molecule"},  // List of species that have eirene sources 
    .injection_time = {0.5e-3, 0.5e-3, 0.5e-4},          // Injection time sets the timescale or rate of the BGK operator
    .core_coll_factor = {75.0, 75.0, 7.5},                   // Factor by which injection_time is reduced in the core
    .damping_factor= {2.0e-1, 2.0e-1, 8.0e-1},       // This is the minimum fraction of the local 
                                                                                     // temperature that the Maxwellian source can have
                                                                                    // Important for stability
    .half_domain = true,
  };

  struct gkyl_gyrokinetic_multib app_inp = {
    .name = "hstep26",
    ...
    ...
    .eirene = eirene,

  };

Extra Fix

This PR also fixes a typo in the field line tracing and then also adds an option to compress cells near the divertor as part of the nonuniform tokamak grids.
The typo was causing some regions where only one root is found (outer PF region for STEP for example) to be traced wrong. The divertor compression is just a useful option.

PDFS with instructions and documentation of the operator:

In this pdf the operator and numerical parameters are described. The document gives guidelines for how to set them.
Eirene_Coupling_Short.pdf

In this pdf we have additional instructions, links to tutorials, etc. for SOLPS-ITER installation and run set up. It also includes some tips about grid generation for Eirene.

SOLPS steps for Perlmutter.pdf

[Maxwell-Rosen]

Maybe I'm ignorant, but I don't know what this b67 (or b1011) means. So if I'm a gkeyll person who doesn't know EIRENE, can we have a more descriptive names for what makes these two files different? We can put a comment that this means the files end in b67, but the function name should make the purpose and operation clear.

[Maxwell-Rosen]

The convention us usually _source_M0.gkyl, so maybe we should keep that for these formats

[Maxwell-Rosen]

Please put braces around single line if statements and for loops

[Maxwell-Rosen]

changing this to mapc2p_int might be clearrer and more organized

[manauref]

Some initial comments:

1. "They exchange data through I/O every time Gkeyll outputs data": does Gkeyll then alter the user's num_frames or I/O triggers so that this exchange happens with some reasonable frequency? because what if the user set t_end =1e-3 and num_frames=2? we'll only have one exchange in a whole milisecond.
2. For the record I heard that EDGE2D couples to Eirene without involving SOLPS. They have their own grid generation tools that they use for Eirene. This would be a preferable route in the future.
3. In the input file description above, can you add comments to the lines with numeric parameters please? you could also reference the corresponding symbol/quantity in your DR/latex write up so one can make the connection, if applicable.
4. Neither the DR, nor the pdf in it, nor the description above explain what core_coll_factor and damping_factor are. Also, greater emphasis/documentation with regards to how coupling_time is chosen should appear above. It might even be good to explain what the coupling_species are.
5. Why is .half_domain = true, in the eirene input file table necessary? I think the code should figure this out under the hood.
6. The GK-Neutral_coupling repo has good README's. It'd be good if you add a "installation" section, that at the very least lists the dependencies (i.e. python modules, fortran?, etc).
7. We wrote a document with a list of steps for installing on Perlmutter. Although much of that is now in scripts in this PR, it might be good to put that (as PDF) around here somewhere.
8. Also, it would be good to attach here a PDF with a high-level description of how to generate the EIRENE grid/files needed. It can be 1 or a half page, with links to the relevant tutorials in the SOLPS/EIRENE modules/repos.
9. It's a bit unfortunate not to have a regression test, though I understand why that's done. That said I think you should:
   9a) add one anyway, and put pre-processor statements around the eirene table so that if someone runs it without eirene, then it just doesn't do that part but still runs.
   9b) run the reg test on your Perlmutter build, on cpu and gpu, and ensure it is valgrind and compute-sanitizer clean.

[akashukla]

Some initial comments:

1. "They exchange data through I/O every time Gkeyll outputs data": does Gkeyll then alter the user's `num_frames` or I/O triggers so that this exchange happens with some reasonable frequency? because what if the user set t_end =1e-3 and num_frames=2? we'll only have one exchange in a whole milisecond.

2. For the record I heard that EDGE2D couples to Eirene without involving SOLPS. They have their own grid generation tools that they use for Eirene. This would be a preferable route in the future.

3. In the input file description above, can you add comments to the lines with numeric parameters please? you could also reference the corresponding symbol/quantity in your DR/latex write up so one can make the connection, if applicable.

4. Neither the DR, nor the pdf in it, nor the description above explain what `core_coll_factor` and `damping_factor` are. Also, greater emphasis/documentation with regards to how `coupling_time` is chosen should appear above. It might even be good to explain what the `coupling_species` are.

5. Why is `.half_domain = true,` in the eirene input file table necessary? I think the code should figure this out under the hood.

6. The [GK-Neutral_coupling](https://github.com/jRoeltgen/GK-Neutral_coupling) repo has good README's. It'd be good if you add a "installation" section, that at the very least lists the dependencies (i.e. python modules, fortran?, etc).

7. We wrote a document with a list of steps for installing on Perlmutter. Although much of that is now in scripts in this PR, it might be good to put that (as PDF) around here somewhere.

8. Also, it would be good to attach here a PDF with a high-level description of how to generate the EIRENE grid/files needed. It can be 1 or a half page, with links to the relevant tutorials in the SOLPS/EIRENE modules/repos.

9. It's a bit unfortunate not to have a regression test, though I understand why that's done. That said I think you should:
   9a) add one anyway, and put pre-processor statements around the eirene table so that if someone runs it without eirene, then it just doesn't do that part but still runs.
   9b) run the reg test on your Perlmutter build, on cpu and gpu, and ensure it is valgrind and compute-sanitizer clean.

1. No we don't modify it. The user just has to set the configuration space output frequency to be reasonable. In the pdf I updated I have added guidelines for this. As discussed in the CEDA meeting, people don't really know how to do it correctly per se.
   3.  Agreed, we want that eventually too.
2. I added descriptions in the input file example and I added guidelines about the numerical parameters in the PDF.  I  listed some physical processes and grid limitations to consider.
   5.  Ok I made it part of gk_geom now.
3. Ok dependencies added to that README in a branch. Will make a new release when we are ready to merge.
4. Ok I added a pdf of that document. It has some additional tips and things for grid generation
5. The attached doc from 7. has links and the order of the steps in general and all that.1. No we don't modify it. The user just has to set the configuration space output frequency to be reasonable. In the pdf I updated I have added guidelines for this. As discussed in the CEDA meeting, people don't really know how to do it correctly per se.
   3.  Agreed, we want that eventually too.
6. I added descriptions in the input file example and I added guidelines about the numerical parameters in the PDF.  I  listed some physical processes and grid limitations to consider.
   5.  Ok I made it part of gk_geom now.
7. Ok dependencies added to that README in a branch. Will make a new release when we are ready to merge.
8. Ok I added a pdf of that document. It has some additional tips and things for grid generation
9. The attached doc from 7. has links and the order of the steps in general and all that.
10. I added rt_gk_multib_step_eirene_2x2v_p1 and added the necessary data to gyrokinetic/data/eirene. compute sanitizer says :
    ========= LEAK SUMMARY: 0 bytes leaked in 0 allocations
    ========= ERROR SUMMARY: 0 errors