# ARC

0. About

   Adaptive Ray-tracing with CUDA (ARC) is an Astronomy module designed to efficiently track radiation in a cosmological volume and its effects on the ionization state of the gas using the power of CUDA to take advantage of the power of multi-level parallelization (multiple parallel GPUs, each with thousands of parallel cores). The version of the code presented here is designed to track a cube of fixed sized over any given period of cosmic time on a fixed grid with size 2^n. (Current version tracks (10 Mpc)^3 starting at z=30 with PLANCK parameters, but will soon be updated to allow freedom of choice for these parameters).

   In depth code description and accuracy tests are presented in full detail here:

   ARC: adaptive ray-tracing with CUDA, a new ray tracing code for parallel GPUs
   https://arxiv.org/pdf/1807.07094.pdf

1. Begin by cloning this repository:

   > git clone https://github.com/blakehartley/ARC

2. Make sure that you have versions of openMPI and CUDA which are mutually compatible installed/loaded. On a supercomputer (e.g. Zaratan at UMD), this is accomplished with:

   > module unload <unwanted modules>
   >
   > module load openmpi cuda

   HPC managers will typically ensure that the default versions of these modules are mutually compatible.

3. Once you are sure that openMPI and CUDA are installed and/or loaded, the Makefile for ARC needs to be updated for the location of CUDA include files. This is accomplished using:

   > './configure'

4. ARC uses a Makefile located in the './src/' directory for the managing the directory structure and compiling of the code. Navigate to this folder with `cd src`. The following codes may be used as desired in the folder './src/':

   - 'make' - This will compile the code, creating the necessary object files and binary, which is located in the directory './bin/'.
   - 'make clean' - This will remove intermediate compilation files. It should be used if you want to make a change to any source code file besides arc_host.cpp.

   - 'make cleandat' - This will clean out the data folders entirely. It is ideal for cleaning up unnecessary work.

5. Initialize

   Before the code may be run, several parameters should be inspected and changed, as desired. These parameters are located in two places:

   './parameters.txt' - These parameters are designed to be changed between runs. These include the location of input density and halo files, bursty flag, time step type, simulation duration, output number to start on, time between outputs, escape fraction, and halo mass cutoff.

   './src/arc.h' - These parameters are designed to be fixed between runs, and require a compile to change. These include grid dimension, number of frequency bins, number of species, initial temperature of the gas, number of time slices between bursts, helium fraction, memory buffer size, ray splitting condition, and boundary conditions.

   In general, 'parameters.txt' should be the only one that requires a change, and the code should run once the grid and halos are correctly pointed to.

6. The current version of the code is designed to be run on a multiple of 8 parallel GPUs, using MPI and CUDA. The code is run with a command such as the following:

   mpirun -n 8x ./bin/arc

   where x is some integer, as desired. An example of the type of SLURM script used for running the code is included in './run.csh'. This script is one that I have used to run it on both UMD's Deepthought2 and JHU's Bluecrab. It is possible to run the code on a smaller or larger number, or non-multiple of 8, and MPI will take care of over/underloading GPU's as necessary. However, an integer multiple of 8 will ensure optimal saturation of computational resources (i.e. at 15 nodes, the final 16th sub-volume will take its own computational cycle, making the whole code hang up).

7. Code outputs

   The outputs of the code are stored in the './dat/' folder. The code produces the following outputs:

   './dat/sum.dat' - This file contains a summary of various parameters at each time step of the code. TODO: './dat/sum.info' contains a description of each of the parameter columns of sum.dat.

   './dat/hydrogen/' - 

   './dat/helium/' - 

   './dat/photons/' - 

   './dat/restart/' - 

8. Contact the author

   I may be contacted at blakehartley@gmail.com for any questions regarding this code.

   I wrote ARC from the ground up for my own use in my thesis work, and as such the code is self-contained and designed to work with the files I had at the outset of the project. However, I tried to keep the code as flexible and modular as possible, so it shouldn't be too difficult to apply it to different file types, or to utilize pieces of the code (the CUDA ray-tracing, CUDA ionization integrator, halo-matching algorithm, etc.) for different codes with comparable structure. CUDA is a uniquely powerful framework for these types of computations, and I believe that any large scale project with these types of calculations would benefit from the utilization of CUDA.