Python C Bindings For Supporting Rapid Development

Often times for rapid development of low level code it can be beneficial to leverage a language such as Python. To support quickly (ie a few hours) developing and testing a CUDA C implementation of a computationally intensive algorithm Garrett Wright of the GFDL team leveraged this approach.

To keep it simple I will assume we have some C/C++/Fortran or any language that can be incorporated by a standard C header file. Here is mine:

1. Get/make the shared library. Here I'll just leverage the header defining the highest function I need as a C function to exemplify it really doesn't matter what is underneath this. It was CUDA C, but could have been CUDA Fortran, Fortran, C++, or /any/ language with a path to standard C bindings like this...

   /* Authored by Garrett.Wright@noaa.gov for GFDL */ #ifndef SET_ABSORPTION_COEF_H_ #define SET_ABSORPTION_COEF_H_

   typedef double REAL_t;

   #ifdef __cplusplus extern "C" #endif int naive_launch(int nL, int nF, REAL_t* out, REAL_t* f, REAL_t T, REAL_t P, REAL_t* Vnn, REAL_t* Snn_ref, /REAL_t A,/ REAL_t Yair, REAL_t* Yself, REAL_t* En, REAL_t* n, REAL_t* d, REAL_t* glo, REAL_t* ghi, REAL_t* Ps);

   #endif

2. Leverage python standard library packages platform and ctypes to runtime link/load your C library into Python.

   /* Authored by Garrett.Wright@noaa.gov for GFDL */

   lib

   if platform.system()=='Microsoft': print "wut" exit(666) elif platform.system()=='Darwin': libA = ctypes.cdll.LoadLibrary('./libabsorptionCoef.so') elif platform.system()=='Linux': libA = ctypes.cdll.LoadLibrary('./libabsorptionCoef.so') else: print "what is this? attempting anyway....", platform.system() libA = ctypes.cdll.LoadLibrary('./libabsorptionCoef.so')

3. We also use a powerful feature of numpy, namely numpy.ctypeslib.ndpointer, that exposes the contiguous C (or F !) stored array as a pointer.

   /* Authored by Garrett.Wright@noaa.gov for GFDL */

   define C binding

   _naive_launch64 = libA.naive_launch _naive_launch64.restype = ctypes.c_int _naive_launch64.argtypes = [ctypes.c_int, ctypes.c_int, numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), ctypes.c_double, ctypes.c_double, numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS'), numpy.ctypeslib.ndpointer(dtype=numpy.float64,flags='C_CONTIGUOUS') ]

4. By writing a simple function, we allow Python to attempt type inference (ie make life way easier). This is also a practical place to decorate with boiler plate checks etc etc.

   /* Authored by Garrett.Wright@noaa.gov for GFDL */

   map

   def naive_launch64(nL, nF, out, f, T, P, Vnn, Snn_ref, Yair, Yself, En, n, d, glo, ghi, Ps): return _naive_launch64(nL, nF, out, f, T, P, Vnn, Snn_ref, Yair, Yself, En, n, d, glo, ghi, Ps)

5. Then I wrote a Python wrapper that called simple functions to setup, run the (CUDA) C code, save, and display results. Such code can also be imported and leveraged interactively, a very powerful feature for rapid development/testing.

   /* Authored by Garrett.Wright@noaa.gov for GFDL */ #

   python wrapper

   def cuTest(Lry,Temp,Pres,bw=1024,spc=1): # easily map an multidimensional array into sub arrays M,Vnn,Snn_ref,A,Yair,Yself,Elo,n,d,ghi,glo = Lry # and get meta data
   numberOfLines = Lry.shape[1]

    #generate a sample space
    F = numpy.linspace(0,bw,num=bw/spc,endpoint=True).astype(numpy.float64)                                                                                                                                                                                           
    numberOfFreqSamples = F.shape[0]
    #create a zeroed array to store our outputs
    out = numpy.zeros(F.shape[0],dtype=F.dtype)
   
    # For testing we supply a value that would be supplied from the model
    #  (along with a Temp and Pres which are supplied to this function)
    #  We assume this is global, but the array would allow for spatial fluctuations in future
    Ps_co2=0.00039
    Ps = numpy.ones(numberOfLines).astype(numpy.float64)*Ps_co2
   
    print "Invoking C (CUDA) library"
    res = naive_launch64(numberOfLines, numberOfFreqSamples, out, F,     # setup in wrapper                                                                                                                                       
                         Temp, Pres,                                     # supplied                                                                                                             
                         Vnn, Snn_ref, Yair, Yself, Elo, n, d, glo, ghi, # from HITRAN database, which I've used python to quickly parse and pack into numpy arrays (not shown)                                                                                                                               
                         Ps)
    if res==0:
        print "returned from C code with succes"
    else:
        print "code went boom, C returned errcode %d" %(res)
        exit(1)
   
    putResult(F,out,numberOfLines) # a function to write output as file (arbitrary, not shown)
   
    matplotlib.pyplot.plot(F,out) #generate plot of our result
    matplotlib.pyplot.show()      # display plot                                                                                                     
   
    return
    #<snip>
   

6. Enjoy the syntactic sugar of python with your bare-metal code under the hood :)