Need a unified source of standard netCDF attributes

[mhidas]

Many of our processes that create netCDF files require templates to set global (and variable) attributes. Currently this is being done in a variety of different ways, from multiple sources, often setting the same basic attributes (project, acknowledgements, etc..):

• the Matlab Toolbox has text files like this for global attributes, and imosParameters.txt for variable attributes
• IMOSnetCDF.py uses text files like this for both global and variable attributes. The format used is consistent with CDL, though the files are not complete CDL files. (It also reads a copy of the imosParameters.txt file from the Toolbox, which is out of date now...)
• Loz has a generate_netcdf_att.py script which reads global and variable attributes from a config file
• the ACORN current_generator code simply sets attributes in a acorn_constants.py file.

There may be others too...

There are two issues here:

1. We have redundant code doing the same thing in different ways.
2. We have redundant versions of the same standard global attributes in several locations.

It would be helpful to come up with a solution that removes, or at least minimises, both of these issues.

[mhidas]

In my opinion CDL would be the most appropriate format to use for these templates. These can be very easily converted into a netCDF file with ncgen and then opened as a NetCDF4 Dataset object.

As for the location, something like data-services/lib/netcdf/templates would make sense. The tricky bit will be how to make this available to the Matlab Toolbox as well...

[lbesnard]

converted into a netCDF file with ncgen

the only issue with this is we would have to do a subprocess call from python, which is never an ideal solution.

[mhidas]

Format options for storing netCDF attributes

There are many ways fixed attribute values (such as the license, disclaimer and acknowledgement strings required by the IMOS conventions) could be stored. The idea is to keep these separate from code that creates netCDF files. Here are some options

CDL (Common Data Language)

Currently, IMOSnetCDF.py in data-services uses text files like this, which are not complete CDL files, but they store global and variable attributes the same way as CDL.

Pros

• CDL is the standard text representation of a NetCDF file. Some existing tools (e.g. nco) can already read/write it.
• Besides attributes, it can also represent the file structure (dimensions, variable types) and include actual data values.
• Already have code in data-services to parse attribute definitions (https://github.com/aodn/data-services/blob/master/lib/python/IMOSnetCDF.py)

Cons

• No standard Python package to read in CDL directly (cdlparser could be a starting point, but doesn't seem to be actively maintained)

conf (Windows INI syntax)

e.g. https://github.com/aodn/data-services/blob/master/SOOP/SOOP_XBT/DELAYED/generate_nc_file_att

Pros

• Simple human-readable text format
• Standard Python package configparser can read them
• Already have code to add attributes from these files to an open NetCDF4.Dataset object (https://github.com/aodn/data-services/blob/master/lib/python/generate_netcdf_att.py).

Cons

• Can't specify data type of an attribute, so tricky to create numeric attributes, or string attributes that look like numbers (see e.g. SRS OC BODBAW - change generate_netcdf_att library to handle case whe… #846)

json

Pros

• Standard Python package json to read files
• Handles attribute types (can distinguish between string, numeric and even lists).
• Hierarchical structure could be very similar to Python NetCDF object

Cons

• Need to re-write existing attribute files.

IMOS Toolbox

e.g.
global attributes: https://github.com/aodn/imos-toolbox/blob/master/NetCDF/template/global_attributes_timeSeries.txt
variable attributes: https://github.com/aodn/imos-toolbox/blob/master/IMOS/imosParameters.txt

Pros

• Already maintained & up to date (in imos-toolbox repo).
• Already have code in IMOSnetCDF.py to pares imosParameters.txt (variable attributes only)
• Same attributes used in all ANMN data.

Cons

• Toolbox-specific functionality (e.g. getting values form deployment db), which would need to be ignored for non-Toolbox use.
• Need to write new Python code to parse global attributes file.
• Stored in a separate repo.

Python

Attribute values can be specified directly in a Python module and simply imported, e.g. https://github.com/aodn/data-services/blob/master/ACORN/current_generator/acorn_constants.py

Pros

• Handles data types, i.e. can specify string, numeric or list attributes.
• Flexible, can include calculated values.
• No parsing required.

Cons

• (Using code to store configuration?)

Other possibilities

• NcML - like CDL but in XML format.
• YAML
• TOML

[mhidas]

@smancini @lbesnard @ggalibert @bpasquer To be discussed.

[ocehugo]

@mhidas,

I've a code that read cdl from python.

It need some refactoring into regex (its pretty primitive). It doens't support groups, but there are a battery of tests from the unidata website cdls and is pure python using only the re module.

It works for my needs but I would like to extend with more regex and support groups.

Let me know if you wanna have a go.

PS: there is also TOML.

[lbesnard]

https://github.com/rockdoc/cdlparser
https://github.com/powellb/seapy/blob/e8fcdcc72fe9203e36e357e1fde43270176d54d6/seapy/cdl_parser.py

[lbesnard]

After doing a bit of testing, I'm of the opinion that the https://github.com/rockdoc/cdlparser code could be what we are after.
It parses CDL files, generates a netcdf file or keep the netcdf object open in python.

example:

1. in bash

ncdump http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-SST/9HA2479_Pacific-Sun/2010/IMOS_SOOP-SST_MT_20101212T120000Z_9HA2479_FV01_C-20120528T071954Z.nc > /tmp/soop.cdl

2. in python

from cdlparser import CDL3Parser

myparser = CDL3Parser(close_on_completion=False, file_format='NETCDF4_CLASSIC')
ncdataset = myparser.parse_file("/tmp/soop.cdl",ncfile="/tmp/soop.nc")
ncdataset.close()

However the generated netcdf file and the orginal cdl ends up missing some attributes (and maybe other things). The code would have to be forked an improved

[bpasquer]

When I was working on generating files with the netcdfgenerator 2 years ago, I remember having issues to generate NetCDF4 file using CDL It's probably been improved since then and so If the cdlparser meets our needs like @lbesnard is suggesting (but further tests are needed), using CDL is the best option in my opinion.

[ocehugo]

if you are going to use https://github.com/rockdoc/cdlparser, check if they implemented python3 support.

I did a pr to support python 3 sometime ago. It didn't got into upstream because my pr was disorganized . My ode already worked fine for my case, so i didn't push forward. I couldn't remember much why I did a new code...maybe something cdlparser was failing, or I was not aware of it at the time, or I was too optmistic to finish my code with groups support and all the bells.

Anyway, I don't see a problem with raw python code either, but json is a clear win if you plan to read those things over the wire.

[lbesnard]

re https://github.com/rockdoc/cdlparser, I was having issues with the generated file missing many attributes, data. This is because I used file_format='NETCDF4_CLASSIC' instead of file_format='NETCDF4'. Apparently a bug from the netcdf library.

So, doing the following (see below), will create a correct NetCDF file from a CDL format. Will keep on testing with other datasets:

1. in bash

ncdump http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-SST/9HA2479_Pacific-Sun/2010/IMOS_SOOP-SST_MT_20101212T120000Z_9HA2479_FV01_C-20120528T071954Z.nc > /tmp/soop.cdl

2. in python

from cdlparser import CDL3Parser

myparser = CDL3Parser(close_on_completion=False, file_format='NETCDF4')
ncdataset = myparser.parse_file("/tmp/soop.cdl", ncfile="/tmp/soop.nc")
ncdataset.close()

3. bash, to check the diff between the new netcdf and the cdl file

diff /tmp/soop.cdl < (ncdump /tmp/soop.nc)

[lbesnard]

check if they implemented python3 support.

No They didn't but we use python 2.7 anyway