Comparison with zarr, N5

[clbarnes]

There is quite a lot of interest in this kind of dataset-on-disk storage and I was wondering if you'd come across N5 and Zarr. They have very similar goals and layout on disk, with the addition of chunking large arrays (just like HDF5) and suitability for generic key-value stores (e.g. S3 buckets, redis). There's a lot of discussion around them and they are currently trying to converge on a spec (zarr v3) to cover everyone's use cases.

N5 was designed for volumetric microscopy for neuroscience, and is popular in the java/imageJ community; zarr has caught on more with e.g. netCDF.

[dragly]

Hi @clbarnes! Yes, I was just looking into Zarr the other day. There is an issue on their side about looking into similarities and collaboration. See my most recent comment there: zarr-developers/community#6 (comment)

In short, I think the specification of Exdir is a bit simpler, which is an attractive property in itself, but Zarr has some really good momentum. And Zarr's spec is also pretty simple already - perhaps as simple as it should be? It might make sense to add that small amount of complexity at the benefit of features like chunked storage.