Line Data Format using HDF5

no image

Rybno Geo Data is using perhaps the most advanced line database in the world of airborne geophysics.

One of the first decision the author had to make when embarking on a software package for processing of data from airborne geophysics was the format of line data. As the generation of maps was to be done with the Generic Mapping Tools, the formats to be used there were considered. Because of the very superior efficiency of the considered design which was finally chosen, a format based on HDF5 was developed. The author also notices that many modern software developments which have higher demands in data storage and retrieval are often moving in the same direction.

A more detailed description is available as PDF document for download (222,481 Bytes) .  

The image on the right shows a screenshot of the Java program HDFView which is freely available for viewing the content of HDF5 files. It shows clearly the hierarchical structure of the data sets which are contained in the file, here with the name rad12na.h5. There is an internal path reflecting the hierarchical structure, starting with rad/flights/. Under flights appear all channels of all flights starting with flight number 101 etc. The channel of potassium counts, K_cts, is then shown containing one version of the channel with number 0. The version C is not a separate version but just an alternative link, here to version 0, but could point to any other version should they exist, signifying the finally selected default version.

What makes the use of HDF5 so attractive is their efficiency and the possibility to add also attributes (named parameters of almost any type) to any link and data set. In the image on the right shows the red-coloured "A" in the lower right of the folder symbol that there are attributes, here for Revision and Version. Flights have the attributes "Doy" for day of the year and "Year". All data relevant for the project can be added to one file, either as attributes or in separate data sets, although several files may be used covering possibly different processing streams like magnetometer data and radiometric data. This makes this format especially attractive in a professional environment where many projects must be managed simultaneously.

The data can be compressed before written to the file, further reducing i/o volume. Rybno Geo Data uses normally a moderate level of compression. Data are generally stored in well defined chunks which can directly be accessed with the use of B-trees. All this ensures the superior performance of this database.