How to improve HDF5 I/O(write file) efficiency?

fortran, hdf5

Solution

It seems that you are fragmenting your data into groups for each time step (I'm just guessing from what you wrote). It may be more efficient to add an additional dimension to all of your datasets which would represent the time step and get rid of the groups because you could buffer a bunch of iterations together before each write.

In clear, instead of:

/time-1-group
    /time-1-group/DataSetA -> 2d array
    /time-1-group/DataSetB -> 2d array
    ...
/time-2-group
    /time-2-group/DataSetA -> 2d array
    /time-2-group/DataSetB -> 2d array
    ...
...

you would have this:

/DataSetA -> 3d array where third index is time
/DataSetB -> 3d array where third index is time
...

You would have to use chunked datasets and select the chunk size with care to optimize I/O efficiency (and as I said above you could have more than one time step per chunk).

Problem

I have very much time-related scientific data to write, means data should be written into hdf5 file every several seconds. My hdf5 file structure is designed below: - create many time group, like time-1-group, time-2-group, time-3-group, and etc... - In time group, many dataset are created, like DataSetA, DataSetB, DataSetC, and etc... - Write data into dataset above. API used: HDF5-Fortran Run this program, everything is ok, but speed is slow, How to improve hdf5 write action efficiency? Thank you very much.

Original source