Increase default PnetCDF header size to avoid I/O hangs

[amstokely]

This PR increases the default Parallel NetCDF (PnetCDF) header size to 128 KB to reduce the likelihood of header reallocation during MPAS I/O. In certain situations, when MPAS overwrites existing string attributes or variables with larger values, the NetCDF header can grow beyond its preallocated padding, requiring PnetCDF to reallocate the header during ncmpi_enddef, which can lead to an I/O hang. This behavior was identified as the root cause of the hang reported in MPAS-Workflow issue #384. By increasing the default header size, this PR decreases the likelihood that header reallocation is required when string attributes or variables are overwritten with larger values. Preliminary testing of the calculation that previously triggered the hang indicates that this change resolves the issue without impacting calculation results or I/O performance.

Fixes #1385

[wkliao]

(Sorry to jump into the discussion.)
There is another way to do this without creating an MPI_Info object, which is to call ncmpi__enddef(), i.e.

ierr = ncmpi__enddef(file->ncidp, 0, 131072, 0, 0);

You can also set argument h_minfree, which ensures at least this amount of free space in the header section.

FYI. When adding new data objects into an existing file which causes the file header section to grow, PnetCDF must move the data section to a place with a higher file offset, which can be expensive, especially when the size of exiting file is large. The application program most likely just ran very slowly, but not hanging.

[mgduda]

@wkliao Thanks very much for the suggestion to consider ncmpi__enddef as an alternative to passing a hint to ncmpi_create. I do like that this approach would require fewer lines of new code.

If we were to open an existing file for writing and call ncmpi___enddef (even if we didn't define any new dimensions or variables), would a header reallocation always be forced in case the original header was less than the size of the v_align argument to nmcpi__enddef?

[amstokely]

(Sorry to jump into the discussion.) There is another way to do this without creating an MPI_Info object, which is to call ncmpi__enddef(), i.e.

ierr = ncmpi__enddef(file->ncidp, 0, 131072, 0, 0);

You can also set argument h_minfree, which ensures at least this amount of free space in the header section.

FYI. When adding new data objects into an existing file which causes the file header section to grow, PnetCDF must move the data section to a place with a higher file offset, which can be expensive, especially when the size of exiting file is large. The application program most likely just ran very slowly, but not hanging.

@wkliao This issue only surfaced when running MPAS through MPAS-JEDI under very specific conditions, which made it difficult to track down. Is there a check you’re aware of that we could have put in place to make this easier to catch?

[wkliao]

Let me use two PnetCDF terminologies to help explain.

• "header extent" is referred to as the size of file header section and
• "header size" is the true size of header (i.e. metadata).

Subtracting the two gives you the free space available in the header section.

If we were to open an existing file for writing and call ncmpi___enddef (even if we didn't define any new dimensions or variables), would a header reallocation always be forced in case the original header was less than the size of the v_align argument to nmcpi__enddef?

In this case, PnetCDF will check if the file extent of the existing file is aligned with v_align. If it is not, PnetCDF will move the data section to a high file offset that is aligned with v_align.

Please note that v_align is not the size of file extent. It is used to align the header extent to a multiple of v_align. For example, say the file header size is 2000 bytes and v_align is 1024, then the file extent will become 2048.

[wkliao]

Is there a check you’re aware of that we could have put in place to make this easier to catch?

Two PnetCDF APIs can be used to query the header extent and header size of an existing file. They can also be called after ncmpi_enddef():

• ncmpi_inq_header_extent()
• ncmpi_inq_header_size()

If the free space is sufficiently large, then just call ncmpi_enddef().
Otherwise, moving the data section is inevitable.