The core of the best practices for OpenFOAM at Pawsey is to put some effort in diminishing the amount of files generated by the solvers. Solving a large case with the default file handling of OpenFOAM will generate a large number of small result files. This because each of the processors (ranks) assigned to the different parts of the decomposed domain will write their own output. So, at the end, the amount of files generated by the solver with default file handling is: number of ranks X number of variables X number of output steps (times). So, an example large case with 1024 processors solving 10 variables and saving a 100 time steps will end up with 1,024,000 result files.

OpenFOAM interaction with each of the files triggers in I/O operations such as a read, write, open, close, or stat (returning information about a file). A large number of I/O operations has an impact on both the user's application, as well as other applications running on the system. For this reason, all users have a quota limit of 1 million files on /scratch.

Several of the default options in OpenFOAM cases will result in an increase in I/O operations, and some tips are presented here to help reduce I/O load and improve performance.

Collated file handler

The concept of "collating" OpenFOAM output is: to merge into a single file the results of a variable corresponding to a group of processors (ranks). For example, if the "collating" groups are formed every 32 ranks, then results from processors 0 to 31 are going to be merged together, and results from processors 32 to 63 are going to be merged together, etc. For the example large case mentioned above, when collating every 32 ranks, the number of files to be generated would be: (number of ranks/32) X number of variables X number of output steps resulting in only 32,000 files. This is a much smaller number, which alleviates dramatically the overload of the filesystem caused by I/O operations.

The collated file handler is only available since openfoam-org-6 and openfoam-v1806 versions, therefore we recommend users to port their applications to this versions or later in order to take advantage of this feature. Due to the importance of this feature, we have made the collating file handler the default for the installations provided at Pawsey, contrary to common installations elsewhere.

Settings for the use of the collated option consist of two parts. Part-1 is the indication that the collated option itself is going to be used. Part-2 is the definition of the number of ranks to be used in each collating group.

Settings part-1: Define the file handler with -fileHandler collated

The collated file handler can be indicated in the <case>/system/controlDict file:

//...
OptimisationSwitches
{
	fileHandler collated;
}
//...

Or via the command line when executing all the solvers/tools:

$ decomposePar -fileHandler collated

As mentioned in the note above, this is already the default for Pawsey installations and there is no action needed from the user at Pawsey (unless the user needs to revert to fileHandler uncollated for testing purposes).

Settings part-2: Define the size of the groups of ranks to be collated with FOAM_IORANKS

Use the following syntax to define group of ranks of size G

$ export FOAM_IORANKS='(0 G 2G ... mG)'

For example, for grouping every 32 ranks a case with 256 processsors use:

$ export FOAM_IORANKS='(0 32 64 96 128 160 192 224 256)'

This should be set in your host environment when interacting with OpenFOAM from the command line, or in the Slurm job script before any commands invoking OpenFOAM tools or solvers. The use of groups of 32 ranks is our recommendation for Setonix.

Example of the resulting directory structure

When using the collated option, directory names for the decomposition of a case differ a bit from the usual default in common installations (where uncollated file handler is default). For example, when collating every 32 tasks, the resulting directory structure would look like:

$ export FOAM_IORANKS='(0 32 64 96 128 160 192 224 256)'


$ decomposePar
/*---------------------------------------------------------------------------*\
=========               |
\\      / F ield        | OpenFOAM: The Open Source CFD Toolbox
 \\    /  O peration    | Website: https://openfoam.org
  \\  /   A nd          | Version: 8
   \\/    M anipulation |
\*---------------------------------------------------------------------------*/
Build : 8-30b264cc33cd
Exec : /opt/OpenFOAM/OpenFOAM-8/platforms/linux64GccDPInt32Opt/bin/decomposePar
...
Processor 254: field transfer
Processor 255: field transfer
End

$ ls
0 		constant 				processors256_160-191 		processors256_32-63 		system
0.orig 	processors256_0-31 		processors256_192-223 		processors256_64-95
Allrun 	processors256_128-159 	processors256_224-255 		processors256_96-127

Best practices for OpenFOAM

The open-source CFD software package OpenFOAM is capable of utilizing MPI and is well-suited for use on a parallel system like Setonix. However, there are some features of OpenFOAM that many users overlook that could help improve the performance of their OpenFOAM applications. These are our recommendations:

• Use ~50,000 cells per processor
  As a rule of thumb, we recommend to assign around 50,000 computational cells per processor for parallel simulations. This rule is not written in stone, but it has proven to give good performance on Setonix. From this starting point, users can then test and modify their decomposition looking for the top performance sweet spot which, in our experience, is usually between ~30,000 cells and ~100,000 cells per processor.
  When performing tests, remember to set the case so that it only runs for a fixed and limited number of iterations (ideally up to an estimated maximum of ~15mins).
  And to evaluate performance, remember that  a faster execution is not all that counts, but the amount of Service Units that were used (ExecutionTime X AllocatedCores).
• Time-stepping
  OpenFOAM writes output at each time-step, so a smaller time-step will result in a large number of files being written and I/O operations. Ideally, a user would set their time-step to largest possible for the simulation. However, some of the OpenFOAM solvers can make use of adaptive time-stepping (using larger time-steps when possible and scaling it as need). Several of the parameters in the controlDict dictionary that control adaptive time-stepping are:
  • adjustTimeStep yes|no
    Adjusts the time step, usually based on the Courant number
  • maxCo <val>
    Maximum Courant number to use (e.g., maxCo 0.5)
  • maxDeltaT <val>
    Maximum time step to use (e.g., maxDeltaT 0.1)
• purgeWrite <N>
  This is a controlDict parameter that controls how time directories are stored and overwritten. A low value of N means that only a low number of result files are kept on disk (we recommend to use a low value if possible). For example, with purgeWrite 2, data from time steps will be written into 2 directories, i and i+1. Data from subsequent time steps will overwrite these directories in a cyclical pattern; that is, time step i+2 will write to directory i, time step i+3 will write to directory i+1, etc. For steady-state problems, only 1 directory is necessary and previous time step data can be overwritten, so a value of purgeWrite 1 should be used. (Only if needed, you can turn off the limit with purgeWrite 0.)
• runTimeModifiable no
runTimeModifiable option in controlDict allows the modification of problem parameters while the simulations are running. But we strongly recommend to set it to no. OpenFOAM simulations tend to have a large overhead in the startup phase; the runTimeModifiable flag is a major factor in this behavior. Turning off this option can greatly reduce this initial overhead and result in faster simulations.
• writeFormat binary
  Writing output in binary format is faster than writing in ASCII and users are encouraged to use binary. OpenFOAM defaults to writing output in ASCII format. While this allows users to easily read and edit files, it does affect performance (particularly with large simulations).  Set in controlDict.
• -fileHandler collated + export FOAM_IORANKS='(0 32 64 96 ... totalRanks)'
  Useful for reducing the number of output files (explained above).
• File curation
  Users are strongly encouraged to delete files on /scratch when they are no longer needed, rather than simply wait for file purging to remove them.  Because of OpenFOAM's popularity on our systems, it's possible for our users to generate hundreds of millions of small files, which is difficult for our purge program to keep up with.  As a result, unnecessary files will continue to remain on the system.  More information on how to remove files with munlink can be found here.

Related pages

• OpenFOAM
• OpenFOAM: Example Slurm Batch Scripts
• OpenFOAM: Additional topics on containers
• Containers