...
The open-source CFD software package OpenFOAM is capable of utilizing MPI and is well-suited for use on a parallel system like Setonix. However, there are some features of OpenFOAM that many users overlook that could help improve the performance of their OpenFOAM applications. These are our recommendations:
- Use ~50~60,000 cells per processor
As a rule of thumb, we recommend to assign around 5060,000 computational cells per processor for parallel simulations. This rule is not written in stone, but it has proven to give good performance on Setonix. From this starting point, users can then test and modify their decomposition looking for the top performance sweet spot which, in our experience, is usually between ~30~40,000 cells and ~100,000 cells per processor.
When performing tests, remember to set the case so that it only runs for a fixed and limited number of iterations (ideally up to an estimated maximum of ~15mins).
And to evaluate performance, remember that a faster execution is not all that counts, but the amount of Service Units that were used (ExecutionTimeXAllocatedCores). - Time-stepping
OpenFOAM writes output at each time-step, so a smaller time-step will result in a large number of files being written and I/O operations. Ideally, a user would set their time-step to largest possible for the simulation. However, some of the OpenFOAM solvers can make use of adaptive time-stepping (using larger time-steps when possible and scaling it as need). Several of the parameters in thecontrolDictdictionary that control adaptive time-stepping are:adjustTimeStep yes|no
Adjusts the time step, usually based on the Courant numbermaxCo <val>
Maximum Courant number to use (e.g., maxCo 0.5)maxDeltaT <val>
Maximum time step to use (e.g., maxDeltaT 0.1)
purgeWrite <N>
This is acontrolDictparameter that controls how time directories are stored and overwritten. A low value of N means that only a low number of result files are kept on disk (we recommend to use a low value if possible). For example, withpurgeWrite 2, data from time steps will be written into 2 directories, i and i+1. Data from subsequent time steps will overwrite these directories in a cyclical pattern; that is, time step i+2 will write to directory i, time step i+3 will write to directory i+1, etc. For steady-state problems, only 1 directory is necessary and previous time step data can be overwritten, so a value ofpurgeWrite 1should be used. (Only if needed, you can turn off the limit withpurgeWrite 0.)- r
unTimeModifiable norunTimeModifiableoption incontrolDictallows the modification of problem parameters while the simulations are running. But we strongly recommend to set it tono. OpenFOAM simulations tend to have a large overhead in the startup phase; therunTimeModifiableflag is a major factor in this behavior. Turning off this option can greatly reduce this initial overhead and result in faster simulations. writeFormat binary
Writing output in binary format is faster than writing in ASCII and users are encouraged to use binary. OpenFOAM defaults to writing output in ASCII format. While this allows users to easily read and edit files, it does affect performance (particularly with large simulations). Set incontrolDict.-fileHandler collated+export FOAM_IORANKS='(0 32 64 96 ... totalRanks)'
Useful for reducing the number of output files (explained above).- File curation
Users are strongly encouraged to delete files on/scratchwhen they are no longer needed, rather than simply wait for file purging to remove them. Because of OpenFOAM's popularity on our systems, it's possible for our users to generate hundreds of millions of small files, which is difficult for our purge program to keep up with. As a result, unnecessary files will continue to remain on the system. More information on how to remove files with munlink can be found hereCheck the information about Deleting Large Numbers of Files on Lustre Filesystems.