Problems with native Tensorflow distribution strategies

The new software stack (August 2025) provides two new tensorflow modules updated to be compatible with the new GPU driver:

tensorflow/2.15.1-rocm6.3.3 (Horovod is only available in this module.)
tensorflow/2.17-rocm6.3.3

Unfortunately, the native strategy for multi-node multi-GPU distributed training (tf.distribute.MultiWorkerMirroredStrategy()) resulted to be unstable and is showing random failures with both modules! (We are currently investigating how to solve these issues.)

Therefore, if your workflow requires multi-node multi-GPU distributed training, the only stable option is to keep using Horovod for this purpose (only available in the 2.15.1 module).

For single node multi-GPU distributed training, the native strategy tf.distribute.MirroredStrategy() works fine for both modules.

Single GPU training works fine for both modules.

TensorFlow