{"type":"doc","content":[{"type":"bodiedExtension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt","parameters":{"macroParams":{"name":{"value":"WarningAfterAugust2025"},"atlassian-macro-output-type":{"value":"INLINE"}},"macroMetadata":{"macroId":{"value":"d528bcc2-5aa4-4d80-a1d1-a19218f7c865"},"schemaVersion":{"value":"1"},"title":"Excerpt"}},"localId":"eb81516c-5f10-4594-b956-8f34f3cf9d06"},"content":[{"type":"extension","attrs":{"extensionType":"com.atlassian.confluence.migration","extensionKey":"legacy-content","text":"","parameters":{"cxhtml":"900

Problems with native Tensorflow distribution strategies

The new software stack (August 2025) provides two new tensorflow modules updated to be compatible with the new GPU driver:

tensorflow/2.15.1-rocm6.3.3 (Horovod is only available in this module.)
tensorflow/2.17-rocm6.3.3

Unfortunately, the native strategy for multi-node multi-GPU distributed training (tf.distribute.MultiWorkerMirroredStrategy()) resulted to be unstable and is showing random failures with both modules! (We are currently investigating how to solve these issues.)

Therefore, if your workflow requires multi-node multi-GPU distributed training, the only stable option is to keep using Horovod for this purpose (only available in the 2.15.1 module).

For single node multi-GPU distributed training, the native strategy tf.distribute.MirroredStrategy() works fine for both modules.

Single GPU training works fine for both modules.

","nestedContent":{"type":"doc","content":[{"type":"bodiedExtension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"column","parameters":{"macroParams":{"width":{"value":"900"}},"macroMetadata":{"macroId":{"value":"08c20e5f-de9a-4e7d-9bc3-252c160dd7a0"},"schemaVersion":{"value":"1"},"title":"Column"}}},"content":[{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Problems with native Tensorflow distribution strategies","type":"text"}]},{"type":"paragraph","content":[{"text":"The new software stack (August 2025) provides two new tensorflow modules updated to be compatible with the new GPU driver:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"tensorflow/2.15.1-rocm6.3.3","type":"text","marks":[{"type":"code"}]},{"text":" (Horovod is only available in this module.)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"tensorflow/2.17-rocm6.3.3","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Unfortunately, the native strategy for multi-node multi-GPU distributed training (","type":"text","marks":[{"type":"strong"}]},{"text":"tf.distribute.MultiWorkerMirroredStrategy()","type":"text","marks":[{"type":"code"}]},{"text":") resulted to be unstable and is showing random failures with both modules! ","type":"text","marks":[{"type":"strong"}]},{"text":"(We are currently investigating how to solve these issues.)","type":"text"}]},{"type":"paragraph","content":[{"text":"Therefore, if your workflow requires multi-node multi-GPU distributed training, the only stable option is to keep using Horovod for this purpose (only available in the ","type":"text"},{"text":"2.15.1","type":"text","marks":[{"type":"code"}]},{"text":" module).","type":"text"}]},{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"single node","type":"text","marks":[{"type":"em"}]},{"text":" multi-GPU distributed training, the native strategy ","type":"text"},{"text":"tf.distribute.MirroredStrategy()","type":"text","marks":[{"type":"code"}]},{"text":" works fine for both modules.","type":"text"}]},{"type":"paragraph","content":[{"text":"Single GPU training works fine for both modules.","type":"text"}]}]}]}],"version":1},"reason":"A column macro in a rich body macro can't be created or edited in the new editor."}}}]},{"type":"bodiedExtension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt","parameters":{"macroParams":{"atlassian-macro-output-type":{"value":"INLINE"}},"macroMetadata":{"macroId":{"value":"8d6970c6-317c-45b7-ae7e-47a441642e78"},"schemaVersion":{"value":"1"},"title":"Excerpt"}},"localId":"16a2ddf1-e93c-41cc-888d-5105d53886da"},"content":[{"type":"paragraph","content":[{"text":"TensorFlow is an open-source software library for machine learning.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"By taking advantage of multiple GPU nodes, it is possible to run machine learning and deep learning tasks at scale. Pages in this section of the Pawsey documentation will cover details on how to do it.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"bodiedExtension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"section","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"3a5b8f57-9014-4698-93f1-cca2d120d4af"},"schemaVersion":{"value":"1"},"title":"Section"}},"localId":"0a6be0a4-b100-4749-822b-798214af2605"},"content":[{"type":"table","attrs":{"layout":"default","localId":"da7088de-3c67-444c-9ba2-64964e4680fc"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Pages in this section:","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"pagetree","parameters":{"macroParams":{"root":{"value":"2025.TensorFlow"}},"macroMetadata":{"macroId":{"value":"935ad3fc-d2ca-4194-aaf3-52dc8b27badf"},"schemaVersion":{"value":"1"},"title":"Page Tree"}},"localId":"71c79708-d5f5-40c2-b16e-c61e0f3923c2"}}]}]}]}]}]}],"version":1}