...
Python is available on all Pawsey systems. Developers of Python codes should pay special attention to computational performance and always consider using available high-performance Python tools. This section of the documentation includes the description of selected Python libraries and extensions which are particularly useful for increasing performance of a Python code.
Versions installed in Pawsey systems
To check the current installed versions, use the module avail command (current versions may be different from content shown here):
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 1. Checking for installed versions |
|---|
| $ module avail python
--------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages ---------------------
python/3.9.15 python/3.11.6 (D)
--------------------------------------- /opt/cray/pe/lmod/modulefiles/core ----------------------------------------
cray-python/3.9.13.1 cray-python/3.10.10 (D) |
|
We recommend the use of python installed by Pawsey instead of cray-python.
Many scientific and managing tools have been also installed in our software stack. These are named with the prefix py-, and sometimes their flavour is also identified by a suffix -py<VERSION> after the version corresponding to the tool itself (current versions may be different from content shown here):
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 1. Checking for installed versions |
|---|
| $ module avail py-
---------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications -----------------------
py-astropy/4.2.1 py-astropy/5.1 (D) py-emcee/3.1.1 py-funcsigs/1.0.2
--------------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities ---------------------------
py-boto3/1.26.26 py-setuptools/59.4.0-py3.9.15 py-setuptools/68.0.0-py3.11.6 (D)
py-pip/23.1.2-py3.9.15 py-setuptools/59.4.0-py3.11.6
py-pip/23.1.2-py3.11.6 (D) py-setuptools/68.0.0-py3.9.15
------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages ------------------------
py-cython/0.29.36 py-ipython/8.14.0 py-numpy/1.24.4 py-scikit-learn/1.3.2
py-cython/3.0.4 (D) py-matplotlib/3.8.1 py-numpy/1.26.1 (D) py-scipy/1.11.3
py-dask/2023.4.1 py-mpi4py/3.1.5-py3.11.6 py-pandas/1.5.3
py-h5netcdf/0.10.0 py-netcdf4/1.6.2 py-pandas/2.1.2 (D)
py-h5py/3.8.0 py-numba/0.57.0 py-plotly/5.14.1
------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools ------------------------
py-hatchet/1.3.1 |
|
Parallel programming
Code developers and users using Python are not limited to running serial applications. Modern Python libraries exist that enable the use of distributed-memory and shared-memory parallelisation. The optimal parallelisation approach is highly application- and architecture-dependent. Two well-known Python parallelisation packages supported at Pawsey are mpi4py and Cython/OpenMP.
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 1. Run mpi4py code |
|---|
| $ salloc -N 1 -n 4 -p debug -t 0:01:00
$ module load python/3.9.7<VERSION> py-numpy/1.20.3<VERSION> py-mpi4py/3.1.2<VERSION>
$ srun -n 4 python mpi.py
Rank 0 has sum of ranks [6]; Answer = 6
Rank 2 has sum of ranks [6]; Answer = 6
Rank 1 has sum of ranks [6]; Answer = 6
Rank 3 has sum of ranks [6]; Answer = 6 |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 2. Build a Cython/OpenMP object |
|---|
| $ salloc -N 1 -n 1 -p debug -t 0:05:00
$ module load python/3.9.7<VERSION> py-cython/<VERSION>
$ srun python setup.py build_ext --inplace |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 3. Run Cython/OpenMP code |
|---|
| $ salloc -N 1 -n 1 -c 4 -p debug -t 0:01:00
$ module load python/3.9.7<VERSION> py-cython/0.29.24<VERSION>
$ export OMP_NUM_THREADS=4
$ srun -c 4 python script.py
Estimated value of pi = 3.1415926535896825
Error = 1.1057821325266559e-13
Compute time = 0.724 seconds |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 5. Run NumPy code |
|---|
| $ salloc -N 1 -n 1 -p debug -t 0:05:00
$ module load python/3.9.7<VERSION> py-numpy/1.20.3<VERSION>
$ srun python matrix.py
[ 11.32995843 15.18201852 13.1489202 12.46920531]
[[ -5.52631579e-01 1.13157895e+00 1.57894737e-01 -1.84210526e-01]
[ -1.11111111e-01 5.55555556e-01 -6.76965259e-18 -2.22222222e-01]
[ 7.30994152e-02 -1.54970760e-01 -5.26315789e-02 1.72514620e-01]
[ 3.15789474e-01 -7.89473684e-01 5.26315789e-02 1.05263158e-01]]
0.00292397660819
[ 0.63860313 -0.45638416 0.05517347 -0.18183689]
[[ 0.60206445 0.85629788 -0.619295 -0.00817806]
[ 0.57685743 -0.0186308 -0.31291905 -0.21176471]
[-0.19459178 0.05700502 -0.60213224 0.68340563]
[-0.51661198 -0.51298856 -0.39495839 -0.69860259]] |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 6. Run SciPy code |
|---|
| $ salloc -N 1 -n 1 -p debug -t 0:05:00
$ module load python/3.9.7<VERSION> py-numpy/1.20.3<VERSION> py-scipy/1.7.1<VERSION>
$ srun python fft.py
[ 4.50000000+0.j 2.08155948-1.65109876j -1.83155948+1.60822041j
-1.83155948-1.60822041j 2.08155948+1.65109876j] |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 7. Build a Cython object |
|---|
| $ salloc -N 1 -n 1 -p debug -t 0:05:00
$ module load python/3.9.7<VERSION> py-cython/0.29.24<VERSION>
$ srun python setup.py build_ext --inplace |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 8. Run Cython code |
|---|
| $ salloc -N 1 -n 1 -p debug -t 0:01:00
$ module load python/3.9.7<VERSION> py-cython/0.29.24<VERSION>
$ srun python script.py
Estimated value of pi = 3.1415926535904264
Error = 6.332712132461893e-13
Compute time = 0.547 seconds
|
|
...