Using Python
- Marco De La Pierre
- Alexis Espinosa
Python is a very popular programming language and has successfully found its way into many application areas including data science, machine learning, and scientific computing.
Python can be used in many different ways in scientific computing, depending on the features of the researcher workflows. As a result, Pawsey provides different options to use and expand Python capabilities on the Setonix supercomputer and the other systems.
Versions installed in Pawsey systems
To check the current installed versions, use the module avail command (current versions may be different from content shown here):
$ module avail python --------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages --------------------- python/3.9.15 python/3.11.6 (D) --------------------------------------- /opt/cray/pe/lmod/modulefiles/core ---------------------------------------- cray-python/3.9.13.1 cray-python/3.10.10 (D)
We recommend the use of python installed by Pawsey instead of cray-python.
Many scientific and managing tools have been also installed in our software stack. These are named with the prefix py-, and sometimes their flavour is also identified by a suffix -py<VERSION> after the version corresponding to the tool itself (current versions may be different from content shown here):
$ module avail py- ---------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications ----------------------- py-astropy/4.2.1 py-astropy/5.1 (D) py-emcee/3.1.1 py-funcsigs/1.0.2 --------------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities --------------------------- py-boto3/1.26.26 py-setuptools/59.4.0-py3.9.15 py-setuptools/68.0.0-py3.11.6 (D) py-pip/23.1.2-py3.9.15 py-setuptools/59.4.0-py3.11.6 py-pip/23.1.2-py3.11.6 (D) py-setuptools/68.0.0-py3.9.15 ------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages ------------------------ py-cython/0.29.36 py-ipython/8.14.0 py-numpy/1.24.4 py-scikit-learn/1.3.2 py-cython/3.0.4 (D) py-matplotlib/3.8.1 py-numpy/1.26.1 (D) py-scipy/1.11.3 py-dask/2023.4.1 py-mpi4py/3.1.5-py3.11.6 py-pandas/1.5.3 py-h5netcdf/0.10.0 py-netcdf4/1.6.2 py-pandas/2.1.2 (D) py-h5py/3.8.0 py-numba/0.57.0 py-plotly/5.14.1 ------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools ------------------------ py-hatchet/1.3.1
How to use Python at Pawsey
Python is currently installed on Pawsey systems, in the form of software modules.
Do NOT use system Python
Upon login, you may notice that the following commands are already available: python , python2 , python3. These are Python interpreters installed by the system Operating System for its own purposes. They are old versions with limited functionalities.
Do NOT use them for your scientific workflows.
Cray Python is not recommended
The Setonix supercomputer also comes with a vendor-provided cray-python module. Some of the packages included in this module are quite old, therefore at this stage we recommend not to use it, and use instead the Pawsey-provided modules (see below).
Pawsey Software Stack Python is the recommended option
Pawsey provides Python as part of the Pawsey Software Stack. This software stack contains a couple of Python installations that can be used by loading the specific module, as described in the following sections.
Bare Python module
A module with the basic Python interpreter is available on all Pawsey systems. Available versions can be checked via the module avail python command. And they can be used by loading the desired version. For example:
$ module load python/3.11.6
Once loaded, the executables python and python3 are available for use, for instance:
>>> python --version
Python 3.11.6
Modules for scientific Python packages
On the Setonix supercomputer, Pawsey curates a set of popular Python packages for scientific computing; where relevant, these packages are configured to leverage Cray MPI for distributed computing. The set can be inspected with module avail :
$ module avail py- .. ------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages ------------------------ py-cython/0.29.36 py-ipython/8.14.0 py-numpy/1.24.4 py-scikit-learn/1.3.2 py-cython/3.0.4 (D) py-matplotlib/3.8.1 py-numpy/1.26.1 (D) py-scipy/1.11.3 py-dask/2023.4.1 py-mpi4py/3.1.5-py3.11.6 py-pandas/1.5.3 py-h5netcdf/0.10.0 py-netcdf4/1.6.2 py-pandas/2.1.2 (D) py-h5py/3.8.0 py-numba/0.57.0 py-plotly/5.14.1 ..
When any of these modules are loaded, its dependencies, including the Python interpreter, will also be loaded. When launched, the interpreter will be able to import the loaded scientific packages.