...

Versions installed in Pawsey systems

To check the current installed versions, use the module avail command (current versions may be different from content shown here):

$ module avail python
--------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages ---------------------
   python/3.9.15    python/3.11.6 (D)

--------------------------------------- /opt/cray/pe/lmod/modulefiles/core ----------------------------------------
   cray-python/3.9.13.1    cray-python/3.10.10 (D)

We recommend the use of python installed by Pawsey instead of cray-python.

Many scientific and managing tools have been also installed in our software stack. These are named with the prefix py-, and sometimes their flavour is also identified by a suffix -py<VERSION> after the version corresponding to the tool itself (current versions may be different from content shown here):

$ module avail py-
---------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications -----------------------
   py-astropy/4.2.1    py-astropy/5.1 (D)    py-emcee/3.1.1    py-funcsigs/1.0.2

--------------------------- /software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities ---------------------------
   py-boto3/1.26.26              py-setuptools/59.4.0-py3.9.15    py-setuptools/68.0.0-py3.11.6 (D)
   py-pip/23.1.2-py3.9.15        py-setuptools/59.4.0-py3.11.6
   py-pip/23.1.2-py3.11.6 (D)    py-setuptools/68.0.0-py3.9.15

------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages ------------------------
   py-cython/0.29.36         py-ipython/8.14.0           py-numpy/1.24.4         py-scikit-learn/1.3.2
   py-cython/3.0.4    (D)    py-matplotlib/3.8.1         py-numpy/1.26.1  (D)    py-scipy/1.11.3
   py-dask/2023.4.1          py-mpi4py/3.1.5-py3.11.6    py-pandas/1.5.3
   py-h5netcdf/0.10.0        py-netcdf4/1.6.2            py-pandas/2.1.2  (D)
   py-h5py/3.8.0             py-numba/0.57.0             py-plotly/5.14.1

------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools ------------------------
   py-hatchet/1.3.1

Before you begin

Basic information on these installation tools can be found in their websites (external links):

...

To install Python packages on top of the bare Python interpreter, pip and setuptools are available on Pawsey systems as modules (note the py- prefix for Python packages):

$ module load python/3.911.7 6 py-pip/2123.1.2-py3.911.76

$ module load python/3.911.7 6 py-setuptools/5768.40.0-py3.911.7

To install Python packages on top of Pawsey-provided scientific Python packages, use the Python collection module instead (pip and setuptools included):

$ module load hpc-python-collection/2022.05-py3.9.7

6

Once you choose one of these two options to install Python packages, you should keep using it, to avoid conflicts and errors arising from packages installed in different locations.

Pawsey Python modules also preconfigure the shell environment by default to provide a meaningful location for user-specific package installations:

export PYTHONUSERBASE=/software/projects/<project-id>/<user-name>/setonix/python

export PATH=$PATH:$PYTHONUSERBASE/bin

...

$ module load python/3.9.7<VERSION_OF_PYTHON>
$ module load py-pip/21.1.2-py3.9.7<VERSION>-py<VERSION_OF_PYTHON>

$ module load py-numpy/1.20.3<VERSION>

Now let's proceed with the installation. We're going to use pip with the --user  flag to install in the Python user directory. (Users cannot install software in system directories on shared Pawsey supercomputer systems.)

...

$ module load python/3.9.7<VERSION_OF_PYTHON>
$ module load py-pip/21.1.2-py3.9.7<VERSION>-py<VERSION_OF_PYTHON>

$ module load py-numpy/1.20.3<VERSION>

$ sg <projectcode> -c 'pip install --user --no-deps -r requirements.txt'
Collecting astropy==4.1 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/74/9c/a1e51955d4a2af497a507c323409ebe55c122a91c438d2884d918360efc1/astropy-4.1-cp36-cp36m-manylinux1_x86_64.whl (10.3MB)
     |████████████████████████████████| 10.3MB 16.0MB/s 
Requirement already satisfied: numpy==1.19.0 in /pawsey/sles12sp3/python/3.6.3/numpy/1.19.0/lib/python3.6/site-packages/numpy-1.19.0-py3.6-linux-x86_64.egg (from -r requirements.txt (line 2)) (1.19.0)
Installing collected packages: astropy
Successfully installed astropy-4.1

...

Change to the tqdm source code directory and then load the required modules, namely python and py-setuptools:

$ module load python/3.9.7 <VERSION_OF_PYTHON> py-setuptools/57.4.0-py3.9.7<VERSION>-py<VERSION_OF_PYTHON>

Finally, execute the build and installation process:

$ sg <projectcode> -c 'python setup.py build'
running build
running build_py
creating build
[...]
$ sg <projectcode> -c 'python setup.py install --user'
running install
running bdist_egg
running egg_info
[...]
Finished processing dependencies for tqdm==4.61.1

...

$ python -m venv $MYGROUP$MYSOFTWARE/manual/pythonEnvironments/python-venv # create a virtual environment 
$ source $MYGROUP$MYSOFTWARE/manual/pythonEnvironments/python-venv/bin/activate # activate the environment, updating python paths 
$ sg <projectcode> -c 'pip install astropy==4.1' # install astropy in the environment 
$ deactivate # deactivate the environment, now cannot load packages installed in venv. 

...