Pawsey supercomputers run Linux-based operating systems. This page describes the software environment you will use to interact with Pawsey systems.

Consult the User Guide of a supercomputing system to know what operating system it runs.

On this Page:

Command-line interface

Once connected to a login node via SSH, users interact with the supercomputer through a command-line interface provided by a Unix shell. The shell interprets commands that users type and prints the command output, if there is any. For example, the cd command is used to navigate from one directory to another:

$ cd <other_dir>

A shell environment can be used by invoking it on the command line or including it as the interpreter in scripts.

Use bash

Users are strongly recommended to use the Bash shell. Bash is the default shell on all Pawsey supercomputing systems, and it has had sufficient functionality for most use cases. For more details regarding Bash usage, refer to the Bash Reference Manual (external site).

Environment variables

Environment variables are dynamically named string values within the command-line environment. The way in which environment variables are set, displayed and used may vary based on the shell environment. The following assumes the use of the Bash shell.

To display a complete list of current environment variables, use the env command.
To access the value of an environment variable, prepend the $ (dollar) character to its name. For example, running the command echo $USER prints the value of the USER variable.
To set the value of an environment variable, use the following syntax:

VARIABLE_NAME=<value>

Any command executed within the shell can access an environment variable if that variable has been exported. To do so, use the export command:

export VARIABLE_NAME

Use double quotation marks when setting a value that includes whitespace.
Variables that hold multiple directory locations often use the colon symbol ( : ) to separate the directories.

Pawsey environment variables

There are several environment variables set up by Pawsey on its HPC systems.

Table 1. Predefined variables that are available when you log into Pawsey supercomputing systems

Variable name	Purpose	Example values
PAWSEY_CLUSTER	Hostname of the system	setonix
PAWSEY_OS	Current operating system	centos7.6
PAWSEY_PROJECT	Default project for the user	pawsey####, director####
MYSCRATCH	Default `/scratch` directory for the user	/scratch/$PAWSEY_PROJECT/$USER
MYSOFTWARE	Default `/software` directory for the user	`/software/$PAWSEY_PROJECT/$USER`

Compilation and runtime variables

When compiling software, the compilers check directories listed in various environment variables for headers and libraries. You may need to set CPATH to the paths to C/C++ header files that need to be included in a build. LIBRARY_PATH points at paths to static libraries. If these files are part of a system module, loading the module should add the appropriate directories to the relevant variables. When you invoke a program using the command-line interface, the shell environment checks the locations listed in the PATH variable for programs matching the name provided. If the program has been compiled with dynamically linked libraries, the OS checks locations listed in LD_LIBRARY_PATH for these libraries. For software installed as a system-wide module, these variables will be updated with appropriate paths when the module is loaded.

See Modules for more information.

File permissions

File permissions control access to files on the HPC filesystems. Linux file permissions are used at Pawsey. The permissions of files in a directory can be viewed using the ls -l command. For each file, it prints a line like the one shown in terminal 1.

Terminal 1. Listing file permissions

> ls -l
> -rwxr--r--+  1 user group 253 Jul 01 2018 test.sh

In particular, the output comprises

A character indicating the file type: - for a file, d for a directory, and l for a link.
Three characters indicating the owner permissions for the file: r for read, w for write, x for execute, s for the setuid bit, or - otherwise.
Three characters indicating the group permissions for the file: r for read, w for write, x for execute, s for the setgid bit or - otherwise.
Three characters indicating the other permissions for the file: r for read, w for write, x for execute, t for the sticky bit or - otherwise.
A + character if discretionary access control lists provide additional access, where the getfacl command can provide further details.
The number of hard links to the file.
The owner of the file.
The group of the file.
The size of the file in bytes.
The timestamp that the file was last modified.
The name of the file.

For more details regarding permissions, you can refer to the Wikipedia article on filesystem permissions, particularly the section on symbolic notation: File-System Permissions (external site)

Warning

Do not set any of your directories to be world-readable. This has the risk that every user on the supercomputer can read the data, even if it is only for a short time.

Use the group-sticky attribute, which can be set using the chmod command:

$ chmod g+S <directory name>

If a directory has the group-sticky attribute, new files copied to the directory will change to have the same group as the directory. This is useful when installing applications under /software. If a directory does not have the group-sticky attribute, then new files copied to the directory will change to have the group of the user who copied the files. This can be useful in /scratch if you do not want other members of the project group to be able to access, modify or delete files. Some file transfer programs (like WinSCP) by default will ignore a group-sticky bit. You can either edit the program’s configuration or fix the permissions after the upload.

File ownership and permissions for Lustre filesystems

There are quotas set on a per-user and per-project basis for the /software filesystem, and a quota set on a per-user basis on the /scratch filesystem. Files must belong to the appropriate user and group to be correctly counted against the quotas, as well as ensure consistent read/write/execute permissions.

If you encounter a quota error while writing, compiling, or transferring files on /software, it is most likely that this is because the files are counting against your personal quota rather than your project's group quota. Common causes of this issue are:

Moving files from /home instead of copying them
Using the -a or -p flag with cp or rsync
An incorrectly configured scp client

If there is a singular file that needs to be fixed, you can change which group it belongs to by using the chgrp command.

If there are a large number of files that need to be fixed, then you can use the find command instead.

Below are examples, line 1 shows chgrp to fix a single file and line 2 shows the use of the find command to fix the files in your /software directory. The <myproject> part must be replaced with your project code (e.g. pawsey0123) and <myfile> is the name of the file (if you're changing just a single file).

Terminal 2. Fixing file group

> chgrp <myproject> <myfile>
> find /software/projects/<myproject>/$USER/ -group $USER -exec chgrp <myproject> {} \;

Text editors

The following command-line and graphical text editors are available on the login nodes. These are typically used for making minor changes to job scripts, but can also be used for code development.

Emacs

The Emacs editor is available as a system program. To launch the graphical version of Emacs, ensure you are connected using X11 forwarding and type emacs.

To launch the command-line version of Emacs, use the -nw flag:

$ emacs -nw

To exit Emacs, press Ctrl+x then Ctrl+c. More information can be found on the Emacs webpage (external site).

Nano

The Nano editor is installed as a module. As such, to use the editor first load the nano module, then run the nano command. To exit Nano, press Ctrl+x. More information can be found on the Nano webpage (external site).

Vim

The Vim editor is installed as an operating system package. To launch it, use the vim command. To exit Vim, press the Esc key then type :wq to save and exit, or :q! to just exit the program. More information can be found on the Vim webpage (external site) .

Related pages

Connecting to a Supercomputer

External links

Browser not supported