Module 1 : Getting Started with Setonix| Expand |
|---|
| title | Here are questions from Module 1 training sessions...... |
|---|
|
| Question | Answer / Solution |
|---|
| Would sbatch -M on Setonix work? | Currently not supported. As Setonix is a single system replacing multiple systems, you may not need to use the -M flag once the second phase of Setonix arrives combining the functionality of Magnus, Zeus and Topaz into one system. | | Do we need to create a new id_rsa key? | You can either copy over existing public keys between the old and new /home filesystems, or create a new key, whichever suits your work better. | | Will Zeus be decommissioned at the same time as Magnus? | Yes. |
|
Module 2: Supercomputing Filesystems| Expand |
|---|
| title | Here are questions from Module 2 training sessions... |
|---|
|
| Question | Answer / Solution |
|---|
| Does our existing allocation on HSM get transferred to Acacia? Do we need to reapply for the larger allocation? | HSM allocations will not be transferred automatically to Acacia; users have to apply. | | Where are the old filesystems mounted? I can’t see it at the moment on Setonix. | /oldgroup and /oldhome are available only on copy partition of Setonix, and only for the duration of the migration. They are not available on the login or compute nodes. The /newscratch filesystem is available on the copyq partition of Zeus for the duration of the migration. | | Will the /oldhome and /oldgroup cease when Magnus is decommissioned? Because Topaz still uses software on /group. | The /oldhome and /oldgroup filesystems will be decommissioned with or shortly following Magnus and Zeus. Other supercomputing systems using these filesystems will transition to using the new filesystems before this occurs. | | Will the old group filesystem be accessible from Setonix login node? | No, it will be available on the copy partition only. | | What is the access path to Acacia? Will data transfer be more complicated? | Access is through an S3 client. There may be some initial learning involved at first. However, once template scripts are set up for data transfer workflows it should then be a smooth process. | Reference datasets - should datasets that are used frequently by multiple users but from the same project live in reference or Acacia?
| If the users are all from the same project, Acacia is the appropriate choice. The reference data sets on /scratch are intended for large datasets used by a number of project groups. | | Can link to the documentation be provided? | The following links are provided: | | Would it be possible to increase the file limit on /software? | The limit set for /software should be enough in general. If needed, a ticket could be opened on increasing quota on /software | | Is it planned to have OpenFOAM 2.4 or could it be user installed with singularity? | Old versions of OpenFOAM are difficult to install. We encourage researchers to upgrade their workflows to more recent versions. We are trying to make 2.4 work in a container. However, it may not be possible to provide this version. | | Does pawseyAccountBalance also let you know about the number of files in /scratch and /software (to allow users to monitor that)? | You can use the command: | Code Block |
|---|
lfs quota -g $PAWSEY_PROJECT /software |
It provides you the quota (both for size and file limit) for your group in a given mount point. |
|
Module 3: Using Modules & Containers| Expand |
|---|
| title | Here are questions from Module 3 training sessions... |
|---|
|
| Question | Answer / Solution |
|---|
| Just wanted to confirm that pawsey_prgenv is loaded by default which is required by singularity/3.8.6 | Yes, the pawsey_prgenv module is loaded by default; it is actually needed not only by Singularity, but to make visible all of the modules built by Pawsey staff. | | Do we have to recompile custom software that is compiled in the singularity image with AOCC? | You will only need to recompile if the software in the image has used hardware architecture flags during compilation that are different to the Setonix hardware architecture; this is uncommon in containers, but possible in principle. Note that there are three compilation environments available on Setonix (PrgEnv-gnu and PrgEnv-cray in addition to PrgEnv-aocc) with PrgEnv-gnu available by default. | | Would Pawsey's singularity 3.8.6 module be able to see 3.8.9 image? | There shouldn't be any problem. | | Is there an ordering that you can use to load all the modules in a single command line or do you have to sequentially enter in multiple lines? | The Pawsey software stack only has one tier of module hierarchy based on the choice of programming environment. The HPE Cray software stack has a deeper hierarchy. The modules automatically loaded include default choices for these. For the compiler hierarchy, we recommend switching branches of the hierarchy first in a separate module command, for example switching from PrgEnv-gnu to PrgEnv-cray, and then load dependant modules in a separate module command. For other Cray hierarchies, e.g. I/O libraries, loading multiple modules in the same line will work, as suggested by the output of module spider. For instance, cray-hdf5 enables a hierarchy containing, among others, cray-netcdf, and the following one-line command will work as expected: module load cray-hdf5 cray-netcdf. | | Does module swap also need module version? The examples on slide 16 don't seem to include that. | You will need to provide a version for any type of loading for modules provided by Pawsey. So for example, for python provided by Pawsey, you will need to write module load python/3.9.7` but cray modules have defaults and do not need explicit module versions. | | How will access to domain-level software which required a licence on Magnus/Zeus etc be handled on Setonix? | Same policy as Magnus. We do provide installations of popular software, but we don’t provide licenses. The process for gaining access to Pawsey installations of licensed software varies depending on the license terms, we recommend submitting a help desk request for a particular software. |
|
Module 4: Installing & Maintaining Your Software (Developers)| Expand |
|---|
| title | Here are questions from Module 4 training sessions... |
|---|
|
| Question | Answer / Solution |
|---|
| For OpenMP, what is the status of hyperthreading? | To be clear, OpenMP does not requiere hyperthreading to work. The number of OpenMP threads that will be used per basic task is set by the Slurm sbatch option, --cpus-per-task=<c> (or -c <c>). Together with the OpenMP setting OMP_NUM_THREADS-<c>.
All this work fine with the default behaviour of Hyperthreading, which is: disabled.
If a user knows that an application can benefit from hyperthreading, then it can be enabled using the appropriate Slurm sbatch option, --threads-per-core=2. |
|
|
|
Module 5: Submitting & Monitoring your Job| Expand |
|---|
| title | Here are questions from Module 5 training sessions... |
|---|
|
| Question | Answer / Solution |
|---|
| Do you prefer for us to transfer to new scratch using the copyq partition on Zeus or the copy partition on Setonix? | Both options can be used. /newscratch is available on Zeus and /oldgroup is available on Setonix | | Hyperthreading is switched off by default, do we use --cpus-per-task to switch it on or is there something else we need to do? | Hyperthreading is disabled by default. If an application can benefit from it, hyperthreading can be enabled using the appropriate Slurm sbatch option, --threads-per-core=2. | | Are there plans to extend the number of nodes? | Yes, Phase 2 will add CPU nodes as well as GPU nodes. Note that the number of cores per node has increased from 24 cores on Magnus to 128 cores on Setonix. | | Is there any memory affinity within the 2 sockets in a node? | Milan cpus and memory do have affinity but the current version of slurm and hwloc on the system can result in poor placement of execution threads relative to memory. We are trying to address this shortcoming and future versions of slurm should resolve this. |
|
Module 6: Using Data Throughout the Project Lifecycle| Expand |
|---|
| title | Here are questions from Module 6 training sessions... |
|---|
|
| Question | Answer / Solution |
|---|
| Should 7TB of images (used in the same project) that used to be on HSM be transferred to Acacia or copied to reference datasets on Setonix | Datasets that are generally used by a single project should reside in Acacia and not as reference datasets on Setonix. | | Has data from HSM been transferred to Banksia | Please submit a help desk ticket so we can follow up. | | Do we still have access to scron jobs to automate the syncing between Acacia and files on /scratch? | The scrontab is installed but not currently enabled. Once it is enabled, porting to Setonix should be seamless. We expect it to be enabled soon. Since the functionality of Magnus and Zeus is combined on Setonix, job dependencies may instead be used for some scrontab use cases. | | Is there a limit on number of characters for bucket name? | Bucket name has to be between 3 and 63 characters long | | If your program can read directly from Acacia via the S3 interface, why would you copy to /scratch first? For large files that can take some time. | Using /scratch to stage data has the advantage of when a job fails the data doesn't need to be pulled back from Acacia again but instead read from /scratch which is much faster. The compute node bandwidth to /scratch is much higher than to Acacia. Higher bandwidth ensures that its easier for compute nodes to access data from within Setonix. Essentially Acacia is not a filesystem but an object storage system for storing much larger capacities of data as compared to what was stored on /group. By having a separate staging job to pull in the data from Acacia, the larger simulation job is not wasting allocation to retrieve this data. | | How do we request larger project allocations on Acacia? | It depends on how much data you are requesting: up to 10 TB you can raise a help desk ticket, for larger then 10TB, it has to be done through data application portal. | | Are project and access keys needed for each user? | Yes each user in a project will get their own keys. And a separate pair of keys for their own storage.
Users should not be sharing their access keys, not even with colleagues on the same project. | | As /group will be retired in 3 months, should data on Topaz be moved now? | /group will be decommissioned following the migration, along with the old /scratch and old /home filesystems.
New file systems are expected to be mounted on Topaz at some point in the following months.
If you need to work on Topaz now, then keep the data there. But if you plan to work on Topaz only in the later part of the year then you can transfer the data now. | | Can we create subdirectories in a bucket? | To organise your data within a bucket you can use a prefix path, but this does not create actual subdirectories or "sub-buckets". No subdirectories or "sub-buckets" are possible inside an S3 bucket in Acacia.
The prefix path (a series of words separated with slashes "/" ) is part of the object name. But S3 clients recognise the prefix as an organisational method. And then, in practise, list and operate each word in the prefix as a pseudo-folder used for organising your data.
But it's important to remember that the prefix is indeed part of the object name, because some commands operate with it as that and not as pseudo-folders. | | Is is there a limit to the number of buckets? | There isn't a limit as such. But it is a good practice to have fewer buckets with more metadata per object. | What's the motivation behind one level of buckets (rather than using directory trees)? | Object storage does not support directory trees. It is easier to pull objects with metadata in the object itself than in the path or directory. You can also use prefixes to emulate directories. | | If the metadata is basically replacing the need for subdirectories, how is information put into the metadata? Can a user specify key words that can be later searched? | Metadata in the form of "key-value" pair is assigned to an object while uploading it. Syntax for different S3 clients vary slightly. rclone copy uses --header option. It is important to assign meaningful metadata when you create an object. Documentation can be found at User metadata - User Support Documentation - Pawsey Documentation |
|
Ask Me Anything Sessions| Expand |
|---|
| title | Here are questions from AMA sessions... |
|---|
|
| Question | Answer / Solution |
|---|
| My login was rejected a few times and my account was disconnected. | Setonix has a number of login nodes are available via the setonix.pawsey.org.au hostname using a round-robin system. When there is an issue with a login node it is removed from the round robin, but it may take time for this change to propagate across the domain name servers on the internet. In this case, for a short period of time it may be possible to inadvertently connect to a login node in maintenance mode where you will be disconnected. In this instance it is recommended to try again, or temporarily set a specific login node hostname (such as setonix-01.pawsey.org.au) as a workaround. We do not recommend setting specific hostnames in the longer term. | | What is the status of scrontab? | It is inactive still, but will be turned on during subsequent updates. | | Does the increase from 1.1 Petaflops (Magnus) to 50 Petaflops mean that there will be close to 10000 CPUs on Setonix | The jump to 50 Petaflops is mostly attributed to the GPUs that Setonix phase 2 will have. Each node on Setonix will have 128 cores as compared to the 24 on Magnus | | Will mpifileutils be available on Setonix? The user uses this utility extensively for copying / removing files. | 'mpifileutils ' utility is not available currently on Setonix. However, there is a Spack recipe for it. If you have Spack in your shell you can then load mpifileutils using module load spack/0.17.0; spack spec -I mpifileutils@0.11 +lustre +xattr spack install mpifileutils@0.11 +lustre +xattr Spack usage is covered in our documentation. | | Will a script for fixing file permissionsbe available on Setonix? It is currently available on Zeus | Currently it is not, but we will look to add a similar utility script onto Setonix. | How can we preserve file permissions without using the file permission script above? - chmod access does not preserve source permissions. (provides read access but not write access in the destination)
- When a job is run by a different user after the walltime elapse, chmod doesn't give write access to the temp files produced in the previous run by another user and the job fails.
| - For transferring files,
- file permissions can be preserved with rsync -p ;
- group ownerships can be preserved with rsync -g
- To fix file permissions after they've been copied generated, these two commands can be handy:
- restore file permissions: chmod -R g+rwX <directories/files> (remove "w" if only read access is needed);
- restore group ownerships: chgrp -R $PAWSEY_PROJECT <directories/files>
- Before running a process that generates new files/directories, executing the following allows to generate them with the right file permissions: umask 007
|
|
|