Versions Compared

Version Old Version 2 New Version 3
Changes made by

Cristian Di Pietrantonio

Chris Harris

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Column

These new filesystems will have the following limits:

FilesystemTimeCapacityFile Count
/homeDuration of project(s)1 GB per person10,000 files per person
/softwareDuration of project256 GB per project100,000 files per project
/scratch30 days per file1 PB per project1,000,000 files per project

The legacy old home and group filesystems will be mounted temporarily on the data mover nodes of Setonix copy partition of Setonix as /oldhome and /oldgroup to allow for data transfers.

The new scratch filesystem will be mounted temporarily on the copyq partition of Zeus as /newscratch to allow for data transfertransfers.

Warning

Do not use legacy filesystems as part of active compute job workflows on Setonix, as they are not mounted on compute nodes and will be decommissioned following the migration.

The Migrating Data to Setonix page provided more information about transferring data from the old filesystems to Setonix.

...

  1. Log in to Setonix for the first time.
  2. Migration of Migrate data from existing to new filesystems:
    • Migrate and update configuration files from the legacy /home filesystem using the data mover nodes /oldhome on the Setonix copy partition.
    • Migrate working data from the legacy /scratch (or /group) filesystem to the new /scratch filesystem using the data mover nodes /newscratch on the Zeus copyq partition.
    • Migrate any project data from /group to the Acacia object store.
    • Migrate historical job scripts for reference from the legacy /home, /group or /scratch to the /software filesystem using the data mover nodes.
  3. Familiarisation Familiarise yourself with the available modules and versions available on Setonix.
  4. Installation by the project group of any Install any additional required domain-specific software for yourself or your project group using the /software filesystem.
  5. Preparation of Prepare job scripts for each step of computational workflows by keeping templates in the /software filesystem, including:
    1. Staging of data from the Acacia object storage or external repositories using the data mover nodes
    2. Pre-processing or initial computational jobs using appropriate partitions
    3. Computationally significant jobs using appropriate partitions
    4. Post-processing or visualisation jobs using appropriate partitions
    5. Transfer of data products from /scratch to the Acacia object store or other external data repositories
  6. Submission of Submit workflows to the scheduler, either manually using scripts or through workflow managers.

...