Listing the contents of your account

Owned by Sean Fleming
Last updated: Oct 04, 2023

Back up important data

Take regular backups of important data. Data in object storage is spread across various servers,
which protects against disk and server failures, but this does not protect the data from accidental deletion. See here for Conditions of Use and Data Storage and Management Policy.

Data organization

Acacia has a flat structure. You create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders.

However, S3 clients present your data as if it was organized in the fashion you are familiar with from file systems. This is achieved through the use of pseudo folders (also known as prefixes).

If an object's name contains a forward slash ( / ), it is interpreted as a folder separator.

For example, you create a bucket called pawsey033-research-data-2019, and then upload four files to this bucket with the following object names:

  • Experiments/Schedule.xls

  • Experiments/Lab_inventory.xls

  • Results/Summary.pdf

  • Code/workflows.tar.zip

  • annual-report.pdf

The prefixes (Experiments, Results and Code) and delimiter (/) will present as 3 folders when you perform a list query, with 2 objects in the Experiments folder and 1 object in each of the Results and Code folders. The annual-report.pdf key does not have a prefix or delimiter so its object appears directly at the root level of the bucket. Despite all this there is only 1 bucket with 5 objects inside it.

Client support

  • Pseudo folders are automatically created if you upload whole folders, since the object names will include the folder name and a /.


Pawsey portal

The Pawsey user portal (https://portal.pawsey.org.au) provides a summary of your accounts on Acacia, including available and total quota, number and name of buckets, number of objects and so on.


Listing with Rclone

Rclone provides a number of different commands to show the data in your accounts, and these are briefly described below, with links to the full documentation given at the bottom of the page.

Tree

Rclone tree lists the contents of your account in a tree like fashion. 

rclone tree [flags] account:path


It has many options for controlling the listing, for example you can print the date of last modification of objects with -D. For a summary of all the options available refer to the Rclone documentation.


ncdu

The rclone ncdu command displays a text based user interface allowing you to navigate an account. It is most useful for answering the question - "What is using all my allocation?".

rclone ncdu account:path


ls*

There are several list commands that begin with ls and control the search and output parameters:

  • ls to list size and path of objects only
  • lsl to list modification time, size and path of objects only
  • lsd to list buckets only
  • lsf to list objects and buckets in easy to parse format
  • lsjson to list objects and buckets in JSON format

ls,lsl,lsd are designed to be human-readable. lsf is designed to be human and machine-readable. lsjson is designed to be machine-readable.

Note that ls and lsl recurse by default - use --max-depth 1 to stop the recursion.

The other list commands lsd,lsf,lsjson do not recurse by default - use -R to make them recurse.

rclone ls* [flags] account:path


Related pages