Copying and sharing objects
- Sean Fleming
- Lachlan Campbell
Copying an object
To copy a single object from Acacia:
> rclone copy <SOURCE> <TARGET>
Where:
<SOURCE>is the key of the object on Acacia you want to download, so profile name (with its colon, :), bucket name, pseudo folder (optional), and object name.<TARGET>is the filesystem path where you want the object to be downloaded to.
Example:
|
Setonix /scratch
If copying to Setonix /scratch file system please be aware that rclone sets the modtime to the value it gets from the S3 storage. This could result in data being purged from /scratch even though it has not been on the file system for 21 days. To prevent this you can use the --local-no-set-modtime option to rclone:
$rclone copy --local-no-set-modtime courses01:cou01/data-shell.zip /scratch/pawsey0***
Multiple sources cannot be indicated in the basic syntax so, in case we need to indicate the copy of multiple specific files, we should use the --include option of the command (in agreement with rsync syntax). Therefore, the <SOURCE> will indicate the path containing those files and the --include option indicate the specific files to copy:
|
Sharing an object
By default buckets and the objects in them are private, meaning that for user accounts only the user has administrative rights and access, while for projects all members have full rights and access but no-one outside the project.
You can however allow anyone access to buckets on Acacia using a time-limited presigned URL (Uniform Resource Locator (external link)). You can generate a URL that can be used to access your buckets and associate it with a specific action. You can share the URL, and anyone with it and internet access can perform the action as if they were you. The URL will expire and no longer work when it reaches its expiration time. The maximum expiration time for a presigned URL is 7 days from the time of creation.
A more complete and continuing bucket access control can be granted to other Acacia users and projects using policies. These consist of JSON formatted text files which contain the specific configuration desired. Using policies you can give list, upload, download, and delete permissions. The guide to using policies is available in the advanced usage section here.
Access control should be guarded carefully
- Because presigned URLs grant access to your buckets to whoever has the URL, we recommend that you protect them appropriately. For more details about protecting presigned URLs, see Limiting presigned URL capabilities (external link).
- Access control lists (ACLs) can also be used to control access on Acacia, but are not recommended by Pawsey (or Amazon - Access control list overview - (external link)). They are easy to misuse if you are not familiar with them, for example you may grant another user access to your bucket while unintentionally removing all your rights at the same time.
To share an object using rclone, use the link command:
> rclone link <PROFILE_NAME>:<BUCKET_NAME>/<OBJECT_NAME>
Where:
<PROFILE_NAME>is the name you gave the storage space (endpoint + account credentials to specific storage) when configuring rclone.- Note that colon (
:) is always needed after the profile name when using rclone
- Note that colon (
<BUCKET_NAME>is the name of the bucket containing the object.<OBJECT_NAME>is the name of the object to share.
If you supply the --expire flag, it will set the expiration time otherwise it will use the max expiry (7 days).
Use the --unlink flag to remove existing public links to the file or folder. Note not all backends support "--unlink" flag - those that don't will just ignore it.
If successful, the last line of the output will contain the link.