How to move data from Puhti to Allas and vice versa?
There are several options for moving data between Puhti and Allas. This page summarizes how to move data using a-commands, Swift, Rclone and S3cmd clients.
All the required packages and software needed for the clients are already installed on Puhti. To activate Allas and authenticate to a project you need to run the commands:
module load allas
allas-conf
Move data with a-commands
a-commands are easy-to-use tools for basic usage of Allas. The main commands to use for moving data between Puhti and Allas are:
Move data with Swift
The Swift client provides the options upload and download for moving data:
swift upload <bucket name> <file name>
swift download <bucket name> <file name>
For more information, see Swift client.
Move data with Rclone
Rclone is another client using which you can move data between Puhti and
Allas. For example, you could create a bucket called 2000620-raw-data in
Allas using the command:
rclone mkdir allas:2000620-raw-data
Uploading a file called file.dat to that bucket can be done using the
rclone copy command:
rclone copy file.dat allas:2000620-raw-data/
Downloading the file back to Puhti is done with the same rclone copy command:
rclone copy allas:2000620-raw-data/file.dat .
Note
Another destination directory can be specified in the rclone copy command
as well. If this directory does not exist, Rclone will create it for you.
rclone copy allas:2000620-raw-data/file.dat my-new-folder
For more information, see Using Allas with Rclone from Puhti.
Move data with S3cmd
For moving data between Puhti and Allas, S3cmd provides the functions:
For more information, see S3 client.