How to move data from Puhti to Allas and vice versa?
There are several options on how to move your data between Puhti and Allas. This page gives you guidance for the data movement with clients a-commands, swift, rclone and s3cmd.
All the required packages and software needed for the clients are already installed on Puhti. To authenticate to a project you need to run command
Move data with a-commands
a-commands provides easy-to-use tools for basic usage of Allas. The functions to use for data movement between Puhti and Allas are:
Move data with Swift
Swift client provides functions upload and download for data movement:
swift upload <bucket name> <file name>
swift download <bucket name> <file name>
Move data with rclone
Also, with client rclone you can move data between Puhti and Allas. Creating a bucket called 2000620-raw-data to Allas can be done with command:
rclone mkdir allas:2000620-raw-data
Uploading a file called file.dat to that bucket can be done with command
rclone copy file.dat allas:2000620-raw-data/
Downloading the file back to Puhti is done with the same
rclone copy command:
rclone copy allas:2000620-raw-data/file.dat
Note: If you give a destination parameter name in the download command, rclone creates a directory where the download goes:
rclone copy allas:2000620-raw-data/file.dat doh
For further guidance for using rclone with Allas from Supercomputers, see Using Allas with rclone from Puhti.
Move data with s3cmd
For data movement between Puhti and Allas s3cmd provides functions:
More examples of using s3cmd from Supercomputers can be found here.
Last edited Fri Sep 4 2020