Using Allas with Rclone
This chapter contains instructions for using Allas with Rclone in the Roihu computing environment. Rclone provides a very powerful and versatile way to use Allas and other object storage services. It is able to use both the S3 and Swift protocols (and many others). At the moment, the S3 protocol is the default option on the CSC Roihu server.
WARNING: Rclone should not be used to copy, move or rename objects inside Allas. Rclone provides commands for these operations but they don't work correctly for files larger than 5 GB.
WARNING: If a rclone data upload process for a over 5 GB file gets interrupted, please remove the partially uploaded object before restarting the upload process. Otherwise rclone sometimes reports a successful data upload even though not all data has been copied to Allas.
The basic syntax of Rclone:
rclone subcommand options source:path dest:path
The most frequently used Rclone commands:
- rclone copy – Copy files from the source to the destination, skipping what has already been copied.
- rclone sync – Make the source and destination identical, modifying only the destination.
- rclone move – Move files from the source to the destination.
- rclone delete – Remove the contents of a path.
- rclone mkdir – Create the path if it does not already exist.
- rclone rmdir – Remove the path.
- rclone check – Check if the files in the source and destination match.
- rclone ls – List all objects in the path, including size and path.
- rclone lsd – List all directories/containers/buckets in the path.
- rclone lsl – List all objects in the path, including size, modification time and path.
- rclone lsf – List the objects using the virtual directory structure based on the object names.
- rclone cat – Concatenate files and send them to stdout.
- rclone copyto – Copy files from the source to the destination, skipping what has already been copied.
- rclone moveto – Move the file or directory from the source to the destination.
- rclone copyurl – Copy the URL's content to the destination without saving it in the tmp storage.
A more extensive list can be found on the Rclone manual pages or by typing the command rclone in Puhti.
Authentication on CSC supercomputers
Below, we describe how Rclone is used in Roihu supercomputer. You can use Rclone also in your local computer.
The first step is to configure connection to a project in Allas. Rclone can use both S3 and Swift protocols but these connections will have different names in rclone commands.
Check allas-conf for more info and additional options.
S3 connection to Allas
You can define a new S3 connection to Allas with command:
orFirst allas-conf asks you to give your CSC password (Haka-password can't be used here). After that, if target project is not given as an argument, it lists all available Allas projects and asks user to pick one. ( Note that allas-conf has often problems with passwords that have characters that have special meaning in bash shell. For example space, *, ; and different quotation marks can cause allas-conf to fail).
The project specific access key pair is stored to the configuration file of rclone in your home directory ($HOME/config/rclone/rclone.conf). Due to this the configuration is not session specific, but applies to all sessions that utilize rclone. S3 keys are permanent so you need to run allas-conf command again only when you wish to set a new default S3 connection in use. Thus, in case of S3 based Allas usage, you normally need just to load the Allas module and then start using Allas.
Allas conf defines two rclone remotes s3allas: and s3allas-project-proj-number:. Both remotes refer to the same Allas project. When a new project is defined with allas-conf, the s3allas: endpoint is changed to refer to the new project, but the older project specific endpoint is preserved in addition to the new project specific endpoint that gets generated.
For example after commands:
Following remotes are available:
| Rclone remote | Target project |
|---|---|
| rclone s3allas: | project_200222 |
| rclone s3allas-project_200111: | project_200111 |
| rclone s3allas-project_200222: | project_200222 |
Rclone with swift
If need to use rclone with Swift based Allas connection, for example for accessing data that your stored to Allas from Puhti and Mahti, the connection is activated with commands:
In Rclone commands, this Swift based connection is referred with remote nameallas:.
In the examples below the S3 based s3allas: remote definition is used, but if you have Swift connection defined, you could replace it
with allas:.
Note, that you can have both allas: and s3allas: functional in the same time and that they can still use different Allas projects.
However, you should avoid mixing protocols. If an object is loaded using allas: do also all operations with allas:.
Create buckets and upload objects
The data in Allas is arranged into containers called buckets. You can consider them as root-level directories. All buckets in Allas must have unique names – you cannot create a bucket if some other project has already used that bucket name. It is a good rule of thumb to have something project- or user-specific in the bucket name, e.g. 2000620-raw-data. See the checklist for how to name a bucket. Further avoid upper case letters and spcial characters, including uderscore (_) in the bucket names. In the object names inside the bucket these scahracters are allowed.
In the case of Rclone, create a bucket:
Upload a file using the commandrclone copy:
The command above creates an object file.dat in the bucket 2000620-raw-data.
If you use rclone move instead of rclone copy, the local version of the uploaded file (file.dat)
is deleted after copying.
The copy and move subcommands only work with files. If you would like to copy all files in a directory, use the copyto or moveto subcommands.
During upload, files that are larger than 5 GB will be split and stored as several objects. The objects are stored automatically in a distinct bucket called <bucket-name>_segments. For example, if you would upload a large file to 2000620-raw-data, the actual data would be stored in several pieces in the bucket 2000620-raw-data_segments. The target bucket (2000620-raw-data) would contain just a manifest object stating which segments comprise the stored file. Operations performed on the manifest object are automatically reflected in the segments. Normally users don't need to operate with the segments buckets at all, and objects inside these buckets should not be deleted or modified.
List buckets and objects
List all the buckets belonging to a project:
rclone lsd s3allas: 0 2019-06-06 14:43:40 0 2000620-raw-data
List the content of a bucket:
rclone ls allas:2000620-raw-data 677972 file.dat
Download objects
Use the same rclone copy and rclone copyto commands to download a file:
If you include a destination parameter in the download command, Rclone creates a directory for the download:
ls doh file.dat
ls -ld doh drwxr-xr-x 3 user staff 96 Jun 6 14:58 doh
Synchronizing a directory
One way of moving data between Allas and the computing environment is synchronization. The difference between copying and synchronizing is that while copying only adds new objects or files from the source to the destination, synchronization can also remove data from the destination, in order to make the destination match the source. This feature makes synchronization very effective but also potentially very dangerous.
For example, a folder named mydata has the following structure:
ls -R mydata mydata/: file1.txt setA setB mydata/setA: file2.txt mydata/setB: file3.txt file4.txt
An example of using sync (note that the destination parameter requires the folder name (mydata)):
rclone ls s3allas:2000620-raw-data
677972 mydata/file1.txt
10927 mydata/setA/file2.txt
1116 mydata/setB/file3.txt
5075 mydata/setB/file4.txt
Let us assume that we are storing new data (file5.txt and file6.txt) in the subdirectory mydata/setC and simultaneously removing the file mydata/setB/file3.txt. When the rclone sync command is executed again, the new data is added to Allas and the object mydata/setB/file3.txt is removed.
rclone sync mydata allas:2000620-raw-data/mydata
rclone ls s3allas:2000620-raw-data
677972 mydata/file1.txt
10927 mydata/setA/file2.txt
5075 mydata/setB/file4.txt
1265 mydata/setC/file5.txt
4327 mydata/setC/file6.txt
In the examples above, Allas has been used as the destination that is changed. However, the command can be used in the reverse direction as well:
This command returns the uploaded data from Allas to the mydata directory. Note however that if you have added new data to mydata after synchronizing the directory with Allas, this data will be erased.
Copying files directly between object storages
Rclone can also be used to directly copy files from another object storage (e.g. Lumi-O, Amazon S3, Google cloud, CREODIAS,...) to Allas. For this both credentials need to be stored in a Rclone configuration file in the users home directory (.config/rclone/rclone.conf). An example is shown below:
[s3allas]
type = s3
provider = Other
env_auth = false
access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
endpoint = a3s.fi
acl = private
[otherobjectstorage]
type = s3
provider = Other
env_auth = false
access_key_id = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
endpoint = yourotherendpoint.com
acl = private
allas-conf
After creating/updating this file, Rclone can be used to copy files
rclone copy otherobjectstorage:bucket-x/object-y s3allas:bucket-z/object-a
or list files from either Allas or the other object storage by using the respective name
rclone lsf otherobjectstorage:.