Local Rclone configuration for Allas
Rclone can be installed in any operating system and thus it provides an effective way to use Allas from any computer. You can download a copy of Rclone to your own computer form the link below.
Once you have Rclone installed you still need to configure the connection to Allas. Rclone is a command line application so both configurarion and actual usage is normally done in a command line: Terminal in Mac and Linux, Command prompt or Powershell in Windows.
Configuring Allas connection in Mac and Linux (Swift and S3)
If you are using Rclone in a local Linux or Mac machine, you can download
allas_conf script to set up the connection to your Allas project.
wget https://raw.githubusercontent.com/CSCfi/allas-cli-utils/master/allas_conf source allas_conf -u your-csc-username -p your-csc-project-name
-uoption to define your CSC username and
-pto define the CSC project you want to use. For example:
source allas_conf -u kkayttaj -p project_2001234
rclone lsd allas:
source allas_conf -u kkayttaj -p project_2001234 -m S3
rclone lsd s3allas:
source allas_conf -u csc-user-name -p csc-project-name --s3remove
Remember to be careful and security aware, when configuring S3 connection to Allas. The S3 keys are stored in a readble format in you home directory and anyone how can read your keys can access Allas until the keys are expilisitly revoked from Allas. Removing the keys from your own computer is not enough to deactivate them.
Configuring Allas connection in Windows
In Windows machines you can't use allas_conf script. So for Windows some connection specific variables need to be checked up in parallel from a Linux/Mac supporting allas_conf script. Easiest could be Puhti, if you use it otherwise. One option is also to use Windows Linux Subsystem and then the instructions above can be followed.
Configure the Allas connetion with command:
Below we describe how to create Swift and S3 connections to Allas.
Configuring Swift connection in Windows
Start the process by executing command:
- Select n to create a New remote
- Name the remote as: allas
- From the list of storage protocols, select the number that defines: OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
- Select authentication option 2 Get swift credentials from environment vars.
- After that select the default blank setting for all the remaining settings until you are back in the starting menu of the confguration process.
- Finally, choose q to stop the configuration process.
In the case of Swift you need to do this configuration only once. In the configuration it is now defined that in case of allas, all data for the connection is red from environment variables.
If you have access to Puhti, then the easiest way to check the values of the variables needed is to open terminal connection to it and activate there connection to the Allas project you wish to use. Another option is to use a utility program allas-get-swift-token-win.zip. If you choose to use the utility program, you (or your local it-support) may have to configure your anti-virus etc software to allow running it.
This Swift access is done in Puhti with commands:
module load allas allas-conf
- OS_STORAGE_URL, Check this value in Puhti with commad echo $OS_STORAGE_URL .
- OS_AUTH_TOKEN, Check this value in Puhti with commad echo $OS_AUTH_TOKEN .
The value in the variable OS_STORAGE_URL stays the same as long as you are accessing the same project. OS_AUTH_TOKEN variable contains the actual authentication token. This token is valid only for 8 hours so you typically need to generate and check it each time you start using Rclone in your computer.
At the moment we don't have a tool for generating this value in Windows so you need to use Puhti (or some other machine that can run allas_conf tool) to generate this project specific temporary token.
For example, if in Puhti running the allas-conf would produce following values:
module load allas allas-conf echo $OS_STORAGE_URL https://a3s.fi:443/swift/v1/AUTH_5d66718fa0ff46ee1b2581b2225ee1a9d echo $OS_AUTH_TOKEN gAAAAABiVAJ08lSVPXH4MqXDb-Vo0KdQgm9reOA7IRf-PavF-RktYpp3K_7kp-5Ck137hsBQl66qRFmd4PE4hEVz0TWwTTmVIPUmmxQ
Then in Windows PowerShell you could set up Allas connection with commands:
$Env:OS_STORAGE_URL = "https://a3s.fi:443/swift/v1/AUTH_5d66718fa0ff46ee1b2581b2225ee1a9d" $Env:OS_AUTH_TOKEN = "gAAAAABiVAJ08lSVPXH4MqXDb-Vo0KdQgm9reOA7IRf-PavF-RktYpp3K_7kp-5Ck137hsBQl66qRFmd4PE4hEVz0TWwTTmVIPUmmxQ"
In the Windows Command Prompt the corresponding set up commands would be:
set OS_STORAGE_URL=https://a3s.fi:443/swift/v1/AUTH_5d66718fa0ff46ee1b2581b2225ee1a9d set OS_AUTH_TOKEN=gAAAAABiVAJ08lSVPXH4MqXDb-Vo0KdQgm9reOA7IRf-PavF-RktYpp3K_7kp-5Ck137hsBQl66qRFmd4PE4hEVz0TWwTTmVIPUmmxQ
With these settings done, you should be able to check your buckets with command:
rclone.exe lsd allas:
Configuring S3 connection in Windows
Remember that you need to be careful and security aware, when configuring S3 connection to Allas. The S3 keys are stored in a readble format in you home directory and anyone how can read your keys can access Allas until the keys are expilisitly revoked from Allas. Removing the keys from your own computer is not enough to deactivate them.
To check your S3 keys, open connection to Puhti (or some other machine that can run allas_conf tool). If you don't yet have S3 connetion to your Allas project activated, open it with commands:
module load allas allas-conf -m S3
Then switch to you Windows Command Prompt and start the rclone configuration process by executing command:
- Select n to create a New remote
- Name the remote as: s3allas
- From the list of storage protocols, select the number that defines: Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
- Next you are asked to Choose your S3 provider. Select option that provides Any other S3 compatible provider
- The select that you Enter AWS credentials in the next step
- Give the AWS access key. You can check this in Puhti with command: grep access_key $HOME/.s3cfg | cut -d " " -f3
- Give the AWS secret access key. You can check this in Puhti with command: grep secret_key $HOME/.s3cfg | cut -d " " -f3
- Region 1
- Endpoint: a3s.fi
- Location constant: (blank, just press entwer)
- acl: 1
- Edit advanced config: n
- Remote config: y Yes, this is ok.
- q, quit config
After this you can access your Allas project using Rclone remote s3allas. For example:
rclone.exe lsd s3allas:
This connection stays active in your laptop until you delete it.