Using Allas and Lumi-O from LUMI supercomputer
At the moment the LUMI object storage service, LUMI-O, is not yet in use and tools for using object storage services are not by default installed in LUMI-C. In this document we describe how you can install commonly used object storage clients to LUMI-C and how to configure connection to Allas object storage services. These tools can be used to access Lumi-O too when it becomes available.
Installing Object Storage tools for Allas and Lumi-O
In this example we do the installation to the /project area in Lumi so that all project members can utilize these tools. In this example we use example project 462000007. Please use your own project number when installing these tools for your own use.
Once you have logged in to Lumi, move to the /project disk area and make there a directory called allas.
cd /project/project_462000007 mkdir allas cd allas
git clone https://github.com/CSCfi/allas-cli-utils cd allas-cli-utils
Allas-cli-utils has a list of software dependencies that needs to be be installed. We use here the lumi-container-wrapper tool to install most of them (s3cmd, Python OpenStack clinet, Python swift client, zstdmt and crypt4gh). The installation is done with commands:
module load LUMI lumi-container-wrapper conda-containerize new --prefix /project/project_462000007/allas allas-dependencies.yaml
Finally we need to modify a_env_conf file a bit so that a-tools works smoothly. Open the file:
In this file do following modifications (remember to use your own project number instead of 462000007)
local_host=”lumi” allas_conf_path="/project/project_462000007/allas/allas-cli-utils/allas_conf" tmp_root="/scratch/project_462000007"
Next move to the allas directory and create a set up file allas_setup.sh which will be use to simplify the setup process of Allas tools
cd /project/project_462000007/allas nano allas_setup.sh
export PATH=/project/project_462000007/allas/allas-cli-utils:/project/project_462000007/allas/bin:$PATH alias allas-conf="source /project/project_462000007/allas/allas-cli-utils/allas_conf"
Now the installation is ready. In the future you and your group members need to only run the setup commands described below, to enable object storage tools and to open connection to Allas and Lumo-O
Using Lumi-O and Allas
Once the object storage tools have been installed to the project directory as described above, then opening connection to Allas or Lumi-O requires first setting up the environment with command:
After this commands like allas-conf, rclone, swift or a-put should work in the same way as in Puhti and Mahti.
Running command allas-conf starts normal configuration process for a swift based connetion to Allas:
If you want to configure connetion Lumi-O, run command:
For rclone, Lumi-o configuration provides two rclone remotes: lumi-o: and lumi-pub: . The buckets used by lumi-pub will be publicly visible in URL: https://project-number.lumidata.eu/bucket_name.
Note, that you can have active connection to both Lumi-O and Allas in the same time.
For example, if you would first open Allas connection with command:
source /project/project_your-project-number/allas/allas_setup.sh allas-conf
rclone lsd lumi-o:
rclone lsd allas:
Copying data from Allas to Lumi-O could now be done with command:
rclone copyto -P allas:bucket-in-allas/object lumi-o:bucket-in-lumi-o/object