Skip to content

A new version of SD Connect and SD Desktop will be available from Monday, September 28. The major upgrade will introduce significant improvements, but also includes changes that are not compatible with the current version of the service. Click here to review the available support materials.

Warning!

Puhti and Mahti computing services have been decommissioned and no new jobs are accepted or executed on its compute nodes. Puhti and Mahti login nodes and storage services are planned to remain available until 15 October 2026. Clean up unnecessary files and move any data you need to keep by 31 August 2026. See the Roihu data migration guide for instructions on transferring your data to Roihu.

Large file transfer using S3

FirecREST HPC API implements asynchronous large file transfer model using an S3 object storage as a staging medium. On Roihu, the S3 storage used is Allas. FirecREST uses its own dedicated tenant with dynamically created user-specific buckets.

How it works

Upon sending an API request to upload or download task, FirecREST:

  • Creates a user-specific bucket in its own Allas tenant.
  • Generates pre-signed S3 URLs for the data transfer, using multi-part model if necessitated by the given fileSize.
  • Schedules a new Slurm DataTransferJob using account (your project) defined in the API request, to either transfer files from Allas to Roihu after your upload is completed, or from Roihu to Allas for you to download.
  • Returns pre-signed URLs to API client for transferring the data.

Examples

Various detailed examples for implementing S3 data transfer using bash, .NET and PyFirecREST can be found in FirecREST v2 user guide.