Skip to content

Docs CSC now features an automatic Finnish translation. Click here for more information.

Warning!

Puhti and Mahti are being decommissioned in stages, and their storage areas will become fully unavailable from 15 October 2026. Clean up unnecessary files and move any data you need to keep by 31 August 2026. See the Roihu data migration guide for instructions on transferring your data to Roihu.

Puhti scratch is very full: keep only active data there and move or delete everything else. No new Puhti scratch quota will be granted.

Large file transfer using S3

FirecREST HPC API implements asynchronous large file transfer model using an S3 object storage as a staging medium. On Roihu, the S3 storage used is Allas. FirecREST uses its own dedicated tenant with dynamically created user-specific buckets.

How it works

Upon sending an API request to upload or download task, FirecREST:

  • Creates a user-specific bucket in its own Allas tenant.
  • Generates pre-signed S3 URLs for the data transfer, using multi-part model if necessitated by the given fileSize.
  • Schedules a new Slurm DataTransferJob using account (your project) defined in the API request, to either transfer files from Allas to Roihu after your upload is completed, or from Roihu to Allas for you to download.
  • Returns pre-signed URLs to API client for transferring the data.

Examples

Various detailed examples for implementing S3 data transfer using bash, .NET and PyFirecREST can be found in FirecREST v2 user guide.