Large file transfer using S3

FirecREST HPC API implements asynchronous large file transfer model using an S3 object storage as a staging medium. On Roihu, for example, the S3 storage used is Allas. FirecREST uses its own dedicated tenant with dynamically created user-specific buckets.

How it works

Upon sending an API request to upload or download task, FirecREST:

• Creates a user-specific bucket in its own Allas tenant.
• Generates pre-signed S3 URLs for the data transfer, using multi-part model if necessitated by the given fileSize.
• Schedules a new Slurm DataTransferJob using account (your project) defined in the API request, to either transfer files from Allas to Roihu after your upload is completed, or from Roihu to Allas for you to download.
• Returns pre-signed URLs to API client for transferring the data.

Examples

Various detailed examples for implementing S3 data transfer using bash, .NET and PyFirecREST can be found in FirecREST v2 user guide.

Do you find this page useful?

Yes

Thank you for your feedback!

No

Thank you! Your feedback is appreciated.

Please let us know how we could improve this page by contacting us or by creating an issue in GitHub. You can also edit this page yourself and contribute your improvements by creating a pull request.