Skip to content

Call for Roihu pilot projects is now open! Read the article to learn more & apply for early access.

Table of contents of user guide

Uploading and encrypting data

Files uploaded to SD Connect are automatically encrypted and decrypted during download, using the service’s integrated key management system. This functionality supports all file types and formats, with a maximum file size of 100 GB. Larger files or folders can be uploaded programmatically.

Note

All members of the same CSC project have access to upload and download files stored via SD Connect. This can be limited by sharing files with Read to SD Desktop permission to a different CSC project Academic type. Please contact servicedesk@csc.fi (subject: SD Connect) for assistance.

Step by step

1. Plan the number of folders needed

SD Connect is built on a cloud object storage infrastructure. Files can only be uploaded into top-level or 'main folders' created with SD Connect: a top-level 'box' used to store files or folders. This has several implications for how your data should be organized and managed:

  • Once files are uploaded to SD Connect, they cannot be edited or modified. It is therefore important to plan the folder structure in advance. To simplify data management and avoid issues, it is recommended to create a separate folder for each dataset or experiment. Avoid placing too many files in a single folder, each folder can contain up to 500.000 segmented files.

  • Subfolders not supported: Uploading files into subfolders is not supported.

  • Upload duration: Uploading large files or large batches may take several hours. Uploads are automatically stopped after 8 hours.

  • File segmentation: Uploaded files are automatically split into segments to optimize storage and performance. This segmentation is not visible in the user interface but can affect performance.

  • The user interface might be slower when there are more than 2500 files for each folder. In this case please use the command-line tools for upload and automated key management.

2. Plan folder names

When creating folders in SD Connect, specific naming rules must be followed to ensure compatibility, which requires some planning.

Note

Top level folder name can not be modified after their creation with SD Connect. These rules apply only to top-level folders created in the service, not to subfolders uploaded from a local computer.

Folder names must:

  • start with a lowercase letter or a number.
  • be between 3 and 63 characters long.
  • use Latin alphabets (a-z), numbers (0-9) and dash (-).
  • be unique across all existing folders in all projects in SD Connect and Allas. If you can't create a new folder, another project may already use the name you have chosen. To avoid this situation, it is good practice to include project specific identifiers (e.g., project ID number or acronym) in the folder name.

Folder names must not contain:

  • Uppercase letters, underscore (_) and accent letters with diacritics or special marks (åäöe') are not allowed.
  • all folder names are public; please do not include any confidential information.
  • Folder names can't be modified afterwards.

3. Accept cookies

File uploads are supported in Google Chrome and Mozilla Firefox (incognito mode not supported) browsers.

On first use, a browser pop-up may request cookie consent. Accepting cookies enables file and folder uploads. This action is required only once.

4. Upload and encrypt files to a new folder

  1. Log in to SD Connect.
  2. Select the correct CSC project in the top left corner.
  3. Click Upload in the top right corner.
  4. In the new window, name the destination folder for your files taking into consideration that some caractes are not allowed: uppercase letters, underscores (_), and letters with diacritics or special marks (e.g., å, ä, ö, é) (please see paragraph 2).
  5. Click Select Files to open a browser window and choose files for upload. If you want to upload folders, drag and drop them into the window. Click Upload to start automatic encryption and upload.
  6. Notification about the status of upload will appear and be visible until the upload is completed. Notification also includes a link to the destination folder.
  7. Once the upload is finished, the encrypted files will display the extension .c4gh, this means that they have been successfully encrypted.
  8. Now the files are accessible for downloading and sharing via SD Connect or for analysis, editing or annotation once imported via SD Desktop.
  9. Once files are uploaded and stored, the service begins consuming billing units. The default storage quota is 10 TB. When this limit is reached, uploads will no longer be accepted until additional storage is allocated.

SD Connect Upload

5. Upload and encrypt files to an existing folder

  1. Select the correct folder (by double-click).
  2. Click Upload in the top right corner and follow steps from 4 to 6 in above paragraph.

6. Create a folder

You can create a folder and upload files to it later.

  1. Click Create folder.
  2. Name your folder taking into consideration that some characters are not allowed: uppercase letters, underscores (_), and letters with diacritics or special marks (e.g., å, ä, ö, é) (please see paragraph 2).
  3. Click Save.

SD Connect Create folder

The next steps in this guide