Skip to content

Docs CSC now features an automatic Finnish translation. Click here for more information.

Warning!

Puhti and Mahti are being decommissioned in stages, and their storage areas will become fully unavailable from 15 October 2026. Clean up unnecessary files and move any data you need to keep by 31 August 2026. See the Roihu data migration guide for instructions on transferring your data to Roihu.

Puhti computing services have been decommissioned and no new jobs are accepted or executed on its compute nodes. Puhti login nodes and storage services are planned to remain available until 15 October 2026.

Extend CSC-based ML containers using sandbox on Roihu

This guide is also part of our Machine learning guide.

Motivation

Using the pre-installed PyTorch modules on Roihu is convenient for many workflows. However, when working with different libraries, users may need PyTorch versions or other libraries that have not been installed by CSC.

For example, the latest fairchem-core release requires a recent PyTorch version. In this guide, we demonstrate how to build a custom Apptainer container based on the CSC ml-base image and install PyTorch 2.13.0 with CUDA 13.0 together with fairchem-core 2.22.0.

The workflow uses an Apptainer sandbox as a writable environment during installation. Once the required software has been installed, the sandbox is converted into a portable .sif image that can be used for subsequent FairChem workloads.

Note

The commands below are intended to be run on Roihu, but it should be easily adaptable to other clusters.

Allocate resources

Start an interactive GPU job with the required resources, here we ask for 1 GPU for 1 hour, we do not encourage users to ask more than what they need since GPUs are valuable resources:

salloc --account=project_***** \
    --nodes=1 \
    --ntasks-per-node=1 \
    --cpus-per-task=72 \
    --time=01:00:00 \
    --partition=gpuinteractive \
    --gres=gpu:gh200:1

Set the Apptainer cache directory

Set the Apptainer cache directory to $TMPDIR to avoid filling your home directory:

export APPTAINER_CACHEDIR="$TMPDIR/apptainer-cache"

Create a writable sandbox from ml-base

Initialize an Apptainer sandbox using the CSC ml-base image:

apptainer build --fakeroot --sandbox "$TMPDIR/mlbase" \
    docker://satama.csc.fi/r_installation_aida/ml-base:rocky9.7_gcc12_py3.12_cuda13

After a successful build, you should see a message similar to:

INFO:    Build complete: /tmp/shanshan/715617/mlbase

The exact path will depend on your $TMPDIR.

The --sandbox option creates a writable directory containing the extracted container filesystem. This allows you to install and modify software interactively before creating the final SIF image.

Warning

Do not create a sandbox on the shared Lustre file system (for example on /scratch, /projappl or /home) as it will create a lot of small files which can slow down the system for all users!

Create the users directory

Create the users directory inside the sandbox:

mkdir -p "$TMPDIR/mlbase/users"

Enter the sandbox

Start a writable shell inside the sandbox. The --nv option makes the NVIDIA GPU and relevant NVIDIA libraries available inside the container:

apptainer shell --fakeroot --writable --nv \
    --contain --cleanenv \
    --bind="$TMPDIR:/tmp" \
    "$TMPDIR/mlbase"

You are now working inside the writable mlbase sandbox.

Configure the pip cache

Inside the container, configure pip to use a cache directory under /tmp:

export PIP_CACHE_DIR=/tmp/pip-cache

Because $TMPDIR is bound to /tmp inside the container, the pip cache is stored outside the container image.

Workaround for useradd and groupadd

If package installation fails because useradd or groupadd cannot be executed in the sandbox, replace these commands with /usr/bin/true:

cp /usr/bin/true /usr/sbin/useradd
cp /usr/bin/true /usr/sbin/groupadd

This prevents package installation scripts from failing when they attempt to create system users or groups.

Install PyTorch 2.13.0 with CUDA 13.0

Install PyTorch 2.13.0 and torchvision using the CUDA 13.0 PyTorch wheel index:

pip install torch==2.13.0 torchvision \
    --index-url https://download.pytorch.org/whl/cu130

Optionally, verify the installation:

python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"

The output should show the installed PyTorch version and indicate whether CUDA is available.

Install fairchem-core 2.22.0

pip install fairchem-core==2.22.0

Verify:

python -c "from fairchem.core import pretrained_mlip, FAIRChemCalculator; print('fairchem ok')"

Exit the sandbox

Once all required software has been installed, exit the container:

exit

You should now be back in the host environment.

Build the SIF image

Convert the writable sandbox into a standard Apptainer SIF image:

apptainer build --fakeroot fairchem.sif "$TMPDIR/mlbase"

The resulting image will be in the file fairchem.sif. The SIF format is a portable, read-only Apptainer image that can be used for subsequent jobs.

Verify the SIF image

Check the size of the generated container:

ls -lh fairchem.sif

The SIF file will typically be several GB in size, depending on the packages installed in the container.

Test the container

Finally, verify that both FairChem and PyTorch can be imported from the generated SIF image:

apptainer exec fairchem.sif \
    python -c "import fairchem.core, torch; print(torch.__version__)"

In order to access the GPU, add the --nv flag. In addition, on Roihu, we can use csc-common-bind command to list the bind mounts to common disk areas such as /scratch and /projappl:

apptainer exec --bind="$(csc-common-bind)" --nv fairchem.sif \
    python -c "import torch; print('PyTorch:', torch.__version__); print('CUDA available:', torch.cuda.is_available())"
If the commands complete successfully, the custom FairChem container has been built successfully. You can move the resulting SIF file to the proper location such as in the project's directory in /projappl.