Skip to content

Docs CSC now features an automatic Finnish translation. Click here for more information.

Warning!

Puhti and Mahti are being decommissioned in stages, and their storage areas will become fully unavailable from 15 October 2026. Clean up unnecessary files and move any data you need to keep by 31 August 2026. See the Roihu data migration guide for instructions on transferring your data to Roihu.

Puhti computing services have been decommissioned and no new jobs are accepted or executed on its compute nodes. Puhti login nodes and storage services are planned to remain available until 15 October 2026.

Submitting jobs across architectures

Overview

On Roihu, the CPU nodes are x86 and the GPU nodes are ARM, so the system has separate login nodes for each architecture. Normally you should submit CPU jobs from roihu-cpu.csc.fi and GPU jobs from roihu-gpu.csc.fi. If you have a valid reason to cross architectures, for example when a workflow on a CPU node needs to launch a GPU job, some extra care is needed to make sure the job does not end up using software built for the wrong architecture.

The problem is that Slurm copies the whole submitting environment to the job by default (--export=ALL). Variables such as PATH, LD_LIBRARY_PATH and the Lmod/module state then arrive pointing at binaries and module trees built for the submitting architecture, and the job either fails or silently picks up the wrong software. Avoid this as follows:

  1. Limit what is propagated with --export, so that the target node builds its own environment from scratch. HOME and TERM are usually enough, and Slurm's own SLURM_* variables are always passed. Add only the variables your job genuinely needs.

  2. Start the job in a login shell, given by full path (/bin/bash --login), so that the user environment, including the module system, is initialized on the target node. In batch jobs, also set CSC_ENV_INIT_NON_INTERACTIVE=yes, which makes /etc/profile.d/zz-csc-env.sh initialize the CSC environment in a non-interactive shell. Interactive shells do this by default, as they set PS1.

  3. In batch jobs, set SLURM_EXPORT_ENV=ALL inside the script. Otherwise srun would inherit the restricted --export list instead of the environment the batch script has just set up on the target node.

Interactive job

srun --account=<project> \
    --partition=gputest \
    --nodes=1 \
    --ntasks-per-node=1 \
    --cpus-per-task=72 \
    --gres=gpu:gh200:1 \
    --time=0:15:00 \
    --export=HOME,TERM \
    --pty /bin/bash --login

Batch job

batch.sh
#!/bin/bash --login
#SBATCH --export=HOME,CSC_ENV_INIT_NON_INTERACTIVE=yes
#SBATCH --account=<project>
#SBATCH --partition=gputest
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=72
#SBATCH --nodes=1
#SBATCH --gres=gpu:gh200:1
#SBATCH --time=0:15:00
export SLURM_EXPORT_ENV=ALL

# Write your code here