-
Submitting jobs across architectures
Submitting jobs across architectures
Overview
On Roihu, the CPU nodes are x86 and the GPU nodes are ARM, so the system has
separate login nodes for each architecture. Normally you
should submit CPU jobs from roihu-cpu.csc.fi and GPU jobs from
roihu-gpu.csc.fi. If you have a valid reason to cross architectures, for
example when a workflow on a CPU node needs to launch a GPU job, some extra care
is needed to make sure the job does not end up using software built for the
wrong architecture.
The problem is that Slurm copies the whole submitting environment to the job by
default (--export=ALL). Variables such as PATH, LD_LIBRARY_PATH and the
Lmod/module state then arrive pointing at binaries and module trees built for
the submitting architecture, and the job either fails or silently picks up the
wrong software. Avoid this as follows:
-
Limit what is propagated with
--export, so that the target node builds its own environment from scratch.HOMEandTERMare usually enough, and Slurm's ownSLURM_*variables are always passed. Add only the variables your job genuinely needs. -
Start the job in a login shell, given by full path (
/bin/bash --login), so that the user environment, including the module system, is initialized on the target node. In batch jobs, also setCSC_ENV_INIT_NON_INTERACTIVE=yes, which makes/etc/profile.d/zz-csc-env.shinitialize the CSC environment in a non-interactive shell. Interactive shells do this by default, as they setPS1. -
In batch jobs, set
SLURM_EXPORT_ENV=ALLinside the script. Otherwisesrunwould inherit the restricted--exportlist instead of the environment the batch script has just set up on the target node.
Interactive job
srun --account=<project> \
--partition=gputest \
--nodes=1 \
--ntasks-per-node=1 \
--cpus-per-task=72 \
--gres=gpu:gh200:1 \
--time=0:15:00 \
--export=HOME,TERM \
--pty /bin/bash --login
Batch job
#!/bin/bash --login
#SBATCH --export=HOME,CSC_ENV_INIT_NON_INTERACTIVE=yes
#SBATCH --account=<project>
#SBATCH --partition=gputest
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=72
#SBATCH --nodes=1
#SBATCH --gres=gpu:gh200:1
#SBATCH --time=0:15:00
export SLURM_EXPORT_ENV=ALL
# Write your code here