Inspecting and Controlling CPU Affinity
This tutorial describes how CPU affinities can be inspected and controlled in job scripts on CSC supercomputers. The tutorial scripts are written for the Roihu supercomputer, but are generally also applicable on Puhti, Mahti and LUMI.
What is CPU Affinity?
CPU affinity describes how a running program is placed on the available CPU cores of a supercomputer node. By controlling affinity, we decide which CPU cores a program is allowed to run on, limiting or fixing its placement rather than leaving it entirely to the system.
In high‑performance computing, setting affinity correctly is important for performance. It helps programs make better use of fast processor caches and memory, reduces unnecessary movement between cores, and leads to more stable and predictable runtimes.
Inspecting CPU Affinity in Slurm Jobs
The following example job script can be used for checking the CPU affinities of each Slurm task.
The job script executes csc-print-affinity (available in csc-tools module) via srun:
#!/bin/bash
#SBATCH --job-name=example
#SBATCH --account=<project>
#SBATCH --partition=medium
#SBATCH --time=00:30:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8 --cpus-per-task=48 # The product should be 384
# Run the program
srun csc-print-affinity
Example output for this job:
Task 10 running on node rc6284 core 96-143
Task 11 running on node rc6284 core 144-191
Task 9 running on node rc6284 core 48-95
Task 15 running on node rc6284 core 336-383
Task 12 running on node rc6284 core 192-239
Task 13 running on node rc6284 core 240-287
Task 14 running on node rc6284 core 288-335
Task 8 running on node rc6284 core 0-47
Task 7 running on node rc6283 core 336-383
Task 1 running on node rc6283 core 48-95
Task 3 running on node rc6283 core 144-191
Task 6 running on node rc6283 core 288-335
Task 4 running on node rc6283 core 192-239
Task 2 running on node rc6283 core 96-143
Task 5 running on node rc6283 core 240-287
Task 0 running on node rc6283 core 0-47
This output shows that CSC supercomputers are configured so that, by default, each Slurm task is assigned
its own exclusive set of CPU cores from the allocation.
The number of cores reserved per task is determined by the Slurm option cpus-per-task.
Advanced: Controlling CPU Affinity Manually in Slurm Jobs
Advanced topic
This section describes advanced, manual control of CPU affinity. In practice, this is rarely needed, and the default Slurm configuration is best for most workloads. If you believe you need manual CPU binding, please contact us first for guidance.
If a different placement strategy is needed, the default CPU binding can be disabled by srun --cpu-bind=none.
This removes all CPU affinity restrictions, allowing processes to run on any allocated core on the node.
This is generally undesirable for performance unless combined with explicit manual binding,
which can be done using tools such as numactl.
The following job script provides an example for setting CPU binding manually.
The job script creates the script cpu_bind.<jobid>.sh that
uses numactl for binding tasks to CPU cores in the same way as done by default:
each task is assined a contiguous block of CPU cores based on the task's local ID and
the number of CPUs per task:
Script for full nodes
This script works correctly only on full nodes.
#!/bin/bash
#SBATCH --job-name=example
#SBATCH --account=<project>
#SBATCH --partition=medium
#SBATCH --time=00:30:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8 --cpus-per-task=48 # The product should be 384
# Create a script for binding tasks to CPU cores
BIND_CPU="./bind_cpu.$SLURM_JOB_ID.sh"
cat << 'EOF' > $BIND_CPU
#!/bin/bash
cpus_per_task=${SLURM_CPUS_PER_TASK:-1}
start_core=$((SLURM_LOCALID * cpus_per_task))
end_core=$((SLURM_LOCALID * cpus_per_task + cpus_per_task - 1))
numactl --physcpubind ${start_core}-${end_core} "$@"
EOF
chmod +x $BIND_CPU
# Remove the script on exit
trap "rm -f $BIND_CPU" EXIT
# Run the program with manual binding
srun --cpu-bind=none $BIND_CPU csc-print-affinity
This produces output equivalent to the default CPU binding seen above, confirming that the manual placement reproduces the standard behaviour:
Task 4 running on node rc6283 core 192-239
Task 6 running on node rc6283 core 288-335
Task 7 running on node rc6283 core 336-383
Task 5 running on node rc6283 core 240-287
Task 0 running on node rc6283 core 0-47
Task 1 running on node rc6283 core 48-95
Task 2 running on node rc6283 core 96-143
Task 3 running on node rc6283 core 144-191
Task 15 running on node rc6284 core 336-383
Task 8 running on node rc6284 core 0-47
Task 9 running on node rc6284 core 48-95
Task 10 running on node rc6284 core 96-143
Task 11 running on node rc6284 core 144-191
Task 12 running on node rc6284 core 192-239
Task 13 running on node rc6284 core 240-287
Task 14 running on node rc6284 core 288-335
If needed, the logic in the binding script can be modified to bind CPU cores according to the needs of your application.
Inspecting and Controlling CPU Affinity of MPI+OpenMP Applications
Many HPC applications benefit from binding OpenMP threads to CPU cores. This does not happen automatically but need to be enabled with the following lines in the batch job script:
This tutorial illustrates the meaning of these settings.
First, let's check the default behaviour in the case when threads are not bound.
The following job script exemplifies the use of OMP_* for checking the CPU affinities of each process and OpenMP thread:
#!/bin/bash
#SBATCH --job-name=example
#SBATCH --account=<project>
#SBATCH --partition=small
#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=1000M
# Set the number of threads based on cpus-per-task
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
# Print thread affinities
export OMP_DISPLAY_AFFINITY=true
export OMP_AFFINITY_FORMAT="Process %P level %L thread %0.4n/%0.4N on node %H core %A"
# Run the program
srun <my_openmp_program>
Example output for this job:
Process 2808761 level 1 thread 0000/0004 on node rc6224 core 0-3
Process 2808761 level 1 thread 0001/0004 on node rc6224 core 0-3
Process 2808761 level 1 thread 0002/0004 on node rc6224 core 0-3
Process 2808761 level 1 thread 0003/0004 on node rc6224 core 0-3
Process 2808762 level 1 thread 0000/0004 on node rc6224 core 4-7
Process 2808762 level 1 thread 0001/0004 on node rc6224 core 4-7
Process 2808762 level 1 thread 0002/0004 on node rc6224 core 4-7
Process 2808762 level 1 thread 0003/0004 on node rc6224 core 4-7
This output means that the threads of the processes are free to move between the sets of four cores (0-3 or 4-7), which can lead to worse performance due to increased context switching and thread migration during execution, as opposed to a case where threads are bound to single cores.
In the following example job script, we have enabled thread binding:
#!/bin/bash
#SBATCH --job-name=example
#SBATCH --account=<project>
#SBATCH --partition=small
#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=1000M
# Set the number of threads based on cpus-per-task
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
# Place and bind threads to single cores
export OMP_PLACES=cores
export OMP_PROC_BIND=spread
# Print thread affinities
export OMP_DISPLAY_AFFINITY=true
export OMP_AFFINITY_FORMAT="Process %P level %L thread %0.4n/%0.4N on node %H core %A"
# Run the program
srun <my_openmp_program>
Now the output looks like this:
Process 2808761 level 1 thread 0000/0004 on node rc6224 core 0
Process 2808761 level 1 thread 0001/0004 on node rc6224 core 1
Process 2808761 level 1 thread 0002/0004 on node rc6224 core 2
Process 2808761 level 1 thread 0003/0004 on node rc6224 core 3
Process 2808762 level 1 thread 0000/0004 on node rc6224 core 4
Process 2808762 level 1 thread 0001/0004 on node rc6224 core 5
Process 2808762 level 1 thread 0002/0004 on node rc6224 core 6
Process 2808762 level 1 thread 0003/0004 on node rc6224 core 7
This output means that each thread is bound to its own core, which is often desired for best performance.
In general, when starting to use a new application or configuration, it is essential to use these settings to verify thread placement and confirm that affinity is applied as expected, which is often a key step when diagnosing or optimizing performance on a supercomputer.
If the output would show that several processes or threads are bound to the same core on the same node, for example
Process 2808761 level 1 thread 0000/0004 on node rc6224 core 0
Process 2808761 level 1 thread 0001/0004 on node rc6224 core 1
Process 2808761 level 1 thread 0002/0004 on node rc6224 core 2
Process 2808761 level 1 thread 0003/0004 on node rc6224 core 3
Process 2808762 level 1 thread 0000/0004 on node rc6224 core 0
Process 2808762 level 1 thread 0001/0004 on node rc6224 core 1
Process 2808762 level 1 thread 0002/0004 on node rc6224 core 2
Process 2808762 level 1 thread 0003/0004 on node rc6224 core 3
Process 2808761 level 1 thread 0000/0004 on node rc6224 core 0
Process 2808761 level 1 thread 0001/0004 on node rc6224 core 0
Process 2808761 level 1 thread 0002/0004 on node rc6224 core 0
Process 2808761 level 1 thread 0003/0004 on node rc6224 core 0
Process 2808762 level 1 thread 0000/0004 on node rc6224 core 0
Process 2808762 level 1 thread 0001/0004 on node rc6224 core 0
Process 2808762 level 1 thread 0002/0004 on node rc6224 core 0
Process 2808762 level 1 thread 0003/0004 on node rc6224 core 0
Please do not hesitate to contact us if you need help with ensuring good performance for your application.