Skip to content

High-throughput Nextflow workflow using HyperQueue

The Slurm executor of Nextflow is not suitable for large workflows involving tens of thousands of processes on HPC systems. This is due to the usage of individual jobs/job steps that bloat the Slurm log and degrade the performance of the batch job scheduler. To maximize throughput and performance, we recommend the Nextflow workflow tasks to be run using the HyperQueue meta-scheduler as the executor. This page provides an example batch script for this purpose.


Whenever you're unsure how to run your workflow efficiently, don't hesitate to contact CSC Service Desk.

Example batch script

#SBATCH --partition=medium
#SBATCH --account=project_2001659 
#SBATCH --nodes=3
#SBATCH --exclusive
#SBATCH --time=00:10:00

# Load the required modules
module load hyperqueue
module load nextflow

# Create a per job directory
mkdir -p $wrkdir/.hq-server

# Set the directory which hyperqueue will use 
export HQ_SERVER_DIR=$wrkdir/.hq-server

# Make sure nextflow uses the right executor and
# knows how much it can submit.
echo "executor {
  queueSize = $(( 128*SLURM_NNODES ))
  name = 'hq'
  cpus = $(( 128*SLURM_NNODES )) 
}" > $wrkdir/nextflow.config

cp $wrkdir
hq server start &
srun --cpu-bind=none --hint=nomultithread --mpi=none -N $SLURM_NNODES -n $SLURM_NNODES -c 128 hq worker start --cpus=128 &

num_up=$(hq worker list | grep RUNNING | wc -l)
while true; do

    echo "Checking if workers have started"
    if [[ $num_up -eq $SLURM_NNODES ]];then
        echo "Workers started"
    echo "$num_up/$SLURM_NNODES workers have started"
    sleep 1
    num_up=$(hq worker list | grep RUNNING | wc -l)


cd $wrkdir
nextflow run

# Make sure we exit cleanly once nextflow is done
hq worker stop all
hq server stop

Where would be the nextflow script you want to run. Note that this batch script creates a per job directory and copies the nextflow script there before starting.

$ ls 
$ sbatch
Submitted batch job 137
$ ls WRKDIR-137
$ ls WRKDIR-137 work

More information

Last update: October 5, 2022