Nextflow

Nextflow is a scientific workflow management system for creating scalable, portable, and reproducible workflows. Pipelines are written in a Groovy-based language and can embed scripts in other languages such as R, bash and Python. Nextflow has built-in support for HPC-friendly containers such as Apptainer (= Singularity).

A key advantage is that the pipeline logic is separated from the execution environment: the same script runs in different environments by changing the configured executor, which tells Nextflow where to submit each process. The default local executor runs processes on the machine where Nextflow is launched; on CSC systems the SLURM and HyperQueue executors are the best fit. For choosing a workflow tool more generally, see our high-throughput computing and workflows page.

• Nextflow
  • Available
  • License
  • Installation
    • Nextflow
    • Installation of tools used in Nextflow
  • Usage
    • Nextflow script
    • Running Nextflow pipelines
  • References
  • More information

Available

Versions available on CSC's servers

• Roihu-CPU: 25.10.2-standalone, 26.04.6-standalone (via the bio-apps module)
• Roihu-GPU: 25.10.2-standalone, 26.04.6-standalone (via the bio-apps module)
• LUMI: 22.10.4

Pay attention to usage of Nextflow version

Please note that the nextflow version starting from 23.04.3 can only be used for pipelines built with DSL2. You can downgrade to lower versions for DSL1-compliant pipelines.

License

Nextflow is released under the Apache 2.0 license.

Installation

Nextflow

On Roihu, Nextflow is part of the bio-apps collection and is available on both CPU and GPU nodes. Load the bio-apps module tree and then the Nextflow module:

module load bio-apps/v202603
module load nextflow/26.04.6-standalone

On LUMI, Nextflow is available as a separate module. To access CSC modules on LUMI, first load the CSC module tree into use:

module use /appl/local/csc/modulefiles
module load nextflow

For usage help, use command:

nextflow -h

Installation of tools used in Nextflow

Local installations

By default, Nextflow expects that the analysis tools are available locally. Tools can be activated from existing modules or own custom module installations. See also how to create containers.

On-the-fly Apptainer installations

Containers can be smoothly integrated with Nextflow pipelines. No additional modifications to Nextflow scripts are needed except for enabling the Apptainer engine in the Nextflow configuration file. Nextflow can pull remote container images as Apptainer from container registries on the fly. The remote container images are usually specified in the Nextflow script or configuration file by simply prefixing the image name with shub:// or docker://. It is also possible to specify a different Apptainer image for each process definition in the Nextflow pipeline script.

Most Nextflow pipelines pull the needed container images on the fly. However, when multiple images are needed in a pipeline, it is a good idea to prepare the containers locally before launching the Nextflow pipeline.

Practical considerations:

• Apptainer is installed on login and compute nodes and does not require loading a separate module on CSC supercomputers.
• For binding folders or using other Apptainer settings use nextflow.config file.
• If you are directly pulling multiple Apptainer images on the fly, please use the NVMe disk of a compute node for storing the Apptainer images. For that in your batch job file, utilize local NVMe disk space and then set Apptainer temporary folders as environmental variables. For example, on Roihu, to utilize the node-specific fast local storage in $TMPDIR:

batch_job.sh

export APPTAINER_TMPDIR="$TMPDIR"
export APPTAINER_CACHEDIR="$TMPDIR"

Depending on the partition, the $TMPDIR space on a node will have anywhere from 20 GiB to 600 GiB of available quota in Roihu. The disk space is local to a single node, so move your installations outside of the disk space after the job is finished.

Warning

Although Nextflow supports also Docker containers, these can't be used as such on supercomputers due to the lack of administrative privileges for normal users.

Usage

Nextflow pipelines can be run in different ways in the supercomputer environment:

1. In interactive mode with local executor, with limited resources. Useful mainly for debugging or testing very small workflows.
2. With batch job and local executor. Useful for small and medium size workflows.
3. With batch job and SLURM executor. This can use multiple nodes and different SLURM partitions (CPU and GPU), but may create significant overhead, with many small jobs. Could be used, if each job step for each file takes at least 30 min.
4. With batch job and HyperQueue as a sub-job scheduler. Can use multiple nodes in the same batch job allocation, most complex set up. Well-suited for cases, when the workflow includes a lot of small job steps with many input files (high-throughput computing).

For general introduction to batch jobs, see example job scripts for Roihu.

Note

Whenever you're unsure how to run your workflow efficiently, don't hesitate to contact CSC Service Desk.

Nextflow script

The following minimalist example demonstrates the basic syntax of a Nextflow script.

workflow.nf

#!/usr/bin/env nextflow

greets = Channel.fromList(["Moi", "Ciao", "Hello", "Hola","Bonjour"])

/*
 * Use echo to print 'Hello !' in different languages to a file
 */

process sayHello {

  input:
    val greet

  output:
    path "${greet}.txt"

  script:
    """
    echo ${greet} > ${greet}.txt
    """
}

workflow {

    // Print a greeting
    sayHello(greets)
}

This script defines one process named sayHello. This process takes a set of greetings from different languages and then writes each one to a separate file in a random order.

The resulting terminal output would look similar to the text shown below:

N E X T F L O W  ~  version 23.04.3
Launching `hello-world.nf` [intergalactic_panini] DSL2 - revision: 880a4a2dfd
executor >  local (5)
[a0/bdf83f] process > sayHello (5) [100%] 5 of 5 ✔

Running Nextflow pipelines

Local executor (interactive)

To run Nextflow in an interactive session:

sinteractive --account <project> --cores 2   # replace <project> with your project
module load bio-apps/v202603
module load nextflow/26.04.6-standalone
nextflow run workflow.nf

Note

Please do not launch heavy Nextflow workflows on login nodes.

Local executor (batch job)

To launch a Nextflow job as a regular batch job that executes all job tasks in the same job allocation, create the batch job file:

nextflow_local_batch_job.sh

#!/bin/bash
#SBATCH --time=00:15:00            # Change your runtime settings
#SBATCH --partition=test           # Change partition as needed
#SBATCH --account=<project>        # Add your project name here
#SBATCH --cpus-per-task=<value>    # Change as needed
#SBATCH --mem-per-cpu=1G           # Increase as needed

# Load Nextflow module
module load bio-apps/v202603
module load nextflow/26.04.6-standalone

# Actual Nextflow command here
nextflow run workflow.nf <options>
# nf-core pipeline example:
# nextflow run nf-core/scrnaseq  -profile test,singularity -resume --outdir .

Finally, submit the job to the supercomputer:

sbatch nextflow_local_batch_job.sh

SLURM executor

If the workflow includes only limited number of individual jobs/job steps SLURM executor of Nextflow could be considered.

The first batch job file reserves resources only for Nextflow itself. Nextflow then creates further SLURM jobs for workflow's processes. The SLURM jobs created by Nextflow may be distributed to several nodes of a supercomputer and also to use different partitions for different workflow rules, for example CPU and GPU. SLURM executor should be used only, if the job steps are at least 20-30 minutes long, otherwise it may overload SLURM.

Warning

Please do not use SLURM executor, if your workflow includes a lot of short processes. It would overload SLURM. Use HyperQueue executor instead.

To enable the SLURM executor, set the process.xx settings in nextflow.config file. The settings are similar to batch job files.

nextflow.config

profiles {


 standard {
     process.executor = 'local'
   }

 roihu {
     process.clusterOptions = '--account=<project> --ntasks-per-node=1 --cpus-per-task=4 --ntasks=1 --time=00:15:00'
     process.executor = 'slurm'
     process.queue = 'small'
     process.memory = '10GB'
    }

}

Create the batch job file, note the usage of a profile.

nextflow_slurm_batch_job.sh

#!/bin/bash
#SBATCH --time=00:15:00            # Change your runtime settings
#SBATCH --partition=test           # Change partition as needed
#SBATCH --account=<project>        # Add your project name here
#SBATCH --cpus-per-task=1          # Change as needed
#SBATCH --mem-per-cpu=1G           # Increase as needed

# Load Nextflow module
module load bio-apps/v202603
module load nextflow/26.04.6-standalone

# Actual Nextflow command here
nextflow run workflow.nf -profile roihu

Finally, submit the job to the supercomputer:

sbatch nextflow_slurm_batch_job.sh

This will submit each process of your workflow as a separate batch job to the Roihu supercomputer.

HyperQueue executor

HyperQueue meta-scheduler executer is suitable, if your workflow includes a lot of short processes and you need several nodes for the computation. However, the executor settings can be complex depending on the pipeline.

Here is a batch script for running a nf-core pipeline:

nextflow_hyperqueue_batch_job.sh

#!/bin/bash
#SBATCH --job-name=nextflowjob
#SBATCH --partition=small
#SBATCH --account=<project>
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --mem-per-cpu=2G
#SBATCH --time=01:00:00

# Load the required modules
module load hyperqueue
module load bio-apps/v202603
module load nextflow/26.04.6-standalone

# Create a per job directory
wrkdir=${PWD}/WRKDIR-${SLURM_JOB_ID}

# Set the directory which hyperqueue will use 
export HQ_SERVER_DIR=${wrkdir}/.hq-server
mkdir -p ${HQ_SERVER_DIR}

# Start the server in the background (&) and wait until it has started
hq server start &
until hq job list &>/dev/null ; do sleep 1 ; done

# Start the workers in the background and wait for them to start
srun --overlap --cpu-bind=none --mpi=none hq worker start --cpus=${SLURM_CPUS_PER_TASK} &
hq worker wait "${SLURM_NTASKS}"

# change to the work directory if needed 

cd ${wrkdir}
# Ensure Nextflow uses the right executor and knows how many jobs it can submit
# The `queueSize` can be limited as needed. 

echo "executor {
  queueSize = $(( 40*SLURM_NNODES ))
  name = 'hq'
  cpus = $(( 40*SLURM_NNODES ))
}" >> ${wrkdir}/nextflow.config

# run the Nextflow pipeline here 
nextflow run main.nf <options>

# Wait for all jobs to finish, then shut down the workers and server
hq job wait all
hq worker stop all
hq server stop

Finally, submit the job to the supercomputer:

sbatch nextflow_hyperqueue_batch_job.sh

References

If you use Nextflow in your work, please cite:

Di Tommaso, P., Chatzou, M., Floden, E. et al. Nextflow enables reproducible computational workflows. Nat. Biotechnol. 35, 316–319 (2017). https://doi.org/10.1038/nbt.3820

More information

• Nextflow official documentation
• Master thesis by Antoni Gołoś comparing automated workflow approaches on supercomputers
• Full code Nextflow example from Antoni Gołoś with 3 different executors
• General guidelines for high-throughput computing in CSC's HPC environment
• Official HyperQueue documentation
• CSC's HyperQueue documentation

Do you find this page useful?

Yes

Thank you for your feedback!

No

Thank you! Your feedback is appreciated.

Please let us know how we could improve this page by contacting us or by creating an issue in GitHub. You can also edit this page yourself and contribute your improvements by creating a pull request.