Executors

In the Nextflow framework architecture, the executor is the component that determines the system where a pipeline process is run and supervises its execution.

The executor provides an abstraction between the pipeline processes and the underlying execution system. This allows you to write the pipeline functional logic independently from the actual processing platform.

In other words you can write your pipeline script once and have it running on your computer, a cluster resource manager or the cloud by simply changing the executor definition in the Nextflow configuration file.

Local

The local executor is used by default. It runs the pipeline processes in the computer where Nextflow is launched. The processes are parallelised by spawning multiple threads and by taking advantage of multi-cores architecture provided by the CPU.

In a common usage scenario, the local executor can be useful to develop and test your pipeline script in your computer, switching to a cluster facility when you need to run it on production data.

SGE

The SGE executor allows you to run your pipeline script by using a Sun Grid Engine cluster or a compatible platform (Open Grid Engine, Univa Grid Engine, etc).

Nextflow manages each process as a separate grid job that is submitted to the cluster by using the qsub command.

Being so, the pipeline must be launched from a node where the qsub command is available, that is, in a common usage scenario, the cluster head node.

To enable the SGE executor simply set to process.executor property to sge value in the nextflow.config file.

The amount of resources requested by each job submission is defined by the following process directives:

LSF

The LSF executor allows you to run your pipeline script by using a Platform LSF cluster.

Nextflow manages each process as a separate job that is submitted to the cluster by using the bsub command.

Being so, the pipeline must be launched from a node where the bsub command is available, that is, in a common usage scenario, the cluster head node.

To enable the LSF executor simply set to process.executor property to lsf value in the nextflow.config file.

The amount of resources requested by each job submission is defined by the following process directives:

Note

LSF supports both per-core and per-job memory limit. Nextflow assumes that LSF works in the per-core memory limits mode, thus it divides the requested memory by the number of requested cpus.

This is not required when LSF is configured to work in per-job memory limit mode. You will need to specified that adding the option perJobMemLimit in Scope executor in the Nextflow configuration file.

See also the Platform LSF documentation.

SLURM

The SLURM executor allows you to run your pipeline script by using the SLURM resource manager.

Nextflow manages each process as a separate job that is submitted to the cluster by using the sbatch command.

Being so, the pipeline must be launched from a node where the sbatch command is available, that is, in a common usage scenario, the cluster head node.

To enable the SLURM executor simply set to process.executor property to slurm value in the nextflow.config file.

The amount of resources requested by each job submission is defined by the following process directives:

Note

SLURM partitions can be considered jobs queues. Nextflow allows to set partitions by using the above queue directive.

PBS/Torque

The PBS executor allows you to run your pipeline script by using a resource manager belonging to the PBS/Torque family of batch schedulers.

Nextflow manages each process as a separate job that is submitted to the cluster by using the qsub command provided by the scheduler.

Being so, the pipeline must be launched from a node where the qsub command is available, that is, in a common usage scenario, the cluster login node.

To enable the PBS executor simply set the property process.executor = 'pbs' in the nextflow.config file.

The amount of resources requested by each job submission is defined by the following process directives:

NQSII

The NQSII executor allows you to run your pipeline script by using the NQSII resource manager.

Nextflow manages each process as a separate job that is submitted to the cluster by using the qsub command provided by the scheduler.

Being so, the pipeline must be launched from a node where the qsub command is available, that is, in a common usage scenario, the cluster login node.

To enable the NQSII executor simply set the property process.executor = 'nqsii' in the nextflow.config file.

The amount of resources requested by each job submission is defined by the following process directives:

HTCondor

The HTCondor executor allows you to run your pipeline script by using the HTCondor resource manager.

Warning

This is an incubating feature. It may change in future Nextflow releases.

Nextflow manages each process as a separate job that is submitted to the cluster by using the condor_submit command.

Being so, the pipeline must be launched from a node where the condor_submit command is available, that is, in a common usage scenario, the cluster head node.

To enable the HTCondor executor simply set to process.executor property to condor value in the nextflow.config file.

The amount of resources requested by each job submission is defined by the following process directives:

DRMAA

The DRMAA executor allows you to execute a Nextflow pipeline by using a grid engine that implements the Java binding for the DRMAA interface api (version 1).

Warning

This is an incubating feature. It may change in future Nextflow releases.

In order to be able to use this executor you will need to access the DRMAA libraries provided by your cluster vendor. Commonly these files are named drmaa.jar and libdrmaa.so and they are located in the cluster installation lib folder. Ask to your IT administrator how to find these files.

To enable the PBS executor you will need to set the property process.executor='drmaa' in the nextflow.config file, moreover you will need to specify the drmaa.jar library path on the Nextflow command line by using the -with-drmaa option. For example:

nextflow run <your pipeline> -with-drmaa /some/path/drmaa.jar

Alternatively, instead of specifying the DRMAA library on the command line, you may want to use the environment variable NXF_DRMAA to define it.

Tip

If you get the following error message:

ERROR: java.lang.UnsatisfiedLinkError: no drmaa in java.library.path

It means that Nextflow is unable to find libdrmaa.so file. The most common solution is to include the path where this file is located in the LD_LIBRARY_PATH environment variable.

The amount of resources requested by each job submission is defined by the following process directives:

Ignite

The Ignite executor allows you to run a pipeline by using the Apache Ignite clustering technology that is embedded with the Nextflow runtime.

To enable this executor set the property process.executor = 'ignite' in the nextflow.config file.

The amount of resources requested by each task submission is defined by the following process directives:

Read the Apache Ignite section in this documentation to learn how to configure Nextflow to deploy and run an Ignite cluster in your infrastructure.

Kubernetes

Nextflow provides an experimental support for Kubernetes clustering technology. It allows you to deploy and transparently run a Nextflow pipeline in a Kubernetes cluster.

Nextflow manages each process as a separate pod that is submitted to the cluster by using the kubectl command. Being so, the pipeline must be launched from a node where the kubectl command is available.

The pipeline processes must specify the Docker image to use by defining the container directive, either in the pipeline script or the nextflow.config file. Moreover the pipeline must be executed in a shared file system folder accessible by all Kubernetes cluster nodes.

To enable this executor set the property process.executor = 'k8s' in the nextflow.config file.

The following directives can be used to define the amount of computing resources needed and the container(s) to use:

AWS Batch

Nextflow supports AWS Batch service which allows submitting jobs in the cloud without having to spin out and manage a cluster of virtual machines. AWS Batch uses Docker containers to run tasks, which makes deploying pipelines much simpler.

The pipeline processes must specify the Docker image to use by defining the container directive, either in the pipeline script or the nextflow.config file.

To enable this executor set the property process.executor = 'awsbatch' in the nextflow.config file.

The pipeline can be launched either in a local computer or a EC2 instance. The latter is suggested for heavy or long running workloads. Moreover a S3 bucket must be used as pipeline work directory.

See the AWS Batch page for further configuration details.