Configuration¶

Config syntax¶

A Nextflow configuration file is a simple text file containing a set of properties defined using the syntax:

name = value

Please note, string values need to be wrapped in quotation characters while numbers and boolean values (true, false) do not. Also note that values are typed, meaning for example that, 1 is different from '1', since the first is interpreted as the number one, while the latter is interpreted as a string value.

Config Variables¶

Configuration properties can be used as variables in the configuration file itself, by using the usual $propertyName or ${expression} syntax.

For example:

propertyOne = 'world'
anotherProp = "Hello $propertyOne"
customPath = "$PATH:/my/app/folder"

Please note, the usual rules for String interpolation are applied, thus a string containing a variable reference must be wrapped in double-quote chars instead of single-quote chars.

The same mechanism allows you to access environment variables defined in the hosting system. Any variable whose name is not defined in the Nextflow configuration file(s) is supposed to be a reference to an environment variable with that name. So, in the above example the property customPath is defined as the current system PATH to which the string /my/app/folder is appended.

Config comments¶

Configuration files use the same conventions for comments used by the Groovy or Java programming languages. Thus, use // to comment a single line or /* .. */ to comment a block on multiple lines.

Config include¶

A configuration file can include one or more configuration files using the keyword includeConfig. For example:

process.executor = 'sge'
process.queue = 'long'
process.memory = '10G'

includeConfig 'path/foo.config'

When a relative path is used, it is resolved against the actual location of the including file.

Scope env¶

The env scope allows the definition one or more variable that will be exported in the environment where the workflow tasks will be executed.

Simply prefix your variable names with the env scope or surround them by curly brackets, as shown below:

env.ALPHA = 'some value'
env.BETA = "$HOME/some/path"

env {
     DELTA = 'one more'
     GAMMA = "/my/path:$PATH"
}

Scope params¶

The params scope allows you to define parameters that will be accessible in the pipeline script. Simply prefix the parameter names with the params scope or surround them by curly brackets, as shown below:

params.custom_param = 123
params.another_param = 'string value .. '

params {

   alpha_1 = true
   beta_2 = 'another string ..'

}

Scope process¶

The process configuration scope allows you to provide the default configuration for the processes in your pipeline.

You can specify here any property described in the process directive and the executor sections. For examples:

process {
  executor='sge'
  queue='long'
  clusterOptions = '-pe smp 10 -l virtual_free=64G,h_rt=30:00:00'
}

By using this configuration all processes in your pipeline will be executed through the SGE cluster, with the specified settings.

process {
    withLabel: big_mem {
        cpus = 16
        memory = 64.GB
        queue = 'long'
    }
}

process {
    withName: hello {
        cpus = 4
        memory = 8.GB
        queue = 'short'
    }
}

process {
    withLabel: 'foo|bar' {
        cpus = 2
        memory = 4.GB
    }
}

process {
    withLabel: 'foo' { cpus = 2 }
    withLabel: '!foo' { cpus = 4 }
    withName: '!align.*' { queue = 'long' }
}

process {
    cpus = 4
    withLabel: foo { cpus = 8 }
    withName: bar { cpus = 32 }
}

Scope executor¶

The executor configuration scope allows you to set the optional executor settings, listed in the following table.

The executor settings can be defined as shown below:

executor {
    name = 'sge'
    queueSize = 200
    pollInterval = '30 sec'
}

When using two (or more) different executors in your pipeline, you can specify their settings separately by prefixing the executor name with the symbol $ and using it as special scope identifier. For example:

executor {
  $sge {
      queueSize = 100
      pollInterval = '30sec'
  }

  $local {
      cpus = 8
      memory = '32 GB'
  }
}

The above configuration example can be rewritten using the dot notation as shown below:

executor.$sge.queueSize = 100
executor.$sge.pollInterval = '30sec'
executor.$local.cpus = 8
executor.$local.memory = '32 GB'

Scope docker¶

The docker configuration scope controls how Docker containers are executed by Nextflow.

The following settings are available:

The above options can be used by prefixing them with the docker scope or surrounding them by curly brackets, as shown below:

process.container = 'nextflow/examples'

docker {
    enabled = true
    temp = 'auto'
}

Read Docker containers page to lean more how use Docker containers with Nextflow.

Scope singularity¶

The singularity configuration scope controls how Singularity containers are executed by Nextflow.

The following settings are available:

Read Singularity containers page to lean more how use Singularity containers with Nextflow.

Scope manifest¶

The manifest configuration scope allows you to define some meta-data information needed when publishing your pipeline project on GitHub, BitBucket or GitLab, or when running your pipeline.

The following settings are available:

The above options can be used by prefixing them with the manifest scope or surrounding them by curly brackets. For example:

manifest {
    homePage = 'http://foo.com'
    description = 'Pipeline does this and that'
    mainScript = 'foo.nf'
    version = '1.0.0'
}

To learn how to publish your pipeline on GitHub, BitBucket or GitLab code repositories read Pipeline sharing documentation page.

nextflowVersion = '1.2.3'        // exact match
nextflowVersion = '1.2+'         // 1.2 or later (excluding 2 and later)
nextflowVersion = '>=1.2'        // 1.2 or later
nextflowVersion = '>=1.2, <=1.5' // any version in the 1.2 .. 1.5 range
nextflowVersion = '!>=1.2'       // with ! prefix, stop execution if current version
                                    does not match required version.

Scope trace¶

The trace scope allows you to control the layout of the execution trace file generated by Nextflow.

The following settings are available:

The above options can be used by prefixing them with the trace scope or surrounding them by curly brackets. For example:

trace {
    enabled = true
    file = 'pipeline_trace.txt'
    fields = 'task_id,name,status,exit,realtime,%cpu,rss'
}

To learn more about the execution report that can be generated by Nextflow read Trace report documentation page.

Scope aws¶

The aws scope allows you to configure the access to Amazon S3 storage. Use the attributes accessKey and secretKey to specify your bucket credentials. For example:

aws {
    accessKey = '<YOUR S3 ACCESS KEY>'
    secretKey = '<YOUR S3 SECRET KEY>'
    region = '<REGION IDENTIFIER>'
}

Click the following link to lean more about AWS Security Credentials.

Advanced client configuration options can be set by using the client attribute. The following properties can be used:

For example:

aws {
    client {
        maxConnections = 20
        connectionTimeout = 10000
        uploadStorageClass = 'INTELLIGENT_TIERING'
        storageEncryption = 'AES256'
    }
}

Scope cloud¶

The cloud scope allows you to define the settings of the computing cluster that can be deployed in the cloud by Nextflow.

The following settings are available:

The autoscale configuration group provides the following settings:

Scope conda¶

The conda scope allows for the definition of the configuration settings that control the creation of a Conda environment by the Conda package manager.

The following settings are available:

Scope k8s¶

The k8s scope allows the definition of the configuration settings that control the deployment and execution of workflow applications in a Kubernetes cluster.

The following settings are available:

See the Kubernetes documentation for more details.

Scope timeline¶

The timeline scope allows you to enable/disable the processes execution timeline report generated by Nextflow.

The following settings are available:

Scope mail¶

The mail scope allows you to define the mail server configuration settings needed to send email messages.

For example, the following snippet shows how to configure Nextflow to send emails through the AWS Simple Email Service:

mail {
    smtp.host = 'email-smtp.us-east-1.amazonaws.com'
    smtp.port = 587
    smtp.user = '<Your AWS SES access key>'
    smtp.password = '<Your AWS SES secret key>'
    smtp.auth = true
    smtp.starttls.enable = true
    smtp.starttls.required = true
}

Scope notification¶

The notification scope allows you to define the automatic sending of a notification email message when the workflow execution terminates.

The notification message is sent my using the STMP server defined in the configuration mail scope.

If no mail configuration is provided, it tries to send the notification message by using the external mail command eventually provided by the underlying system (eg. sendmail or mail).

Scope report¶

The report scope allows you to define configuration setting of the workflow Execution report.

Scope weblog¶

The weblog scope allows to send detailed trace scope information as HTTP POST request to a webserver, shipped as a JSON object.

Detailed information about the JSON fields can be found in the weblog description.