Shifter Containers¶
Shifter is container engine alternative to Docker. Shifter works by converting Docker images to a common format that can then be distributed and launched on HPC systems. The user interface to Shifter enables a user to select an image from the dockerhub registry and then submit jobs which run entirely within the container.
Nextflow provides built-in support for Shifter. This allows you to control the execution environment of the processes in your pipeline by running them in isolated containers along-side their dependencies.
Moreover the support provided by Nextflow for different container technologies, allows the same pipeline to be transparently executed both with Docker, Singularity or Shifter containers, depending on the available engine in target execution platforms.
Prerequisites¶
You need Shifter and Shifter image gateway installed in your execution environment, i.e: your personal computed or the
entry node of a distributed cluster. In the case of the distributed cluster case, you should have Shifter installed on
all of the compute nodes and the shifterimg
command should also be available and Shifter properly setup to access the
Image gateway, for more information see the
official documentation.
Note
This feature requires Shifter version 18.03 (or later) and Nextflow 19.10.0 (or later).
Images¶
Shifter converts a docker image to squashfs layers which are distributed and launched in the cluster. For more info on how to Build Shifter images see the official documentation.
How it works¶
The integration for Shifter, at this time, requires you to set up the following parameters in your config file:
process.container = "dockerhub_user/image_name:image_tag"
shifter.enabled = true
and it will always try to search the Docker Hub registry for the images.
Note
if you do not specify an image tag it will fetch the ‘latest’ tag by default.
Multiple Containers¶
It is possible to specify a different Shifter image for each process definition in your pipeline script. For example,
let’s suppose you have two processes named foo
and bar
. You can specify two different Shifter images
specifying them in the nextflow.config
file as shown below:
process {
withName:foo {
container = 'image_name_1'
}
withName:bar {
container = 'image_name_2'
}
}
shifter {
enabled = true
}
Read the Process scope section to learn more about processes configuration.