Amazon S3 storage
Nextflow includes support for Amazon S3 storage. Files stored in an S3 bucket can be accessed transparently in your pipeline script like any other file in the local file system.
In order to access an S3 file, you only need to prefix the file path with the
s3 schema and the
bucket name where it is stored.
For example, if you need to access the file
/data/sequences.fa stored in a bucket named
my-bucket, that file can be accessed using the following fully qualified path:
The usual file operations can be applied to a path handle with the above notation. For example, the content of an S3 file can be printed as follows:
See the Files and I/O section to learn more about available file operations.
Amazon access credentials can be provided in two ways:
Using AWS access and secret keys in your pipeline configuration.
Using IAM roles to grant access to S3 storage on Amazon EC2 instances.
AWS access and secret keys
The AWS access and secret keys can be specified by using the
aws section in the
nextflow.config configuration file as shown below:
accessKey = '<Your AWS access key>'
secretKey = '<Your AWS secret key>'
region = '<AWS region identifier>'
If the access credentials are not found in the above file, Nextflow looks for AWS credentials in the following order:
nextflow.configfile in the pipeline execution directory
The environment variables
The environment variables
defaultprofile in the AWS credentials file located at
defaultprofile in the AWS client configuration file located at
The temporary AWS credentials provided by an IAM instance role. See IAM Roles documentation for details.
More information regarding AWS Security Credentials are available in the AWS documentation.
IAM roles with Amazon EC2 instances
When running your pipeline in an EC2 instance, IAM roles can be used to grant access to AWS resources.
In this scenario, you only need to launch the EC2 instance with an IAM role which includes the
AmazonS3FullAccess policy. Nextflow will detect and automatically acquire the permission to access S3 storage, without any further configuration.
Learn more about Using IAM Roles to Delegate Permissions to Applications that Run on Amazon EC2 in the Amazon documentation.
To use an AWS China region, make sure to specify the corresponding AWS API S3 endpoint in the Nextflow configuration file as shown below:
endpoint = "https://s3.cn-north-1.amazonaws.com.cn"
Read more about AWS API endpoints in the AWS documentation
accessKey = '<Your access key>'
secretKey = '<Your secret key>'
endpoint = '<Your storage endpoint URL>'
s3PathStyleAccess = true
Read AWS configuration section to learn more about advanced S3 client configuration options.