Using Volumes
HPC jobs typically require data. Volumes allow your jobs to access external data, allow multiple jobs within the same workflow to share data, and provide a way to save data after a workflow is completed.
Fuzzball supports the use of ephemeral
volumes that last for the lifetime of a given workflow, or
persistent volumes that retain data
after workflow completion. Volumes can access data from URLs such as http, https, ftp, etc.,
as well as (API-compliant) S3 object stores.
In many cases, it is necessary to provide a password, token, or other secret credential to access volumes such as S3 buckets, private GitHub repos, ftp servers etc. Secrets are saved within the Fuzzball secret store (via the GUI or CLI) and then referenced by scope and name within workflow specification files. Details are provided in the Secrets section of the User manual.
Because of the fact that many storage locations require secrets, it is impossible to provide working examples in all cases. Most of the examples presented in these documents are therefore meant as a guide to help you understand how to use volumes rather than a hands-on tutorial that you can follow.