LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 73 lessons

Lesson 9 of 10

Module 3 · The command line

Workflows, the cloud and reproducible pipelines

Workflows, the cloud and reproducible pipelines

Workflows, the cloud and reproducible pipelines

Turn a chain of scripts into a pipeline that reruns only what changed, runs anywhere and can be repeated exactly. Meet workflow managers, containers, git, the cloud, FAIR data and community pipelines.

⏱ 15 min readModule 3 · The command lineFree
Lesson at a glance7 steps · about 15 min
  1. Why scripts become pipelines
  2. Workflow managers: Snakemake and Nextflow
  3. Containers: Docker and Apptainer
  4. Environment files and version control
  5. The cloud
  6. FAIR data
  7. Community pipelines: nf-core

Why scripts become pipelines

The read-counting script from Variables, loops and scripts does one job on a few files. A real analysis chains many jobs: quality control, trimming, alignment, counting, then a summary report, for every sample. You will meet each of those steps in Module 5. Run by hand or from one long script, a chain like that soon causes problems:

  • If one sample fails half-way, a simple script either stops everything or carries on with a missing file.
  • If you add a sample or change one setting, the script reruns every step for every sample, even the ones that were already finished.
  • The order of steps lives only in your head, or in the order of lines in the script.
  • Moving to a cluster (Bigger jobs, servers and clusters) or another computer means rewriting the commands.

A workflow, or pipeline, is the whole chain written down as separate steps, each with stated inputs and outputs. A workflow manager is a program that reads that description, works out the order, runs each step, and keeps track of what is already done.

In Bigger jobs, servers and clusters you wrapped one script in a Slurm job by hand. A workflow manager can do that wrapping for you: it turns each step for each sample into its own job, submits the ones whose inputs are ready, and waits for them before starting the steps that need their outputs. The description of the analysis stays the same whether it runs on a laptop or a cluster; only a setting changes.

Workflow managers: Snakemake and Nextflow

Two workflow managers dominate bioinformatics. Both are free, both run the same workflow on a laptop, a Slurm cluster or the cloud, and both can run each step in its own conda environment or container.

Snakemake
  • Steps are rules that say how to make output files from input files
  • Written in a language built on Python
  • Works backwards from the files you ask for
Nextflow
  • Steps are processes joined by streams of data called channels
  • Written in its own language, built on Groovy
  • Widely used for large shared pipelines, such as nf-core

The two take different views of the same problem. Snakemake thinks in files: you ask for a result file, and it finds the rules that can make it, then the rules that make their inputs, back to the raw data. Nextflow thinks in data flowing forward: each sample's files pass from process to process like items on a conveyor belt. Both record which steps succeeded, and both can pick up where they stopped after a failure. You do not need to learn either fully now. The aim is to recognise one when you see it, and to know what it does for you.

Below is a tiny Snakemake workflow that runs seqkit stats (Installing software with conda) on two samples and writes one table per sample. It is shown so you can see the shape of a rule; you do not need to install Snakemake for this course.

Snakefile · example, not run in this course
SAMPLES = ["ctrl_1", "treat_1"]

rule all:
    input:
        expand("results/{sample}_stats.tsv", sample=SAMPLES)

rule stats:
    input:
        "data/{sample}_R1.fastq.gz"
    output:
        "results/{sample}_stats.tsv"
    shell:
        "seqkit stats -T {input} > {output}"

Reading the file

  • The file is called Snakefile, with no extension; Snakemake looks for that name in the current folder.
  • SAMPLES is an ordinary Python list of sample names.
  • rule all names the final files you want. expand() fills {sample} with each name in the list, so it asks for results/ctrl_1_stats.tsv and results/treat_1_stats.tsv. By convention the first rule is the target Snakemake aims for.
  • rule stats says how to make one stats file. {sample} is a wildcard, a placeholder Snakemake fills in by matching the file name it needs against the output pattern.
  • shell is the command to run. {input} and {output} are replaced by the real file names, and -T asks seqkit for a tab-separated table.
Terminal
snakemake -n           # dry run: show the plan, run nothing
snakemake --cores 1    # run it, one job at a time
Example output, shortened (snakemake -n) Building DAG of jobs... Job stats: job count ----- ------- all 1 stats 2 total 3

What this does

  • snakemake -n is a dry run: Snakemake works out what it would do and prints the plan, without running any command. Always do this first.
  • snakemake --cores 1 runs the plan using one processor, with the ngs environment from Installing software with conda active so that seqkit is found. With --cores 4, up to four independent jobs run at once.

Reading the output

  • A DAG (directed acyclic graph) is the map of which job feeds which. "Directed" means each link has a direction, from input to output; "acyclic" means no step ever depends, through a loop, on its own result.
  • The table counts the jobs: two stats jobs, one per sample, and one all job that simply checks the final files exist.
Workflows, the cloud and reproducible pipelines · LSMA free course Two input files, ctrl_1 and treat_1, each feed one stats job, which writes one stats table. Both tables feed the rule all. A second panel shows that after a new sample is added, only its stats job runs, while the finished jobs are skipped. 1 · INPUT FILES2 · RULE STATS3 · OUTPUTS4 · TARGET ctrl_1_R1.fastq.gztreat_1_R1.fastq.gz ctrl_1_stats.tsvtreat_1_stats.tsv stats · ctrl_1stats · treat_1all ADD A THIRD SAMPLE AND RUN AGAIN ctrl_1, treat_1: outputs exist, skipped new sample: only its stats job runs
Snakemake builds this graph from the file names alone, before running anything. Jobs on separate rows do not depend on each other, so they can run at the same time. Schematic.

How to read this figure

Follow each row from left to right: an input file goes into one stats job, which writes one table. The two tables meet only at all, the target. The shaded boxes are the jobs that actually run commands. The lower panel shows what happens on a second run: a job whose output already exists, and is newer than its input, is skipped.

Worked example · rerunning only what changed

pipeline: 4 steps per sample · 48 samples = 192 jobs, all finished
one sample's raw file is replaced → its 4 jobs are out of date
workflow manager reruns 4 jobs · a plain script reruns 192

Step by step. Each of the 48 samples passes through 4 steps, so there are 48 × 4 = 192 jobs. Now the sequencing centre sends a corrected file for one sample. The first step for that sample has an input newer than its output, so it is out of date. Its output will change, so the three steps downstream of it are out of date too: 4 jobs in all. The other 47 samples' 188 jobs are untouched. A script that loops over every sample has no record of what is finished, so it runs all 192 again.

What the answer means. If each job takes 20 minutes, that is 80 minutes instead of 64 hours of computing. The saving matters even more in the cloud, where you pay for every hour. Just as important, the workflow manager decides what is out of date from the files themselves, so you cannot forget to rerun a step that depended on the changed file.

Nextflow pipelines are run in a similar way. The shared pipelines described at the end of this lesson are written in Nextflow, and start with a single command.

Snakemake is cited as Köster and Rahmann (2012), Bioinformatics 28:2520–2522, with the update Mölder et al. (2021), 'Sustainable data analysis with Snakemake', F1000Research 10:33. Nextflow is Di Tommaso et al. (2017), 'Nextflow enables reproducible computational workflows', Nature Biotechnology 35:316–319, and nf-core is Ewels et al. (2020), Nature Biotechnology 38:276–278. nf-core asks users to cite the pipeline release they ran, which has its own DOI, as well as the tools inside it. In methods, give the workflow manager's version, the pipeline release and the profile used. The FAIR principles come from Wilkinson et al. (2016), Scientific Data 3:160018.

Containers: Docker and Apptainer

A conda environment (Installing software with conda) records which tools you need. A container goes one step further: it packs the tools, every dependency and a minimal operating system into one file, called an image. Run the same image on a laptop, a cluster or the cloud and the software inside is identical, down to the last library.

Docker is the best-known container system, and most published images are Docker images. Docker normally needs administrator rights, which shared clusters do not give users, so clusters use Apptainer (formerly called Singularity) instead. Apptainer can run Docker images, so one image serves both. The BioContainers project provides a ready-made image for every Bioconda package, which means that a tool you can install with conda can usually be run as a container as well. Workflow managers can fetch the right image for each step themselves; you seldom need to type container commands directly.

Conda environments and containers are not rivals; many projects use both. An environment is quick to create and easy to change while you are still exploring. A container is heavier to build but fixes the software completely, which makes it the better choice once an analysis is settled and has to run unchanged on other computers. A common pattern is to develop with conda, then run the finished workflow with one container per step.

Go deeper: pin a container by version, not by latest

Each image is named with a tag after a colon, usually a version number. One tag is special only by habit: latest, which means whatever the owner uploaded most recently. A workflow that asks for latest can quietly get different software next month. Name an exact version tag instead; BioContainers tags include the tool's version and its build.

For the strongest guarantee, use the image's digest: a long code beginning sha256: that is calculated from the image's contents, so it can never point to anything else. One practical point for clusters: Apptainer stores an image as a single file ending in .sif. Keeping that file with the project means the exact software can be run again even if the online copy is removed.

Environment files and version control

An environment.yml file (Installing software with conda) lists the tools and pinned versions a project needs, so anyone can rebuild the same environment with conda env create -f environment.yml. Keep it in the project folder next to the Snakefile, and it travels with the workflow. Snakemake and Nextflow can also build a separate environment for each step from a file like this.

Scripts, Snakefiles and environment files change as a project grows, and you will want to know what changed, when and why. Version control keeps that history. git is the standard tool. A folder whose history git tracks is a repository, and each saved snapshot of your files, with a short message saying what changed, is a commit. These three commands are enough to start. In this example the project folder holds a short version of the script from Variables, loops and scripts and an environment file, so your file and line counts will differ.

Terminal
cd rnaseq-project
git init -q -b main                                                # 1 · start tracking this folder
git add scripts/count_reads.sh environment.yml                     # 2 · choose files for the next snapshot
git commit -m "Add read-counting script and environment file"      # 3 · save the snapshot
git log --oneline                                                  # list the snapshots so far
[main (root-commit) 3c84319] Add read-counting script and environment file 2 files changed, 9 insertions(+) create mode 100644 environment.yml create mode 100644 scripts/count_reads.sh 3c84319 Add read-counting script and environment file

What this does

  • git init makes the folder a repository by creating a hidden .git folder that will hold the history. -q (quiet) skips the message, and -b main names the main line of history main. You run it once per project.
  • git add marks files to go into the next commit. Nothing is saved yet; you are choosing what to include.
  • git commit -m saves the snapshot with the message in quotes. Write messages that say what changed, so that you can find it again months later.
  • git log --oneline lists every commit, newest first, one line each.

Reading the output

  • root-commit means this is the first commit in the repository. 3c84319 is the start of the commit's unique identifier, a code worked out from its contents and time; yours will be different.
  • 2 files changed, 9 insertions(+): two new files, holding 9 lines between them.
  • create mode 100644 records each new file and its permissions; 100644 means an ordinary file.

Before your first commit on a new computer, tell git who you are with git config --global user.name "Your Name" and git config --global user.email "you@example.org". Commit small and often. Keep data files out of git: it is built for text you write, not for gigabytes of reads. Services such as GitHub and GitLab store a copy of a repository online, for backup and for sharing.

The cloud

The cloud means renting computers and storage from a provider over the internet, such as Amazon Web Services, Google Cloud or Microsoft Azure, and paying only for what you use. Need 200 processors for an afternoon? You rent them, run your workflow and hand them back. Many large public datasets are also stored in the cloud already, so the analysis can go to the data instead of the data coming to you.

The cloud and a university cluster solve the same problem in different ways. A cluster is already paid for and shared, so you may wait in a queue but do not see a bill; the cloud starts machines almost at once, in any number, but charges for every hour. For a learner or a small project, an institutional cluster is usually the simpler start. The cloud earns its place for very large or urgent jobs, or when the data already live there.

Two cautions before you start.

  • Cost. The meter runs while a machine is switched on, whether it is working or not, and storage is charged every month. Moving data out of the cloud, called egress, is usually charged too, and for sequencing data it can be large. Forgotten machines are the classic surprise bill. Set a budget with alerts, shut machines down when a job ends, and delete intermediate files you do not need.
  • Data protection. Human genomic and health data are personal data, and in the UK and EU health and genetic data are a special category with extra legal protection. Never upload patient data to a personal cloud account. Use only systems your institution has approved, with the right agreements and data location, and follow the terms of any controlled-access dataset. Much sensitive data can now be analysed only inside a trusted research environment, a secure system where approved researchers work on the data without downloading it.

FAIR data

A pipeline is only reusable if its data can be found and understood. The FAIR principles, now widely required by funders and journals, ask that data be:

  • Findable: given a permanent identifier, such as an accession number, and described with searchable metadata, the data about the data (organism, tissue, method, date).
  • Accessible: retrievable through a standard, open method. This does not have to mean open to everyone; patient data can be FAIR behind a controlled-access application.
  • Interoperable: in standard formats and vocabularies (the file formats of Module 2), so other tools can read it.
  • Reusable: with a clear licence, a record of how it was made, and enough detail to be used again.

A workflow, an environment file and a git history are the "how it was made" for your results. Depositing raw reads in a public archive, as Module 2 described, covers the data side.

FAIR is easiest when it starts on day one. Keep a sample sheet that links each file to its sample, condition and accession, use the same sample names in file names, scripts and tables, and write down where each raw file came from. Reconstructing that information at the end of a project, when you come to deposit the data or write the methods, takes far longer and is often incomplete.

Community pipelines: nf-core

For common analyses you may not need to write a pipeline at all. nf-core is a community collection of more than a hundred Nextflow pipelines, for RNA-seq, variant calling, single-cell data and much more. Each one is reviewed by the community, has versioned releases and documentation, runs every step in a container and comes with a small test dataset.

Terminal · example, not run in this course
nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir results

What this does

  • nextflow run nf-core/rnaseq downloads the RNA-seq pipeline from nf-core and runs it.
  • -r 3.14.0 fixes the release, so that the run can be repeated exactly. It is shown as an example; check the pipeline's page for current releases, and write the one you used in your methods.
  • -profile test,docker uses the built-in test data and runs each step in a Docker container. On a cluster you would use apptainer and your institution's profile instead.
  • --outdir results is the folder for all outputs, including a report of every tool version used.

A community pipeline is not a black box you can trust blindly. Read its documentation, run the test profile first, and check its quality-control report, just as you would check your own work.

Key point

A reproducible analysis pins four things: the steps (a workflow), the software (an environment file or container), the code's history (git) and the data (FAIR, with identifiers). Pin all four and someone else, or you in a year, can get the same result.

Each one closes a different gap. Without the workflow, the order of steps is lost; without pinned software, a new tool version can shift the numbers; without git, you cannot tell which version of a script made a figure; without identifiers, no one can find the input. You do not need all four for a quick exploration, but you do for anything you publish or hand on.

Further reading · Module 3Snakemake documentation, including its tutorial. · nf-core documentation: how to run community pipelines. · git documentation. · The FAIR principles (GO FAIR). · All reading for this course

Check your understanding

1. A workflow of 30 samples has finished. You change a setting that affects only the final summary step. What will a workflow manager rerun?

2. Your cluster does not give users administrator rights, but the pipeline you want ships Docker images. What is the usual solution?

3. A patient dataset can be downloaded only after an approved access application. Can it still be FAIR?

Up next

Project

A FASTQ report script

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.