Lesson 9 of 10
Module 3 · The command line
Turn a chain of scripts into a pipeline that reruns only what changed, runs anywhere and can be repeated exactly. Meet workflow managers, containers, git, the cloud, FAIR data and community pipelines.
Why scripts become pipelines
The read-counting script from Variables, loops and scripts does one job on a few files. A real analysis chains many jobs: quality control, trimming, alignment, counting, then a summary report, for every sample. You will meet each of those steps in Module 5. Run by hand or from one long script, a chain like that soon causes problems:
- If one sample fails half-way, a simple script either stops everything or carries on with a missing file.
- If you add a sample or change one setting, the script reruns every step for every sample, even the ones that were already finished.
- The order of steps lives only in your head, or in the order of lines in the script.
- Moving to a cluster (Bigger jobs, servers and clusters) or another computer means rewriting the commands.
A workflow, or pipeline, is the whole chain written down as separate steps, each with stated inputs and outputs. A workflow manager is a program that reads that description, works out the order, runs each step, and keeps track of what is already done.
In Bigger jobs, servers and clusters you wrapped one script in a Slurm job by hand. A workflow manager can do that wrapping for you: it turns each step for each sample into its own job, submits the ones whose inputs are ready, and waits for them before starting the steps that need their outputs. The description of the analysis stays the same whether it runs on a laptop or a cluster; only a setting changes.
Workflow managers: Snakemake and Nextflow
Two workflow managers dominate bioinformatics. Both are free, both run the same workflow on a laptop, a Slurm cluster or the cloud, and both can run each step in its own conda environment or container.
- Steps are rules that say how to make output files from input files
- Written in a language built on Python
- Works backwards from the files you ask for
- Steps are processes joined by streams of data called channels
- Written in its own language, built on Groovy
- Widely used for large shared pipelines, such as nf-core
The two take different views of the same problem. Snakemake thinks in files: you ask for a result file, and it finds the rules that can make it, then the rules that make their inputs, back to the raw data. Nextflow thinks in data flowing forward: each sample's files pass from process to process like items on a conveyor belt. Both record which steps succeeded, and both can pick up where they stopped after a failure. You do not need to learn either fully now. The aim is to recognise one when you see it, and to know what it does for you.
Below is a tiny Snakemake workflow that runs seqkit stats (Installing software with conda) on two samples and writes one table per sample. It is shown so you can see the shape of a rule; you do not need to install Snakemake for this course.
SAMPLES = ["ctrl_1", "treat_1"]
rule all:
input:
expand("results/{sample}_stats.tsv", sample=SAMPLES)
rule stats:
input:
"data/{sample}_R1.fastq.gz"
output:
"results/{sample}_stats.tsv"
shell:
"seqkit stats -T {input} > {output}"Reading the file
- The file is called Snakefile, with no extension; Snakemake looks for that name in the current folder.
SAMPLESis an ordinary Python list of sample names.rule allnames the final files you want.expand()fills{sample}with each name in the list, so it asks for results/ctrl_1_stats.tsv and results/treat_1_stats.tsv. By convention the first rule is the target Snakemake aims for.rule statssays how to make one stats file.{sample}is a wildcard, a placeholder Snakemake fills in by matching the file name it needs against the output pattern.shellis the command to run.{input}and{output}are replaced by the real file names, and-Tasks seqkit for a tab-separated table.
snakemake -n # dry run: show the plan, run nothing snakemake --cores 1 # run it, one job at a time
What this does
snakemake -nis a dry run: Snakemake works out what it would do and prints the plan, without running any command. Always do this first.snakemake --cores 1runs the plan using one processor, with the ngs environment from Installing software with conda active so that seqkit is found. With--cores 4, up to four independent jobs run at once.
Reading the output
- A DAG (directed acyclic graph) is the map of which job feeds which. "Directed" means each link has a direction, from input to output; "acyclic" means no step ever depends, through a loop, on its own result.
- The table counts the jobs: two
statsjobs, one per sample, and onealljob that simply checks the final files exist.
How to read this figure
Follow each row from left to right: an input file goes into one stats job, which writes one table. The two tables meet only at all, the target. The shaded boxes are the jobs that actually run commands. The lower panel shows what happens on a second run: a job whose output already exists, and is newer than its input, is skipped.
Worked example · rerunning only what changed
one sample's raw file is replaced → its 4 jobs are out of date
workflow manager reruns 4 jobs · a plain script reruns 192
Step by step. Each of the 48 samples passes through 4 steps, so there are 48 × 4 = 192 jobs. Now the sequencing centre sends a corrected file for one sample. The first step for that sample has an input newer than its output, so it is out of date. Its output will change, so the three steps downstream of it are out of date too: 4 jobs in all. The other 47 samples' 188 jobs are untouched. A script that loops over every sample has no record of what is finished, so it runs all 192 again.
What the answer means. If each job takes 20 minutes, that is 80 minutes instead of 64 hours of computing. The saving matters even more in the cloud, where you pay for every hour. Just as important, the workflow manager decides what is out of date from the files themselves, so you cannot forget to rerun a step that depended on the changed file.
Nextflow pipelines are run in a similar way. The shared pipelines described at the end of this lesson are written in Nextflow, and start with a single command.
Snakemake is cited as Köster and Rahmann (2012), Bioinformatics 28:2520–2522, with the update Mölder et al. (2021), 'Sustainable data analysis with Snakemake', F1000Research 10:33. Nextflow is Di Tommaso et al. (2017), 'Nextflow enables reproducible computational workflows', Nature Biotechnology 35:316–319, and nf-core is Ewels et al. (2020), Nature Biotechnology 38:276–278. nf-core asks users to cite the pipeline release they ran, which has its own DOI, as well as the tools inside it. In methods, give the workflow manager's version, the pipeline release and the profile used. The FAIR principles come from Wilkinson et al. (2016), Scientific Data 3:160018.
Containers: Docker and Apptainer
A conda environment (Installing software with conda) records which tools you need. A container goes one step further: it packs the tools, every dependency and a minimal operating system into one file, called an image. Run the same image on a laptop, a cluster or the cloud and the software inside is identical, down to the last library.
Docker is the best-known container system, and most published images are Docker images. Docker normally needs administrator rights, which shared clusters do not give users, so clusters use Apptainer (formerly called Singularity) instead. Apptainer can run Docker images, so one image serves both. The BioContainers project provides a ready-made image for every Bioconda package, which means that a tool you can install with conda can usually be run as a container as well. Workflow managers can fetch the right image for each step themselves; you seldom need to type container commands directly.
Conda environments and containers are not rivals; many projects use both. An environment is quick to create and easy to change while you are still exploring. A container is heavier to build but fixes the software completely, which makes it the better choice once an analysis is settled and has to run unchanged on other computers. A common pattern is to develop with conda, then run the finished workflow with one container per step.
Go deeper: pin a container by version, not by latest
Each image is named with a tag after a colon, usually a version number. One tag is special only by habit: latest, which means whatever the owner uploaded most recently. A workflow that asks for latest can quietly get different software next month. Name an exact version tag instead; BioContainers tags include the tool's version and its build.
For the strongest guarantee, use the image's digest: a long code beginning sha256: that is calculated from the image's contents, so it can never point to anything else. One practical point for clusters: Apptainer stores an image as a single file ending in .sif. Keeping that file with the project means the exact software can be run again even if the online copy is removed.
Environment files and version control
An environment.yml file (Installing software with conda) lists the tools and pinned versions a project needs, so anyone can rebuild the same environment with conda env create -f environment.yml. Keep it in the project folder next to the Snakefile, and it travels with the workflow. Snakemake and Nextflow can also build a separate environment for each step from a file like this.
Scripts, Snakefiles and environment files change as a project grows, and you will want to know what changed, when and why. Version control keeps that history. git is the standard tool. A folder whose history git tracks is a repository, and each saved snapshot of your files, with a short message saying what changed, is a commit. These three commands are enough to start. In this example the project folder holds a short version of the script from Variables, loops and scripts and an environment file, so your file and line counts will differ.
cd rnaseq-project git init -q -b main # 1 · start tracking this folder git add scripts/count_reads.sh environment.yml # 2 · choose files for the next snapshot git commit -m "Add read-counting script and environment file" # 3 · save the snapshot git log --oneline # list the snapshots so far
What this does
git initmakes the folder a repository by creating a hidden .git folder that will hold the history.-q(quiet) skips the message, and-b mainnames the main line of history main. You run it once per project.git addmarks files to go into the next commit. Nothing is saved yet; you are choosing what to include.git commit -msaves the snapshot with the message in quotes. Write messages that say what changed, so that you can find it again months later.git log --onelinelists every commit, newest first, one line each.
Reading the output
- root-commit means this is the first commit in the repository. 3c84319 is the start of the commit's unique identifier, a code worked out from its contents and time; yours will be different.
- 2 files changed, 9 insertions(+): two new files, holding 9 lines between them.
- create mode 100644 records each new file and its permissions; 100644 means an ordinary file.
Before your first commit on a new computer, tell git who you are with git config --global user.name "Your Name" and git config --global user.email "you@example.org". Commit small and often. Keep data files out of git: it is built for text you write, not for gigabytes of reads. Services such as GitHub and GitLab store a copy of a repository online, for backup and for sharing.
The cloud
The cloud means renting computers and storage from a provider over the internet, such as Amazon Web Services, Google Cloud or Microsoft Azure, and paying only for what you use. Need 200 processors for an afternoon? You rent them, run your workflow and hand them back. Many large public datasets are also stored in the cloud already, so the analysis can go to the data instead of the data coming to you.
The cloud and a university cluster solve the same problem in different ways. A cluster is already paid for and shared, so you may wait in a queue but do not see a bill; the cloud starts machines almost at once, in any number, but charges for every hour. For a learner or a small project, an institutional cluster is usually the simpler start. The cloud earns its place for very large or urgent jobs, or when the data already live there.
Two cautions before you start.
- Cost. The meter runs while a machine is switched on, whether it is working or not, and storage is charged every month. Moving data out of the cloud, called egress, is usually charged too, and for sequencing data it can be large. Forgotten machines are the classic surprise bill. Set a budget with alerts, shut machines down when a job ends, and delete intermediate files you do not need.
- Data protection. Human genomic and health data are personal data, and in the UK and EU health and genetic data are a special category with extra legal protection. Never upload patient data to a personal cloud account. Use only systems your institution has approved, with the right agreements and data location, and follow the terms of any controlled-access dataset. Much sensitive data can now be analysed only inside a trusted research environment, a secure system where approved researchers work on the data without downloading it.
FAIR data
A pipeline is only reusable if its data can be found and understood. The FAIR principles, now widely required by funders and journals, ask that data be:
- Findable: given a permanent identifier, such as an accession number, and described with searchable metadata, the data about the data (organism, tissue, method, date).
- Accessible: retrievable through a standard, open method. This does not have to mean open to everyone; patient data can be FAIR behind a controlled-access application.
- Interoperable: in standard formats and vocabularies (the file formats of Module 2), so other tools can read it.
- Reusable: with a clear licence, a record of how it was made, and enough detail to be used again.
A workflow, an environment file and a git history are the "how it was made" for your results. Depositing raw reads in a public archive, as Module 2 described, covers the data side.
FAIR is easiest when it starts on day one. Keep a sample sheet that links each file to its sample, condition and accession, use the same sample names in file names, scripts and tables, and write down where each raw file came from. Reconstructing that information at the end of a project, when you come to deposit the data or write the methods, takes far longer and is often incomplete.
Community pipelines: nf-core
For common analyses you may not need to write a pipeline at all. nf-core is a community collection of more than a hundred Nextflow pipelines, for RNA-seq, variant calling, single-cell data and much more. Each one is reviewed by the community, has versioned releases and documentation, runs every step in a container and comes with a small test dataset.
nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir results
What this does
nextflow run nf-core/rnaseqdownloads the RNA-seq pipeline from nf-core and runs it.-r 3.14.0fixes the release, so that the run can be repeated exactly. It is shown as an example; check the pipeline's page for current releases, and write the one you used in your methods.-profile test,dockeruses the built-in test data and runs each step in a Docker container. On a cluster you would useapptainerand your institution's profile instead.--outdir resultsis the folder for all outputs, including a report of every tool version used.
A community pipeline is not a black box you can trust blindly. Read its documentation, run the test profile first, and check its quality-control report, just as you would check your own work.
Key point
A reproducible analysis pins four things: the steps (a workflow), the software (an environment file or container), the code's history (git) and the data (FAIR, with identifiers). Pin all four and someone else, or you in a year, can get the same result.
Each one closes a different gap. Without the workflow, the order of steps is lost; without pinned software, a new tool version can shift the numbers; without git, you cannot tell which version of a script made a figure; without identifiers, no one can find the input. You do not need all four for a quick exploration, but you do for anything you publish or hand on.
Further reading · Module 3Snakemake documentation, including its tutorial. · nf-core documentation: how to run community pipelines. · git documentation. · The FAIR principles (GO FAIR). · All reading for this course
Check your understanding
1. A workflow of 30 samples has finished. You change a setting that affects only the final summary step. What will a workflow manager rerun?
2. Your cluster does not give users administrator rights, but the pipeline you want ships Docker images. What is the usual solution?
3. A patient dataset can be downloaded only after an approved access application. Can it still be FAIR?
Up next
Project
A FASTQ report script