Guided project
Module 5 · Sequencing data · Self-checked
Run the whole pipeline on the four practice samples as one script, from raw reads to a table of counts per gene. Because the data were simulated, you know the right answer and can check every step. Nothing is submitted.
Research question
Can one script take four samples from raw reads to a clean count table, and do the counts already hint at the genes that treatment changes?
How to approach this
In the lessons from From sample to reads to Counting reads per gene you ran each tool by hand, often on one sample at a time. This project joins those commands into a single pipeline script: a text file of shell commands that runs from start to finish without you typing anything in between. The goal is not new biology. It is a result you can rebuild from scratch at any time, which is the habit that makes larger projects manageable.
A script is also a record. Six months from now, neither you nor a reviewer will remember which trimming threshold you typed on the day, but the script will say. That is why the methods paragraph in task 4 is written from the script, not from memory: the script is the only place where the settings you actually ran are written down.
- Work on one sample first. Get every step working for ctrl_1, check its output against what you saw in the lessons, and only then wrap the steps in a loop over all four samples (ctrl_1, ctrl_2, treat_1, treat_2).
- Keep the folder layout from From sample to reads (data, qc, trimmed, index, aligned, counts), so each step reads from one folder and writes to the next.
- Run it from empty. When the script works, delete the output folders and run it again. If it only works because of files left over from earlier attempts, it is not finished.
- Use the known answer. The data were simulated, so the checkpoints tell you exactly what a correct run produces. A mismatch is a clue to a bug, not a result to explain away.
- Read the logs after every run. A script that finishes without an error has not necessarily done the right thing. The logs from fastp and HISAT2, and the featureCounts summary, are where a wrong file or a wrong setting shows up, often as a number that is far from the checkpoint.
One pipeline script
The lessons from From sample to reads to Counting reads per gene
- Write scripts/pipeline.sh that runs FastQC, fastp, HISAT2, samtools and featureCounts for every sample listed in practice_samples.csv.
- Use
set -euo pipefail, and keep each tool's log.
Hint. The loop over sample names in Trimming and filtering is the pattern to reuse: replace the fixed list of names with names read from the CSV file. The line set -euo pipefail makes the script stop at the first problem: -e stops on any command that fails, -u stops if you use a variable that was never set (often a typo), and -o pipefail makes a chain of commands joined by | count as failed if any part fails, which matters for the HISAT2-into-samtools step from Aligning reads to a genome. Several tools print their summary to the screen rather than to a file, so send that text to a log named after the sample, as Trimming and filtering and Aligning reads to a genome did. Remember that the HISAT2 index only needs building once, not once per sample.
To get the sample names from the CSV file, skip its header line and keep the column that holds the names, using the tools from Module 3 (tail and cut, with -d, to split on commas). Print the list on its own first and check that it gives exactly the four names, with no header and no blank line, before you loop over it. A stray header in the list would make the loop look for files called after the column name, and with set -e the script would stop there with an error about a missing file.
Checkpoint: one run produces 4 BAM files, their indexes, and counts/practice_counts.txt from scratch.
A QC table
The lessons from Quality control to Aligning reads to a genome
- Collect, per sample: raw pairs, pairs after trimming, reads with adapter trimmed, and overall alignment rate.
Hint. Every number you need is already in a log your script kept. The fastp log (Trimming and filtering) gives reads before and after filtering and the line "reads with adapter trimmed"; the HISAT2 log (Aligning reads to a genome) ends with an "overall alignment rate" line. Watch the units: fastp reports reads, and counts Read1 and Read2 separately, while the table asks for pairs. Searching the logs for a fixed phrase, as Trimming and filtering did with grep, pulls the same line out of all four logs at once.
For example, the ctrl_1 fastp log reports 9,000 reads in Read1 before filtering. Each pair has exactly one read 1, so that is 9,000 pairs. The line "reads passed filter: 18000" counts read 1 and read 2 together, so it too means 9,000 pairs. Put the four samples in one table, one row per sample, so that an unusual sample stands out as it would in MultiQC.
Checkpoint: raw pairs 9,000, 11,000, 10,000 and 8,500; no pairs lost in trimming; alignment rates between 99.6% and 99.9%.
Read the count table
Counting reads per gene
- On the command line, show the gene IDs and the four count columns of counts/practice_counts.txt, sorted by one sample.
- Check in practice_counts.txt.summary that every sample has nearly all its pairs assigned.
Hint. Recall the layout of a featureCounts table from Counting reads per gene: a first line starting with # that records the command, then a header line, then one row per gene. Column 1 is the gene ID, columns 2–6 describe the gene's position and length, and the counts start in column 7, one column per BAM file. Counting reads per gene already showed how to keep columns 1 and 7 onwards. To sort, the values must be compared as numbers, largest first, and the header line must not be sorted in with the genes. In the summary file, the row to read is "Assigned".
The options you need are those from Searching, sorting and pipes in Module 3: -k chooses the column to sort on, -n compares numbers rather than text and -r reverses the order. Without -n, sort compares the characters one by one, so 941 would come after 3161 because 9 comes after 3. Then compare the top genes in the control and treated columns: a gene that moves up the list in both treated samples is a first hint of a treatment effect, though not yet evidence of one.
Checkpoint: 12 genes × 4 samples; PRG005 has the most reads in the control samples, and PRG010, which treatment induces, in treat_1. Assigned pairs: 8,998, 10,999, 10,000 and 8,500.
Methods and limitations
The lessons from From sample to reads to Beyond RNA-seq: other assays and pipelines
- Write a methods paragraph with every tool, version and setting.
- List two limitations of this experiment.
Hint. Most tools print their version when asked with --version; record the versions from the environment you actually ran, not from memory. Write the settings exactly as they appear in your script, including the trimming thresholds and the featureCounts options, and name the genome and annotation files. For the limitations, think about the design of the experiment: how many independent samples each group has, and where the data came from.
A methods paragraph follows the order of the analysis: the reads (instrument, read length, paired-end), quality control, trimming with its thresholds, alignment with the genome and its release, and counting with its options and annotation. Each limitation should say what it means for the result. Two replicates per group, for example, give a poor estimate of how much each gene varies, as From sample to reads explained, so any test of change would have little power. Simulated data have known answers, which makes them good for checking a pipeline but says nothing about real biology.
Checkpoint: someone could rerun your pipeline from your methods alone; your limitations mention two replicates per group and simulated data.
Go deeper: when strict mode stops a script that is fine
set -e stops the script on any command that returns a non-zero exit status, the number a program hands back to say whether it succeeded. Most programs return 0 for success, but a few use other numbers for ordinary results. grep is the classic case: it returns 1 when it finds no match. So a line such as grep -c "WARN" qc/summary.txt, run on a sample with no warnings, stops the whole pipeline although nothing went wrong.
With pipefail the same applies inside a pipe: grep "pattern" file | wc -l fails if grep finds nothing. Where finding nothing is an acceptable answer, write grep … || true, which tells the shell that this one failure is expected. Use it sparingly: the point of strict mode is that real failures are never ignored.
Take it further
Test which genes change with treatment: the next step is a statistical test for differential expression with a tool such as DESeq2 or edgeR, run on this count table and its sample sheet. Run nf-core/rnaseq on the practice data and compare its counts with yours. When there is no reference genome to align to, continue with Module 6, Genome assembly.
If you compare your counts with the nf-core ones, expect small differences rather than identical numbers. The pipeline may use different default settings or a different aligner, and seeing exactly where the two disagree is a good way to learn what each setting does. Large differences, on the other hand, usually mean the two runs used different genome or annotation files.
Everything in this module rested on one assumption: that a reference genome already exists for reads to be aligned to. Module 6 removes it. When no reference exists, the reads have to be joined to each other instead, to rebuild the genome itself. The quality checks and trimming from Quality control and Trimming and filtering still come first, and the alignment ideas from Aligning reads to a genome come back when you check the new genome by aligning reads to it. The ideas of coverage and depth from this module also return, to estimate how large an unknown genome is.
Further reading · Module 5Conesa et al. 2016: the whole RNA-seq workflow, from design to interpretation. · All reading for this course
Up next
Lesson 1
What genome assembly is