LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 73 lessons

Lesson 3 of 7

Module 5 · Sequencing data

Trimming and filtering

Trimming and filtering

Trimming and filtering

fastp removes adapter sequence and low-quality ends, drops reads that become too short, and writes a report, all in one fast step.

⏱ 10 min readModule 5 · Sequencing dataFree
Lesson at a glance7 steps · about 10 min
  1. One sample, one command
  2. What this does
  3. Reading the output
  4. How to read this table
  5. Every sample with a loop
  6. What this does
  7. Reading the output

One sample, one command

Trimming cuts unwanted bases off reads; filtering removes whole reads that are not worth keeping. Quality control found two things to clean up in the practice data: adapter at the ends of some reads, and quality that falls towards the end of each read. Adapter bases are not from your sample, so they cannot match the genome and can stop a read from aligning. Very low-quality bases are often wrong. fastp deals with both in one pass over the read 1 and read 2 files together.

Recall from From sample to reads where adapter in a read comes from. Each fragment is read for a fixed number of cycles, 100 here. When the fragment is shorter than that, the sequencer keeps going after the end of the sample DNA and reads into the adapter joined to it. So adapter is always at the 3′ end of a read, and how much of it there is depends on how short the fragment was. A read from a 70-base fragment ends in 30 bases of adapter; a read from a 300-base fragment has none.

The two files are processed together for a reason. As From sample to reads showed, the tenth read in the read 1 file is the partner of the tenth read in the read 2 file, and the aligner relies on that order. If you trimmed the two files separately and one tool dropped a short read 1 while keeping its read 2, every pair after that point would be out of step. fastp keeps or drops both reads of a pair together, so the two output files stay matched.

Trimming and filtering · LSMA free course Three 100-base reads before and after trimming. Adapter at the 3 prime end is removed, a low-quality tail is cut off, and a read that is good throughout is left unchanged. 1 · BEFORE TRIMMING2 · AFTER TRIMMING adapter at the 3′ end low-quality tail good throughout adapter removed poor end trimmed unchanged Every read starts at 100 bases. Pairs are kept if both reads are ≥ 36.
Navy is sequence from the sample, vermilion is adapter and peach is a run of low-quality bases at the 3′ end, the end read last. After trimming, reads differ in length. Schematic, not to scale.
Terminal
fastp -i data/ctrl_1_R1.fastq.gz -I data/ctrl_1_R2.fastq.gz \
      -o trimmed/ctrl_1_R1.fastq.gz -O trimmed/ctrl_1_R2.fastq.gz \
      --detect_adapter_for_pe --cut_tail --cut_mean_quality 20 --length_required 36 \
      --html trimmed/ctrl_1_fastp.html --json trimmed/ctrl_1_fastp.json
Read1 before filtering: total reads: 9000 total bases: 900000 Q20 bases: 886635(98.515%) Q30 bases: 882253(98.0281%) … Read1 after filtering: total reads: 9000 total bases: 872533 Q20 bases: 872533(100%) Q30 bases: 868415(99.528%) … Filtering result: reads passed filter: 18000 reads failed due to low quality: 0 reads failed due to too many N: 0 reads failed due to too short: 0 reads with adapter trimmed: 1164 bases trimmed due to adapters: 27583

What this does

  • The command is one line split over four with backslashes. The first two lines name the two input files and the two output files; the next line sets the trimming rules (explained in the table below); the last names the reports.
  • --html writes a report with plots to open in a browser. --json writes the same numbers in JSON, a structured text format that programs can read directly; MultiQC uses it to gather fastp results for all samples.
  • fastp prints the summary you see to the screen. It uses the error channel for this, which is why the loop further down saves it with 2>.
  • fastp also does some filtering by default even when you do not ask for it, for example removing reads with too many unknown N bases or too many low-quality bases, and trimming poly-G tails on data from two-colour instruments (From sample to reads). The Filtering result lines report each reason separately, so you can see which rule removed reads.

Reading the output

  • Before filtering: read 1 had 9,000 reads and 900,000 bases, of which 98.5% were Q20 or better and 98.0% Q30 or better.
  • After filtering: still 9,000 reads, but 872,533 bases, because some reads are now shorter. Every remaining base is Q20 or better, and 99.5% are Q30 or better.
  • The lines marked … were left out of the lesson to save space; among them is the same block of numbers for read 2.
  • Before and after are worth comparing for read 2 as well. Read 2 usually starts with slightly lower quality than read 1, as Quality control explained, so it often loses a few more bases to quality trimming.
  • Filtering result: all 18,000 reads (9,000 pairs × 2) passed, and none was removed for low quality, too many unknown bases (N) or short length. 1,164 reads had adapter cut off, 27,583 adapter bases in all.
OptionWhat it does
-i -I / -o -OInput and output files for read 1 and read 2, kept in step.
--detect_adapter_for_peAlso works out the adapter sequence from the reads themselves. For pairs, fastp already trims adapters where the two reads overlap; this adds a second check.
--cut_tail --cut_mean_quality 20Trims from the 3′ end while the average quality in a sliding window is below Q20.
--length_required 36Drops reads shorter than 36 bases after trimming; both reads of the pair go together.

How to read this table

Each row is one option from the command, in the order it appears. The 3′ end (said "three-prime") is the end of a read the sequencer reached last, where quality is lowest; the other end is the 5′ end. A sliding window is a short run of neighbouring bases, by default four in fastp, that moves along the read one base at a time; trimming stops as soon as the average quality inside the window reaches Q20. Averaging over a window means one poor base in the middle of good ones does not cut the read short. The 36-base minimum removes reads too short to place reliably on the genome, because a very short sequence may match many places equally well.

The threshold of Q20 means a 1 in 100 chance that a base is wrong, the arithmetic from the worked example in From sample to reads. Quality trimming at Q20 therefore removes only the clearly poor stretches; a tail at Q25 or Q30 stays, because those bases are still right 99.7% to 99.9% of the time. Some guides use a stricter threshold such as Q30. For counting reads per gene the difference is small, but a stricter threshold removes more bases, so choose one value, use it for every sample and report it.

The minimum length works on the pair, not the single read. If quality trimming leaves read 1 at 80 bases but read 2 at 30, the pair fails --length_required 36 and both reads are dropped. That is why this rule is set low: a high minimum removes whole pairs because one partner had a poor tail.

Worked example · what trimming changed

read 1 bases: 900,000 → 872,533, so 3.1% removed
bases at Q20 or better: 98.5% → 100%
1,164 of 18,000 reads (6.5%) had adapter removed; no read was dropped

Step by step. Bases removed from read 1: 900,000 − 872,533 = 27,467. As a share: 27,467 ÷ 900,000 = 0.0305, or 3.1%. Quality: 886,635 ÷ 900,000 = 0.985, so 98.5% of bases were Q20 or better before; after trimming, 872,533 of 872,533 are, which is 100%. Adapter: 1,164 ÷ 18,000 = 0.0647, or 6.5%.

What the answer means. Trimming removed about 3% of the bases and kept every read. The bases removed were the ones most likely to cause trouble, adapter and poor tails, so the data are cleaner at almost no cost. If trimming had removed, say, a third of the bases, that would point to a problem with the library worth investigating before going further.

Go deeper: finding adapter without knowing its sequence

Take a fragment of 80 bases read with 100-base paired reads. Read 1 covers the fragment from one end, then runs 20 bases into adapter. Read 2 does the same from the other end. Turn read 2 into its reverse complement, the sequence of the opposite strand read in its own direction, and its last 80 bases match the first 80 bases of read 1.

fastp searches for this overlap, allowing a few mismatches. Where the two reads overlap completely, the fragment must end where the overlap ends, so everything after that point in each read is adapter and is cut off, with no adapter sequence supplied. The option --detect_adapter_for_pe adds a second check: fastp also works out the adapter sequence from the reads and searches for it, which helps when errors hide the overlap. With -c, fastp can also use the overlap to correct bases: where the two reads disagree inside it and one base has much higher quality, that base is kept.

Every sample with a loop

The loop below runs the same command for the four samples, using the variable and ${…} pattern from Variables, loops and scripts.

Running the samples one by one by hand works for four, but it invites mistakes: a typing error in one file name, or a changed option in one command, makes one sample different from the others for no reason. A loop applies exactly the same settings to every sample, so any difference between samples comes from the data, not from how you typed the command.

Terminal
for s in ctrl_1 ctrl_2 treat_1 treat_2; do
  fastp -i data/${s}_R1.fastq.gz -I data/${s}_R2.fastq.gz \
        -o trimmed/${s}_R1.fastq.gz -O trimmed/${s}_R2.fastq.gz \
        --detect_adapter_for_pe --cut_tail --cut_mean_quality 20 --length_required 36 \
        --html trimmed/${s}_fastp.html --json trimmed/${s}_fastp.json 2> trimmed/${s}_fastp.log
done
grep "reads with adapter trimmed" trimmed/*_fastp.log
trimmed/ctrl_1_fastp.log:reads with adapter trimmed: 1164 trimmed/ctrl_2_fastp.log:reads with adapter trimmed: 1510 trimmed/treat_1_fastp.log:reads with adapter trimmed: 1493 trimmed/treat_2_fastp.log:reads with adapter trimmed: 1116

What this does

  • The loop variable s takes each sample name in turn. Every file name is built from it, so the four runs differ only in the name.
  • 2> trimmed/${s}_fastp.log saves each run's summary to its own log file, so the screen stays quiet and nothing is lost.
  • After the loop, grep searches all four logs for one line. Given several files, grep puts the file name in front of each match, so you can tell which sample each number belongs to.

Reading the output

Each count is read 1 and read 2 together. To compare samples of different depth, divide by the number of reads: ctrl_1 1,164 ÷ 18,000 = 6.5%; ctrl_2 1,510 ÷ 22,000 = 6.9%; treat_1 1,493 ÷ 20,000 = 7.5%; treat_2 1,116 ÷ 17,000 = 6.6%. The samples are similar, which is what you want: no library is much shorter than the others.

Before moving on, confirm that the loop did what you expect. List trimmed/ and check that there are two trimmed FASTQ files, one HTML report, one JSON file and one log for each of the four samples. If a name in the loop is mistyped, fastp stops for that sample with an error saying it cannot open the input file, the error goes into that sample's log, and the loop carries on with the next one. The screen stays quiet either way, so a missing output file is the only sign.

Then run FastQC again on the trimmed reads, as Quality control suggested, and gather the before and after reports with MultiQC. Adapter content should now be close to zero for every sample, and the trimmed files are the input to alignment in Aligning reads to a genome.

fastp is described by Chen, Zhou, Chen and Gu (2018, Bioinformatics, doi:10.1093/bioinformatics/bty560). Whether trimming helps RNA-seq at all has been tested directly: Liao and Shi (2020, NAR Genomics and Bioinformatics) found that, for counting reads per gene, trimming made little difference when the aligner can leave unmatched read ends unaligned, which supports the advice to trim gently. Trimming matters more for assembly and variant calling, where leftover adapter can create false sequence. Defaults can change between releases, so report the fastp version and every option you set, exactly as in your command.

Key point

Trim gently. Modern aligners cope with a few low-quality bases, and aggressive trimming throws away good data. Remove adapters, trim clearly poor ends, and report the settings.

For example, raising --length_required from 36 to 90 would sound like a way to keep only "good" reads, but it would throw away every pair whose fragment was shorter than 90 bases, and those reads are perfectly good sequence from your sample. Report the settings in your methods as they appear in the command, so that someone else could repeat the step exactly.

Further reading · Module 5Conesa et al. 2016: the whole RNA-seq workflow, from design to interpretation. · All reading for this course

Check your understanding

1. Why must R1 and R2 be trimmed together?

2. What does --length_required 36 do?

3. Trimming removed 27,583 bases of adapter from 900,000. Roughly what share?

Up next

Lesson 4

Aligning reads to a genome

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.