LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 73 lessons

Guided project

Module 3 · The command line · Self-checked

A FASTQ report script

A FASTQ report script

A FASTQ report script

Write one script that summarises every FASTQ file in a folder, the first step of any sequencing analysis. Work through the six tasks, check each against its checkpoint, and tick it off. Nothing is submitted.

⏱ 10 min readModule 3 · The command line · Self-checkedFree
Lesson at a glance7 steps · about 10 min
  1. How to approach this
  2. Set up the project
  3. Count and pair
  4. Write the report script
  5. Make it safe
  6. Adapters
  7. Run, save and document

Research question

Are the four practice FASTQ files complete and of similar quality, so they can go on to analysis?

How to approach this

This project puts the whole module together. The research question has two parts. "Complete" means every file holds the reads it should, and each read 1 file has a partner read 2 file with the same number of reads. "Of similar quality" means no file stands out on read length, GC content, base quality or adapter content. Your script answers both at once by printing one row of numbers per file, so the files can be compared side by side.

Checks like these come first in every sequencing analysis, before any alignment or counting. A file that was cut short in transfer, or a sample with far more adapter than the others, is cheap to spot now and expensive to discover after hours of computing. Writing the checks as a script, rather than typing them, means you can run exactly the same checks on the next batch of files.

Every tool you need has already appeared in this module. The read counts and quality decoding come from Looking inside files, the adapter search from Searching, sorting and pipes, the per-line calculations from Text processing with awk and sed, and the loop and safety lines from Variables, loops and scripts. The task titles below give the lesson numbers within the module.

  • Work through the tasks in order. Each one builds on the last, and the lesson named under each task title shows the method.
  • Try every piece at the command line first, on one file, before you put it into the script. Use ctrl_1_R1.fastq.gz: you already know several of its numbers from the lessons, so you can tell at once whether a command is right.
  • Change one thing at a time. Save the script, run it, check the output, then add the next column. When something breaks, you will know which change did it.
  • Leave the raw data alone. Read from data/, write only to results/, and never edit a FASTQ file in place.
A FASTQ report script · LSMA free course Four FASTQ files, read 1 and read 2 for ctrl_1 and treat_1, go into one script, fastq_report.sh, which writes one table with a row per file and columns for name, reads, mean length, GC percentage, mean quality and adapter percentage. 1 · FOUR FASTQ FILES, ONE SCRIPT ctrl_1_R1.fastq.gz ctrl_1_R2.fastq.gz treat_1_R1.fastq.gz treat_1_R2.fastq.gz fastq_report.sh 2 · ONE ROW PER FILE namereadsmean lengthGC %mean qualityadapter % ctrl_1_R1.fastq.gz ctrl_1_R2.fastq.gz treat_1_R1.fastq.gz treat_1_R2.fastq.gz Saved as results/fastq_report.tsv, with a header line and tabs between columns.
The grey bars are the numbers your script will fill in; the checkpoints tell you what to expect. Schematic, not to scale.
1

Set up the project

The terminal and where to run it and Files, folders and wildcards

  • Make project/ with data/, scripts/ and results/, and copy the course files into data/.
  • Check the files with ls -lh.

Hint. mkdir -p, used in Bigger jobs, servers and clusters, makes a folder and does not complain if it already exists. In the ls -lh output the size is the column immediately before the date; Looking inside files shows how to read the whole line.

File sizes are a first, rough check of completeness. The two files of a pair hold the same number of reads of the same length, so their compressed sizes should be close. A file much smaller than its partner is the usual sign of a download that stopped early; gzip -t, from the lesson on servers, tests whether a compressed file is whole.

Checkpoint: four FASTQ files, two per sample, each between 640 and 720 KB.

2

Count and pair

Looking inside files and Variables, loops and scripts

  • Count the reads in every file.
  • Check that each R1 file has the same number of reads as its R2 file.

Hint. Looking inside files counts the reads in one file: lines divided by 4. Variables, loops and scripts wraps that in a loop. For pairing, notice that basename can strip either ending, _R1.fastq.gz or _R2.fastq.gz, so both files of a pair give the same sample name; print the two counts on one line and compare them.

Pairing matters because tools that use paired reads take them in step: the first read in the R1 file goes with the first read in the R2 file, the second with the second, and so on. If one file has lost reads, every pair after the gap is mismatched, and the tool may stop with an error or, worse, carry on with wrong pairs. Equal counts do not prove the pairs are right, but unequal counts prove they are wrong.

Checkpoint: ctrl_1 has 9,000 reads in each file and treat_1 has 10,000; the pairs match.

3

Write the report script

Text processing with awk and sed and Variables, loops and scripts

  • Write scripts/fastq_report.sh that prints, for each file given: name, reads, mean length, GC% and mean base quality.
  • Hint for quality: in awk, build a lookup of character codes in a BEGIN block, then sum (code − 33) over line 4 of each record.

Hint. Start from the GC program in Text processing with awk and sed: it already counts reads, bases and G and C letters on the sequence lines. Mean length is total bases divided by reads. Quality lies on the lines where NR % 4 == 0. Standard awk has no function that turns a character into its code, which is why the hint above suggests building a lookup table once at the start; after that, each quality character can be looked up, decoded as in Looking inside files and added to a total. Divide by the number of bases at the end. Check against what you know: the H characters at the start of the first read are each worth Q39.

Do the divisions inside awk, not with the shell's $(( … )). Shell arithmetic keeps whole numbers only, so a GC content of 45.3% would come out as 45, and a mean quality just under 35 as 34. awk works with decimals, and its printf lets you choose how many decimal places to print. It also helps to build the script one column at a time: get reads right, then add mean length, then GC, then quality, rerunning on ctrl_1_R1.fastq.gz after each step.

Checkpoint: every file shows a mean length of 100.0 and GC of 45.3%; mean quality is about 35.

4

Make it safe

Variables, loops and scripts

  • Add the shebang, a usage comment and set -euo pipefail.
  • Run it on a file that does not exist and check the exit code.

Hint. The script count_reads.sh in Variables, loops and scripts has all three safety lines and shows how to read the exit code with $? straight after the run. If a misleading line still appears for the missing file, look at where in the loop the line is printed: it should be printed only after every number for that file has been worked out.

This task matters because a report script is often run on many files at once and left unwatched. A run that stops with a clear error costs a minute. A table with one quietly wrong row can be copied into a paper's supplementary data and never noticed.

Checkpoint: the script stops with an error message and a non-zero exit code, and writes no misleading output line.

5

Adapters

Searching, sorting and pipes

  • Add a column with the percentage of reads containing AGATCGGAAGAGC.

Hint. Searching, sorting and pipes counts adapter reads with grep -c on the decompressed file. The percentage is adapter reads divided by all reads, times 100. The worked example in Searching, sorting and pipes does this sum for one file, so use it to check your column.

Treat this column as an estimate. grep finds only exact copies of the 13 letters, so it misses adapters that contain a sequencing error and adapters cut short at the very end of a read, where fewer than 13 letters fit. The true share is therefore a little higher. The trimming tools you will meet in Module 5 look for partial and imperfect matches as well.

Checkpoint: about 5% of reads in each file contain the adapter (431 of 9,000 in ctrl_1 read 1).

6

Run, save and document

Searching, sorting and pipes, Installing software with conda and Bigger jobs, servers and clusters

  • Run the script on all files and save a tab-separated table to results/fastq_report.tsv.
  • Write a short README.md: what the script does, how to run it, and the tools it needs.
  • Optional: write a Slurm script that runs it.

Hint. Print the header line once, before the loop, with the same tab separators as the rows; then send the script's output to the file with > (Searching, sorting and pipes). A good README answers four questions: what the script does, the exact command to run it, what each output column means, and which tools it needs, with the versions you used (Installing software with conda). For the optional part, adapt the Slurm script in Bigger jobs, servers and clusters.

Then record the finished project with git, as in Workflows, the cloud and reproducible pipelines: add the script, the README and the environment file, and commit them with a message that says what they are. Leave the FASTQ files and the results table out of the commit; the script can make the table again at any time.

Checkpoint: the table has a header and four rows, and someone else could run your script from the README alone.

Go deeper: why the mean quality score flatters a read

Task 3 averages Q values, which is the usual quick summary, but it hides something. Q is a logarithm, so the mean of the Q values is not the same as the mean error chance. Take two bases, one at Q40 (an error chance of 1 in 10,000) and one at Q10 (1 in 10). Their mean Q is 25, which sounds like about 1 error in 300. The mean error chance, though, is (0.0001 + 0.1) ÷ 2 = 0.05, or 1 in 20, which is Q13.

So a few very poor bases barely move the mean Q while causing most of the errors. For the files in this project the difference is small, because almost every base scores in the thirties. If you ever need the expected number of errors in a read, turn each Q into an error chance first, 10 to the power −Q/10, and add those up.

Take it further

Put the project on GitHub. Then, after Module 4, use these skills on a full pipeline in Module 5 of this course, Sequencing data, from quality control to gene counts.

GitHub is a website that stores projects managed with Git, a version-control program that records every change to your files so you can see what changed and go back. Put the scripts, the README and the environment.yml there, but not the FASTQ files: Git is designed for text that you edit, and GitHub refuses very large files. In the README, say where the data can be obtained instead.

Look back at what you have built: a script that takes any number of FASTQ files, checks them and reports in a form that you, a colleague or a later program can read. That is the shape of every step of the pipeline in Module 5, where real tools such as FastQC and fastp do the same job in more depth.

The next module, Comparing sequences, builds directly on this. You can now open, count and summarise sequences in the terminal; Module 4 asks what those sequences mean by comparing them with each other. It starts with homology, the idea that two sequences are similar because they share an ancestor, then shows how sequences are lined up in an alignment and how a database search finds the closest known matches to a sequence you do not recognise.

End of Module 3: The command lineYou can stop here: this module is complete on its own. Or carry on to Module 4, Comparing sequences (Intermediate).

Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course

Up next

Lesson 1

Why compare sequences: homology, similarity and identity

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.