Lesson 3 of 10
Module 3 · The command line
Sequencing files can be gigabytes long. You never open them in an editor; you look at the start, the end and the size, and count what is inside. This lesson meets the three formats you will use most.
Small files: cat, head and tail
A text editor loads the whole file into memory before it shows you anything. That is fine for a sample sheet of a few lines, but a single sequencing file can hold tens of millions of lines, and the editor may freeze or crash. The command line works differently: these tools read a file from the start and print as they go, so they can show you the first few lines of a huge file at once.
Four small tools cover most of your looking. cat prints a whole file (the name comes from "concatenate", because it can also join several files one after another). head prints the start of a file and tail the end. wc ("word count") counts lines, words and characters; with -l it counts lines only.
The commands below are run from inside the project folder you built in Files, folders and wildcards, which is why every path starts with data/. Looking before analysing is not a formality. Most failed analyses trace back to a file that was not what its name promised: a header row where none was expected, the wrong chromosome names, a file cut short. A few seconds with head and wc catches these before they turn into confusing errors several steps later.
cat data/practice_samples.csv # print the whole file head -n 3 data/practice_genes.gtf # the first 3 lines tail -n 1 data/practice_genes.gtf # the last line wc -l data/practice_genes.gtf data/practice_genome.fa # count lines
What this does
cat data/practice_samples.csvprints every line of the sample sheet to the screen. Use it only on files you know are short.head -n 3 data/practice_genes.gtfprints the first three lines. The option-nsays how many; without it,headprints ten.tail -n 1 data/practice_genes.gtfprints the last line. Looking at the end of a file is a quick way to check that it is complete: a file cut short by a failed download or a crashed program often ends in the middle of a line.wc -lcounts the lines of each file named after it, and adds a total when you give it more than one.- Everything after a
#is a comment: a note for people, which the shell ignores.
Reading the output
- The first five lines come from
cat. The sample sheet is a CSV file ("comma-separated values"): the first row is a header naming the columns, and each later row is one sample with its condition. Four samples, two controls and two treated. - The next three lines come from
head: the gene prA (ID PRG001), its single exon, and the gene prB. The line after them, fromtail, is the last exon in the file, belonging to prL. - The last three lines come from
wc -l: 27 lines in the GTF file, 1,001 in the genome file and 1,028 altogether. The genome file is in FASTA format: one header line naming the sequence, then the DNA letters spread over many lines. You will take it apart in Text processing with awk and sed.
Two details about these tools save trouble later. First, tail can also start from a given line: tail -n +2 data/practice_samples.csv prints everything from line 2 onwards, which is the sample sheet without its header row. This is a common way to drop a header before counting or sorting the rows, and it works on Linux and macOS alike. Second, wc -l counts lines, not records. In the FASTA file, one sequence is spread over many lines, so 1,001 lines does not mean 1,001 sequences. In a FASTQ file every read takes four lines, as you will see below. Always ask what one line stands for before you read a count as a number of things.
A GTF file describes genes: one feature per line, nine columns separated by tabs. Columns 1, 3, 4, 5 and 7 are the chromosome, feature type, start, end and strand; column 9 holds IDs and names. For longer files, less file lets you scroll (press q to quit).
The nine GTF columns, using the first line above
- Chromosome: chrP, the name of the sequence in the genome file that the feature sits on.
- Source: LSMA, the group or program that made the annotation.
- Feature type: gene or exon here. An exon is a stretch of a gene that is kept in the finished RNA; real annotations also list transcripts, coding regions and more.
- Start: 1500, the first position of the feature.
- End: 2743, the last position. Positions count from 1 at the start of the chromosome, and both ends are part of the feature. The start is always the smaller number, whichever strand the gene is on.
- Score: a dot, meaning "no value".
- Strand: + or -. DNA has two strands, and a gene can be read from either. A + gene is read in the direction the coordinates increase; a - gene, such as prB, is read the other way.
- Frame: a dot again. It is only filled in for protein-coding segments, where it says how the codons line up.
- Attributes: pairs of a key and a quoted value, separated by semicolons, such as gene_id "PRG001". The gene ID is a stable code that programs use; the gene name is the label people use.
The columns are separated by a tab, an invisible character that the terminal shows as a jump to the next column stop. That is why the columns in the output line up only roughly. Inside less, the space bar moves down a page, b moves back a page, and typing / followed by a word searches for it.
Long lines, like the exon lines above, wrap round onto the next screen line and make a table hard to read. less -S turns wrapping off: each line of the file stays on one screen line, and the right and left arrow keys scroll sideways. less never changes the file, so it is a safe way to look at anything.
Go deeper: invisible characters at the ends of lines
Files made on Windows usually end each line with two invisible characters, a carriage return and a newline, where Linux and macOS use the newline alone. The extra carriage return is easy to miss. cat and head look normal, but a script may find that the last column of every line ends with a hidden character, so a sample called ctrl_1 no longer matches ctrl_1 typed elsewhere.
Two quick checks show it. file data/practice_samples.csv describes the file and adds 'with CRLF line terminators' when Windows endings are present. cat -e marks the end of every line with $ and shows a carriage return as ^M just before it. Spreadsheet exports are a common source, so check any sample sheet that has passed through Excel before you trust a script that reads it.
Compressed files
FASTQ files are almost always compressed with gzip (.gz). gzip -dc decompresses to the screen without changing the file, so you can pipe it into other commands. (zcat does the same on Linux, but not on macOS.)
gzip is a compression program: it rewrites a file in a shorter coded form, and the same program can turn it back. Text made of only four letters, like DNA, compresses well, so a compressed FASTQ file is several times smaller than the original. The two options mean -d "decompress" and -c "write the result to the screen instead of to a file". Without -c, gzip -d would replace the .gz file with a full-size uncompressed copy, which is exactly what you want to avoid.
A common mistake is to run cat or head on the .gz file itself. The compressed form is not text, so the screen fills with unreadable symbols, and the terminal may be left showing strange characters afterwards. Press Ctrl+C to stop it; if the terminal still looks wrong, typing reset and pressing Enter restores it. A related check is gzip -t file.gz, which tests that a compressed file is complete and undamaged. It prints nothing if the file is sound and an error if it was, for example, cut short during a download.
The pipe, the | character, sends what one command prints into the next command instead of onto the screen. Searching, sorting and pipes is about pipes; here you only need to know that gzip -dc file.gz | head means "decompress the file, and show me only the start".
ls -lh data/ctrl_1_R1.fastq.gz # -l long listing, -h human-readable sizes gzip -dc data/ctrl_1_R1.fastq.gz | head -n 4 echo $(( $(gzip -dc data/ctrl_1_R1.fastq.gz | wc -l) / 4 )) # number of reads du -sh data # disk use of a folder
What this does
ls -lhlists the file in long form, with sizes in K (kilobytes), M (megabytes) or G (gigabytes) instead of a raw number of bytes.gzip -dc … | head -n 4decompresses the reads and keeps only the first four lines. As soon asheadhas its four lines it stops reading, andgzipstops soon after, so this returns at once even for a file of many gigabytes.- The third line works from the inside out.
gzip -dc … | wc -lcounts the lines of the decompressed file.$( … )puts that number in place,$(( … / 4 ))divides it by 4, because every read takes four lines, andechoprints the result. du -sh data("disk usage") reports how much space the whole folder takes:-sgives one total instead of a line for every file inside, and-hmakes it human-readable.
Reading the output
- -rw-r--r-- are the file's permissions (Variables, loops and scripts explains them). The 1 after them is a link count you can ignore. you you are the owner and the owner's group, then come the size (645 kilobytes), the date and time it was last changed, and the name.
- The next four lines are the first read of the file: one complete FASTQ record. The figure and worked example below take it apart.
- 9000 is the number of reads in the file.
- 2.8M is the total size of the data folder: the compressed FASTQ files plus the genome, annotation and sample sheet.
The read count deserves one check of its own. Shell arithmetic with $(( )) works in whole numbers and drops any remainder, so a damaged file with 36,002 lines would still print 9000. If you need to be sure, look at the remainder as well: replacing / 4 with % 4 in the same line prints the remainder of the division, which should be 0 for a complete FASTQ file. Note too that du reports the space a file takes on the disk, which can be a little larger than the size ls -l shows, because disks store files in fixed-size blocks.
Worked example · reading one FASTQ record
line 2 · the 100 bases
line 3 · + (a separator)
line 4 · one quality character per base: H = ASCII 72 → 72 − 33 = Q39
36,000 lines ÷ 4 = 9,000 reads
Step by step. Every character on a computer is stored as a number, its ASCII code. The capital letter H is stored as 72. FASTQ files store each quality score as the character whose code is the score plus 33, so to decode a character you subtract 33: 72 − 33 = 39. The 33 is there because the codes below 33 are spaces and invisible control characters, which could not be printed; adding 33 makes every score a visible character. For the count of reads, 9,000 reads × 4 lines = 36,000 lines, the number wc -l would report before the division.
What the answer means. Look at the end of the quality line in the output: characters such as C (code 67, Q34), @ (code 64, Q31) and > (code 62, Q29) are lower than the H characters at the start. Quality usually falls along a read, and you will see the same pattern in the quality-control report in Module 5. Notice also that @ can appear inside a quality line, so you cannot find the start of each record by searching for lines that begin with @. Count by position instead: every fourth line is a name.
Quality Q means an error chance of 10−Q/10: Q30 is 1 in 1,000, Q39 about 1 in 8,000.
The same rule written the other way round is Q = −10 × log10(error chance). For Q39: 10−3.9 ≈ 0.000126, which is about 1 in 7,900, rounded to 1 in 8,000. Each step of 10 in Q means ten times fewer errors.
The '+33' encoding is described in Cock et al. (2010), 'The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants', Nucleic Acids Research 38:1767–1771. The paper is also a warning. Older Illumina software, in use until about 2011, wrote quality as the score plus 64, and early Solexa files used a different scale altogether, so check the encoding of old data before you decode it; FastQC reports the encoding it detects. The quality scale itself goes back to the Phred base caller (Ewing & Green, 1998, Genome Research). Remember that a quality score is the sequencer's own estimate of error, not a measured error rate.
Key point
Never decompress large FASTQ files to disk just to look at them. Stream them with gzip -dc … | head; most bioinformatics tools read .gz files directly.
You will see this throughout the course: seqkit, FastQC, fastp and HISAT2 are all given the .fastq.gz files as they are. A second reason to keep files compressed is space: an uncompressed copy of every sample would take several times the disk, and on a shared server that space usually belongs to everyone.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. A FASTQ file has 400,000 lines. How many reads?
2. Which GTF column holds the strand?
3. What does gzip -dc file.gz | head do to file.gz ?
Up next
Lesson 4
Searching, sorting and pipes