Lesson 5 of 10
Module 3 · The command line
awk works through a file line by line and column by column, doing arithmetic as it goes. It is the quickest way to summarise a GTF or FASTQ file without writing a program.
Columns, conditions and sums
An awk program is condition {action}. $5 is column 5, NR the line number, and END runs once at the end. -F '\t' says the columns are separated by tabs.
awk is a small programming language built into every Unix system; its name comes from the initials of its three authors, Aho, Weinberger and Kernighan. It reads its input one line at a time. For each line it splits the text into columns, called fields, and checks the condition. If the condition is true, it runs the action in curly braces. You can leave either part out: with no condition the action runs on every line, and with no action awk prints the line.
The whole program is written inside single quotes, so that the shell passes it to awk untouched. Variables you make up yourself, such as n or total, need no setting up: awk starts every new variable at zero.
In Searching, sorting and pipes you chose rows with grep and columns with cut. awk does both in one step, and it can also calculate. It is also more precise. grep -w "gene" keeps any line where the word appears anywhere; $3 == "gene" tests column 3 only, so a line can never be picked because the word turns up in its attributes. Conditions can be combined: && means "and", so $3 == "gene" && $7 == "-" keeps only the genes on the minus strand.
# gene ID and length for each gene
awk -F '\t' '$3 == "gene" {split($9, a, "\""); print a[2] "\t" $5 - $4 + 1}' \
data/practice_genes.gtf | head -n 3
# number of genes and mean length
awk -F '\t' '$3 == "gene" {n++; total += $5 - $4 + 1}
END {print n " genes, mean length " total / n " bp"}' data/practice_genes.gtfWhat this does
- In the first program, the condition
$3 == "gene"picks the lines whose third column is exactly gene;==means "is equal to". The exon lines are skipped. - For each gene line, the action takes column 9 apart (explained below), then prints the gene ID, a tab (
"\t") and the length. Writing pieces side by side, as ina[2] "\t" $5 - $4 + 1, joins them into one line. - In the second program,
n++adds one tonfor every gene line, andtotal += …adds that gene's length to a running total. Nothing is printed line by line. TheENDblock runs after the last line and prints the count and the mean,total / n. - The second program runs over two lines of the terminal. Because the opening single quote has not been closed at the end of the first line, the shell waits for the rest of the program.
The -F '\t' is easy to leave out, and the result can look almost right. Without it, awk splits each line wherever there are spaces or tabs. Columns 1 to 8 of a GTF line contain no spaces, so they come out the same, but column 9 does: gene_id "PRG001"; gene_name "prA"; would be cut into four fields, and $9 would hold only gene_id. Naming the tab as the separator keeps column 9 in one piece.
Reading the output
- PRG001 1244: the gene runs from 1,500 to 2,743 (Looking inside files), so 2,743 − 1,500 + 1 = 1,244 bases.
- PRG002 2030: 6,926 − 4,897 + 1 = 2,030. PRG003 2236: 11,709 − 9,474 + 1 = 2,236.
- The last line: 12 genes with a mean length of 1,836.25 bases, so their lengths add up to 12 × 1,836.25 = 22,035 bases. bp stands for "base pairs", the usual unit of length for DNA.
split($9, a, "\"") cuts column 9 at each double quote, so a[2] is the text inside the first pair of quotes: the gene ID. Length is end − start + 1 because GTF positions include both ends.
Here is column 9 of the first gene line, cut at each double quote: piece 1 is gene_id , piece 2 is PRG001, piece 3 is ; gene_name and piece 4 is prA. The quote character is written "\"" because a bare quote would end the text; the backslash says "this quote is part of the text". So a[4] would give the gene name.
The "+ 1" is often forgotten. A feature from position 1 to position 3 covers three bases, 1, 2 and 3, but 3 − 1 gives only 2. Not every format counts this way: the BED format, common for genome intervals, starts counting at 0 and leaves the end position out, so there the length is end − start with no + 1. Always check which convention a file uses before you do arithmetic on its positions.
Go deeper: NF, and keeping tabs in the output
awk has a few more built-in variables that save typing. NF holds the number of fields on the current line, so $NF is always the last column, whatever its number. That gives a quick check of a table's shape: awk -F '\t' '{print NF}' data/practice_genes.gtf | sort | uniq -c should report 9 fields for all 27 lines. Any other number means a missing or extra tab somewhere.
A second detail catches many people. When print is given items separated by commas, awk joins them with the output field separator, OFS, which is a single space unless you change it. To keep a tab-separated table tab-separated, set it at the start: awk -F '\t' -v OFS='\t' '{print $1, $4, $5}' data/practice_genes.gtf. Items written side by side with no comma, as in this lesson, are joined with nothing at all.
Every fourth line: FASTQ
In a FASTQ file the sequence is on lines 2, 6, 10 … so NR % 4 == 2 (the remainder after dividing the line number by 4) picks out the sequences.
The % sign is the remainder (or modulo) operator. Line 6 divided by 4 is 1 with 2 left over, so 6 % 4 is 2. As the line number climbs, the remainder cycles 1, 2, 3, 0, 1, 2, 3, 0, in step with the four lines of each record, so each remainder always lands on the same kind of line.
Why count by position instead of searching for a pattern? In Looking inside files you saw that the quality line can contain @, the character that also starts every name line, so a search for lines beginning with @ can pick up quality lines by mistake. The position of a line in its record never changes, so NR % 4 is always right, as long as the file is complete and every record has exactly four lines.
The other remainders work the same way: NR % 4 == 1 picks the name lines and NR % 4 == 0 the quality lines.
gzip -dc data/ctrl_1_R1.fastq.gz \
| awk 'NR % 4 == 2 {n++; b += length($0); gc += gsub(/[GC]/, "")}
END {printf "%d reads, %.1f%% GC\n", n, 100 * gc / b}'
# FASTQ to FASTA: keep the name (with > instead of @) and the sequence
gzip -dc data/ctrl_1_R1.fastq.gz \
| awk 'NR % 4 == 1 {print ">" substr($0, 2)} NR % 4 == 2 {print}' | head -n 4What this does
- The first command sends the decompressed reads into awk. For every sequence line it adds one to
n, adds the line's length tob($0is the whole line, andlength($0)its number of characters), and adds the number of G and C letters togc. The order inside the braces matters: the length is taken first, beforegsubremoves letters from the line. - The
ENDblock prints withprintf, which fills in a template.%dis replaced by a whole number,%.1fby a number with one decimal place,%%prints a percent sign (a single%would start another placeholder), and\nends the line. - The second command turns FASTQ into FASTA, the simpler format that holds sequences without qualities: a header line starting with >, then the sequence. On name lines,
substr($0, 2)takes the line from its second character onwards, dropping the @, and awk prints > in front. Sequence lines are printed as they are. Separator and quality lines match neither condition and are dropped.
One pitfall concerns several files at once. NR counts lines across all the input awk is given, from the first file to the last. Here that does not matter, because gzip -dc sends one stream. If you name several plain files after the awk program, use FNR instead, which starts again at 1 in each file; otherwise one damaged file shifts the count for every file after it. The same position trick answers other questions about reads too: printing length($0) for every sequence line, then sort -n | uniq -c, shows how many reads there are of each length, which becomes useful once reads have been trimmed and no longer all have the same length.
Reading the output
The file holds 9,000 reads, the same number Looking inside files found, and 45.3% of their bases are G or C. With 9,000 × 100 = 900,000 bases in all, that is about 408,000 G or C bases. The four lines after it are the first two reads in FASTA form; compare the first with the FASTQ record in Looking inside files.
GC content, the share of G and C bases, is a property of each genome; the human genome is about 41% GC. A sample whose GC content differs clearly from the others, or shows two humps instead of one in a quality-control plot, can point to contamination with another organism.
gsub(/[GC]/, "") removes every G and C and returns how many it removed, a neat way to count them.
awk was created at Bell Labs in 1977 by Alfred Aho, Peter Weinberger and Brian Kernighan, who described it in The AWK Programming Language (Addison-Wesley, 1988; second edition 2023). sed was written there a few years earlier by Lee McMahon. Both are defined in the POSIX standard. Several versions are in use: GNU awk (gawk) on many Linux systems, mawk as the default on Ubuntu, and the original 'one true awk' on macOS. The core language is shared, but extras such as gawk's gensub function are not. If a script depends on such a feature, say which awk it needs. awk --version identifies gawk and the macOS awk; mawk uses awk -W version instead.
sed and tr: change text
sed is the "stream editor": it reads text line by line and applies an edit to each line as it passes. The most used edit is substitution, written s/old/new/, which replaces the first match of old on each line with new; adding g after the last slash replaces every match on the line. tr ("translate") works on single characters: it swaps one set of characters for another, or with -d deletes them.
The old part of a substitution is a regular expression, with the same special characters you met with grep. That gives you control over where a change happens. s/^chrP/chr1/ changes chrP only at the start of a line, which in a GTF file is column 1, the chromosome; it cannot touch text that happens to contain chrP further along. Anchoring a pattern like this is the easiest way to make an edit change only what you meant.
tr has two classic uses with sequences. Many reference genomes are soft-masked: repeated regions are written in lower-case letters, so tr 'acgt' 'ACGT' turns every base into a capital when a tool needs them all the same. And the reverse complement, the sequence of the opposite strand read in its own direction, takes two short steps: rev reverses each line, and tr 'ACGT' 'TGCA' swaps each base for its partner. Both rev and tr are available on Linux and macOS.
sed 's/chrP/chr1/' data/practice_genes.gtf | head -n 1 | cut -c 1-60 # rename a chromosome grep -v "^>" data/practice_genome.fa | tr -d '\n' | wc -c # genome length
What this does
- The first line replaces chrP with chr1 on every line, keeps the first line of the result, and with
cut -c 1-60keeps its first 60 characters so it fits on the screen.cut -cselects characters, wherecut -fselects tab-separated fields. - The second line removes the FASTA header with
grep -v, deletes every newline character withtr -d '\n'so the 1,000 sequence lines become one long line, and counts its characters withwc -c.
Reading the output
The first line now starts with chr1 and stops part-way through column 9, where the 60 characters run out. The original file is unchanged. The second result is 60,000: the practice genome is 60,000 bases long.
Renaming chromosomes is a real task. Different groups name the same chromosome differently, for example 1 in Ensembl files and chr1 in UCSC files. If the genome and the annotation disagree, a tool that matches them up finds no genes at all, without always telling you why.
When you do need to change a file, sed -i is one of the few commands that behaves differently on Linux and macOS. GNU sed on Linux accepts sed -i 's/chrP/chr1/' file. The sed on macOS expects a backup ending straight after -i, and an empty one has to be written as sed -i '' 's/chrP/chr1/' file. A script written for one will fail or behave oddly on the other. Writing to a new file with >, as the key point below recommends, works the same everywhere.
Worked example · checking the genome length
tr -d '\n' joins the lines; wc -c counts the characters → 60000 ✓Step by step. Looking inside files counted 1,001 lines in the genome file. One is the header, so 1,001 − 1 = 1,000 lines hold sequence. Each full line holds 60 bases, so 1,000 × 60 = 60,000 bases, the same as the command found.
What the answer means. Two independent routes give the same number, so you can trust it. The tr step is not optional: every line ends with an invisible newline character, and wc -c counts it. Without tr the count would be 1,000 too high, 61,000, one extra character per sequence line.
Key point
sed prints a changed copy; the file itself is untouched unless you add -i. Never use -i on raw data.
The safe habit is the same for every tool that changes text: send the changed copy to a new file with >, look at it with head, and keep the original. If the edit was wrong, you fix the command and run it again. With -i on the only copy of a file, a mistaken pattern, such as one that also changes chrP inside a gene name, cannot be undone.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. In awk, what is $4 ?
2. A gene runs from 101 to 200. How long is it?
3. Which lines does NR % 4 == 2 select in a FASTQ file?
Up next
Lesson 6
Variables, loops and scripts