Lesson 4 of 10
Module 3 · The command line
The pipe, | , sends one command's output into the next. Small tools chained together answer real questions about your files in one line.
grep: find lines
grep reads a file line by line and prints every line that contains a pattern. The pattern is a regular expression: a short code for "text that looks like this". Most letters stand for themselves, and a few characters have special meanings: ^ means the start of a line, $ the end of a line, . any single character, and [GC] any one of the characters inside the brackets.
In Looking inside files you read the GTF file by eye, line by line. That works for 27 lines; it does not work for the hundreds of thousands of lines in a real genome annotation. grep lets you ask the file a question instead, such as "which lines are genes?", and get only the answer. Two points about patterns trip up most beginners. grep is case-sensitive, so Exon does not match exon. And the special characters are special even when you did not mean them to be: a dot in a pattern such as PRG001.1 also matches PRG001X1. grep -F treats the whole pattern as plain text, with no special characters, which is the safer choice when you are searching for an ID.
grep -c "^>" data/practice_genome.fa # how many sequences? (^ = start of line) grep -w "gene" data/practice_genes.gtf | cut -f 1,4,5,7 | head -n 3
What this does
grep -c "^>"counts the lines that begin with >. In a FASTA file every sequence starts with exactly one such header line, so the count is the number of sequences. The quotes matter: without them the shell would read > as "save the output to a file", the redirection you meet at the end of this lesson.grep -w "gene"keeps the lines where gene appears as a whole word. Every line of the GTF file contains gene_id, but the underscore counts as part of a word, so gene_id does not match. Only the lines whose feature type, column 3, is gene are kept. Without-w, all 27 lines would match.cut -f 1,4,5,7keeps columns 1, 4, 5 and 7, which Looking inside files named as chromosome, start, end and strand.cutsplits on tabs unless you tell it otherwise.- For a file with another separator,
-dsets it. The sample sheet is comma-separated, socut -d , -f 2 data/practice_samples.csvkeeps its second column, the condition. Ifcutseems to return whole lines unchanged, the separator is usually not the one it expected. head -n 3keeps the first three of those lines.
Reading the output
The 1 says the practice genome holds one sequence, the chromosome chrP. The three lines after it are the first three genes: where each starts and ends, and its strand. The second gene, prB, is on the minus strand.
The second command is also your first longer pipeline: three tools joined by pipes, each doing one small job. grep chooses the rows, cut chooses the columns and head limits the length. The lines flow from one tool to the next without being saved anywhere in between, which is why pipelines stay fast on large files. When a pipeline gives a result you do not expect, take it apart: run the first tool alone with | head after it, then add the next tool, and see at which step the lines stop looking right.
-c counts matching lines; -w matches whole words only; cut -f keeps chosen tab-separated columns. Other useful options: -v (lines that do not match) and -i (ignore case).
For example, -v with the pattern exon would keep every line that is not an exon, and -i would let chrp match chrP. The usual mistake with grep is a pattern that matches more than you meant: gene without -w matches gene_id, and chr1 also matches chr10 to chr19. When a count looks too large, print a few of the matching lines and look at them.
grep was written by Ken Thompson for Unix at Bell Labs in the early 1970s. Its name comes from the editor command g/re/p: globally search for a regular expression and print. The pattern rules are set out in the POSIX standard, which defines basic regular expressions, used by grep by default, and extended ones, switched on with grep -E. The GNU grep manual describes the version on most Linux systems. Versions differ in their extras: the -P option for Perl-style patterns is a GNU feature that other versions may lack. For the adapter string AGATCGGAAGAGC, the source to quote is Illumina's own adapter sequences document, which lists the adapters for each library kit.
sort and uniq -c: count categories
Counting how many lines fall in each category is one of the most common questions you will ask of a file: how many features of each type, how many reads per sample, how many variants per chromosome. Three tools answer it together. cut picks out the column that holds the category, sort puts identical values next to each other, and uniq -c collapses each run of identical lines into one line with a count in front.
cut -f 3 data/practice_genes.gtf | sort | uniq -c # feature types cut -f 7 data/practice_genes.gtf | sort | uniq -c # strands grep -w "exon" data/practice_genes.gtf | grep -o 'gene_name "[^"]*"' \ | sort | uniq -c | sort -k1,1nr | head -n 4 # genes with most exons
uniq -c can only merge identical lines that sit next to each other, which is why sort comes first. Schematic, with five lines instead of the file's 27.What this does
- The first line takes column 3, the feature type, sorts it and counts each value.
- The second line does the same for column 7, the strand.
- The third command is longer.
grep -w "exon"keeps only the exon lines.grep -othen prints only the part of each line that matches the pattern, not the whole line. The pattern gene_name "[^"]*" means: the text gene_name, a space, a double quote, then any run of characters that are not double quotes ([^"] means "any character except a quote", and * means "any number of them"), then the closing quote. The pattern is wrapped in single quotes because it contains double quotes; the shell passes text inside single quotes on unchanged. sort | uniq -ccounts how many exon lines each gene name has.sort -k1,1nrranks those lines by the count, largest first, andhead -n 4keeps the top four.
Reading the output
- 15 exon lines and 12 gene lines: 15 + 12 = 27, the line count from Looking inside files.
- 19 features on the plus strand and 8 on the minus strand, again 27 in all. These count lines, not genes: each gene contributes its gene line and its exon lines.
- prC, prG and prJ have two exons each. The other nine genes have one each: 3 × 2 + 9 × 1 = 15 exons, which matches the first count.
- When counts tie,
sortfalls back to comparing the whole lines, so among the nine one-exon genes the alphabetically first, prA, appears fourth.
The counts are printed with spaces in front. uniq -c pads them so that numbers of different lengths line up in a column.
Counting like this is a sanity check as much as a result. If you know a file should hold 12 genes and the count says 11, something was lost before you started. The same three tools answer the same kind of question in other formats. In a VCF file, for example, the header lines start with #, so grep -v "^#" removes them, cut -f 1 keeps the chromosome column, and sort | uniq -c then gives the number of variants on each chromosome. Only the column number and the lines to skip change from one format to the next.
uniq -c counts repeated adjacent lines, so sort must come first. sort -k1,1nr sorts by the first column, numerically, in reverse. A backslash at the end of a line continues the command on the next. Three genes have two exons: they contain an intron.
An intron is the stretch of a gene between two exons. It is copied into RNA and then cut out, and the exons on either side are joined; this is called splicing. A gene with two exons therefore has one intron. You will see reads that jump across an intron when you align RNA-seq reads in Module 5.
Go deeper: why sort order can change between computers
sort does not simply compare characters by their codes. It follows your locale, the language and region settings of your system, so the same file can sort in a different order on two computers. One may put capital letters before small ones; another may mix them, or ignore punctuation. For counting with uniq -c the counts stay the same, but the order of lines can change, and two outputs then look different when they are not.
Setting the locale for one command removes the doubt. LC_ALL=C sort file.txt sorts by plain character codes, the same everywhere, and is often faster on big files. A second trap is numbers. Without -n, sort compares text, so 10 comes before 9, because the character 1 comes before 9. Use -n for numbers, or add n to -k as in this lesson.
Redirection: save the output
Every command has a standard place to print its results, called standard output, and normally that is the screen. The > character sends standard output into a file instead. Nothing is printed, the file is created if it does not exist, and the folder it goes into must already exist.
Saving output matters for two reasons. A result printed to the screen is gone once it scrolls away, while a file can be opened again, checked and used as the input of the next step. And writing each step's result to results/, the folder you made in Files, folders and wildcards, keeps the inputs and outputs apart, as that lesson set out.
Error messages travel separately, on a second channel called standard error. > does not catch them, which is helpful: if a command fails while its output goes into a file, you still see the error on the screen. To save the errors as well, for example in a log, 2> sends standard error into a file of its own, as in command > out.txt 2> errors.txt. Many bioinformatics tools write their progress messages to standard error for exactly this reason, so that they never mix with the data.
gzip -dc data/ctrl_1_R1.fastq.gz | grep -c "AGATCGGAAGAGC" # reads containing adapter grep -w "gene" data/practice_genes.gtf | cut -f 1,4,5,7,9 > results/genes.tsv wc -l results/genes.tsv
What this does
- The first line decompresses the reads and counts the lines that contain AGATCGGAAGAGC, the first 13 bases of the standard Illumina adapter.
grep -ccounts lines, not reads, but in practice only the sequence lines contain this string of letters, so the count is a number of reads. - The second line keeps the gene lines, keeps five columns (chromosome, start, end, strand and the attributes) and writes them to results/genes.tsv. The ending .tsv means "tab-separated values". Nothing appears on screen, because the output went into the file.
wc -lchecks the new file.
Reading the output
431 reads contain the adapter start. 12 results/genes.tsv means the saved table has 12 lines, one per gene, as expected from the 12 gene lines counted above.
Worked example · reading a pipeline left to right
| grep -c "AGATCGGAAGAGC" → count lines with the Illumina adapter start
431 of 9,000 reads = 4.8% contain adapter
Step by step. The file holds 9,000 reads (Looking inside files). The share with adapter is 431 ÷ 9,000 = 0.0479. Multiply by 100 to get a percentage: 4.79%, which rounds to 4.8%.
What the answer means. About one read in twenty runs into adapter. This is a slight undercount: a read that ends with only the first few adapter bases, fewer than 13, does not contain the whole pattern and is not counted. Trimming tools look for these partial matches too. A share of a few per cent is common, and trimming removes it; if it were much higher, it would suggest that many fragments in the library were very short.
Adapter appears when the DNA fragment is shorter than the read, so the sequencer reads on into the adapter. Trimming and filtering, in Module 5, removes it.
Key point
> overwrites a file without asking; >> adds to the end. Double-check the name before you press Enter.
A second trap: never send a command's output into the file it is reading from. The shell empties the output file before the command starts, so sorting a file "into itself" leaves you with an empty file. Write to a new name, check it, and then rename it if you need to.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. Why sort before uniq -c ?
2. What does grep -v "^#" keep?
3. What is the difference between > and >> ?
Up next
Lesson 5
Text processing with awk and sed