Lesson 6 of 10
Module 3 · The command line
Typing the same command for every sample does not scale to 100 samples. Variables and loops do the repetition; a script saves it so it can be rerun and shared.
Variables and command substitution
Set a variable with name=value (no spaces around =) and use it with $name. $( … ) runs a command and inserts its output.
A variable is a name that stands for a value, so you can write a command once and change only the value. The rule about spaces is strict because the shell splits every command at spaces. If you type sample = ctrl_1, the shell sees three words and tries to run a program called sample, with = and ctrl_1 as its arguments; it reports that the command was not found.
Two kinds of brackets after a dollar sign do different jobs. $( … ), command substitution, runs the command inside and puts what it prints in its place. $(( … )), arithmetic expansion, works out a sum. Shell arithmetic uses whole numbers only, so any remainder of a division is dropped.
In Looking inside files you counted the reads of one file with $(( $(gzip -dc … | wc -l) / 4 )), typing the file name in full. The example below does the same count, but takes the sample name from a variable, which is the first step towards a command that works for any sample.
Choose short, lower-case names that say what the value is, such as sample, reads or outdir. The shell already uses upper-case names for its own settings: HOME holds your home folder and PATH the list of folders searched for programs. If you reuse one of those names by accident, for example by setting PATH to a folder of results, commands stop being found until you open a new terminal. Lower-case names avoid that clash.
sample=ctrl_1
echo "Processing $sample"
echo "Reads: $(( $(gzip -dc data/${sample}_R1.fastq.gz | wc -l) / 4 ))"What this does
- Line 1 stores the text ctrl_1 in the variable
sample. Nothing is printed. - Line 2 prints a message. Inside double quotes the shell still replaces
$samplewith its value; inside single quotes it would print the characters $sample literally. - Line 3 builds the file name data/ctrl_1_R1.fastq.gz from the variable, counts the lines of the decompressed file (36,000), divides by 4 and prints the result as part of the message.
Reading the output
The variable was filled in both times: the message names ctrl_1, and the read count, 9,000, is the same number Looking inside files found by typing the file name in full. Change line 1 to sample=treat_1 and the same two lines would report the other sample.
A variable lives only in the terminal where you set it. Close the window, or open a second one, and $sample is empty again. When a command behaves oddly, echo "$sample" is the quickest check of what the shell will actually use. Scripts, later in this lesson, solve the forgetting problem: the values are written in the file, so they are set again every time it runs.
The braces in ${sample}_R1 mark where the name ends; without them the shell would look for a variable called sample_R1.
Letters, digits and underscores can all be part of a variable name, so the shell keeps reading past the underscore. A dot or a slash cannot, so the name ends there. Braces make the end explicit either way, and they are a good habit. The danger of getting this wrong is quiet: a variable that was never set, such as sample_R1, is replaced by nothing at all, so the shell would look for a file called data/.fastq.gz and fail with a confusing message. The -u in set -euo pipefail, below, turns that silent mistake into a clear error.
A loop over samples
A loop repeats the same commands once for each item in a list. In a for loop the variable after for takes each value in turn, and everything between do and done, the body of the loop, runs once per value.
Loops matter because omics projects rarely have one sample. A study may have six, sixty or six hundred FASTQ files, and every one needs the same steps. Typing a command per file is slow, and it invites the worst kind of mistake: one file processed with a slightly different command from the others. A loop runs exactly the same commands on every file, so any difference in the results comes from the data.
for f in data/*_R1.fastq.gz; do name=$(basename "$f" _R1.fastq.gz) # strip the folder and the ending reads=$(( $(gzip -dc "$f" | wc -l) / 4 )) echo -e "$name\t$reads" done
What this does
data/*_R1.fastq.gzis a wildcard pattern:*stands for any run of characters. Before the loop starts, the shell replaces the pattern with every matching file name, in alphabetical order. Here that is two files.basename "$f" _R1.fastq.gzremoves the folder part (data/) and the given ending, leaving the sample name.$( … )stores it inname.- The next line counts the reads of that file, as before.
echo -eprints the name and count. The-etellsechoto turn\tinto a real tab, so the output is a two-column, tab-separated table.
Reading the output
One line per file: ctrl_1 has 9,000 reads and treat_1 has 10,000. These are the two samples in this module's data/ folder. Add more files that match the pattern and the same four lines handle them, with nothing retyped.
One trap to know: if no file matches the pattern, the shell leaves the pattern as it is, and the loop runs once with f set to the literal text data/*_R1.fastq.gz. The error message then names a file with a star in it. That usually means you are in the wrong folder.
The list does not have to come from a wildcard. You can type the values yourself, as in for s in ctrl_1 treat_1; do … done, and build each file name inside the body with data/${s}_R1.fastq.gz. This is useful when you want only some of the samples, or when the names come from a sample sheet rather than from the folder.
A safe habit before a loop that changes or deletes files is a dry run: put echo in front of the command inside the body. The loop then prints each command it would run instead of running it. Read the list, check that every file name looks right, then remove the echo and run it for real.
echo -e prints.Everything this lesson uses, from variables and $( … ) to for loops and the options of set, is documented in the Bash Reference Manual (Free Software Foundation); inside bash, help set gives a short summary. The case for saving analyses as scripts, rather than as commands typed once and forgotten, is made in Wilson et al. (2014), 'Best practices for scientific computing', PLOS Biology 12:e1001745. A free checker, ShellCheck (shellcheck.net), points out common mistakes such as unquoted variables. When you publish, share the script itself, not a description of it, and state the bash version it was run with.
Save it as a script
Create scripts/count_reads.sh in a text editor (nano scripts/count_reads.sh works in any terminal; save with Ctrl+O, exit with Ctrl+X).
A script is a plain text file of commands that the shell runs from top to bottom, exactly as if you had typed them. Saving work as a script means you can rerun it next month, run it on new data, show a colleague exactly what you did and quote it in your methods. The ending .sh is a convention that tells people it is a shell script; the computer does not need it.
The file goes in the scripts folder of the project layout from Files, folders and wildcards, next to data and results. Keeping scripts apart from data means you can copy, share or put them under version control without dragging large files along.
#!/usr/bin/env bash # count_reads.sh: print the number of reads in each FASTQ file given set -euo pipefail for f in "$@"; do reads=$(( $(gzip -dc "$f" | wc -l) / 4 )) echo -e "$(basename "$f")\t$reads" done
The first line, the shebang, says which program runs the script. "$@" is every argument you pass in. set -euo pipefail makes the script stop at the first error instead of carrying on with bad results.
What this does, line by line
#!/usr/bin/env bashmeans "run this file with bash, wherever it is installed on this computer". The program env looks it up, which makes the script work on more systems than a fixed path would.- The second line is a comment saying what the script is for. Anyone opening the file, including you in six months, reads it first.
set -euo pipefailswitches on three safety settings.-estops the script when a command fails.-ustops it when it uses a variable that was never set.-o pipefailmakes a pipeline count as failed if any command in it fails; normally only the last command's success is checked, so a failedgzipfeeding a workingwcwould go unnoticed.for f in "$@"loops over whatever file names you give the script when you run it, instead of a fixed pattern. The same script then works on any set of files.- The body counts the reads and prints the file's name, without its folder, next to the count.
chmod +x scripts/count_reads.sh # make it executable ./scripts/count_reads.sh data/*_R1.fastq.gz ./scripts/count_reads.sh data/missing.fastq.gz; echo "exit code: $?"
What this does
chmod +x("change mode") adds permission to execute the file, that is, to run it as a program. It only needs doing once../scripts/count_reads.shruns the script. The leading./means "starting from this folder"; for safety, the shell does not look for programs in the current folder unless you say so.- The third line runs the script on a file that does not exist. The semicolon runs the next command afterwards whatever happened, and
echo "exit code: $?"reports how the script ended.
Reading the output
The first two lines are the read counts, now with full file names because this script does not strip the ending. Then gzip complains that the file is missing, and the script stops with exit code 1. Notice what is not there: no line with a file name and a count of 0. Without set -euo pipefail the script could have printed such a line, and a count of 0 in a results table looks like real data.
You can also run a script with bash scripts/count_reads.sh data/*_R1.fastq.gz. This needs no execute permission, because you name the program yourself, and the shebang line is then read as an ordinary comment. Both ways are common. The chmod +x form is tidier for scripts you run often, and it is what lets the shebang choose the program.
When you change a script, rerun it on data whose answer you already know, such as these two files with 9,000 and 10,000 reads. If the familiar numbers come back, the change did not break anything. This small check, repeated after every edit, catches most mistakes before they reach real results.
Worked example · reading permissions
owner: rwx (read, write, execute) · group: r-x · everyone else: r-x
Step by step. The ten characters split into a first character and three groups of three. The first character is - for an ordinary file (it would be d for a folder). Then come the owner's rights, the group's rights and everyone else's. In each group, r means read, w write and x execute, and a - means that right is missing. Before chmod +x the file looked like the FASTQ file in Looking inside files, -rw-r--r--; the command added an x to each group.
What the answer means. You can read, change and run the script; other people on the same computer can read and run it but not change it.
$? holds the last command's exit code: 0 means success, anything else an error. Pipelines and job schedulers rely on it.
Some tools use non-zero codes for answers rather than errors. grep, for instance, ends with code 1 when it finds no matching lines, so in a script with set -e a search that finds nothing stops the script. That is usually what you want, but it is worth knowing when a script stops unexpectedly.
Go deeper: what exit codes above 128 mean
Exit codes are not only 0 or 1. By convention, a program stopped by a signal, a message from the operating system telling it to end, finishes with 128 plus the signal's number. Pressing Ctrl+C sends signal 2, so an interrupted command usually ends with code 130. Signal 9 stops a program at once and gives code 137.
You will meet 137 on servers. When a job uses more memory than it is allowed, the system often kills it with signal 9, and that number in a log may be the only clue. Code 127 means something else: the shell could not find the command at all, usually because of a typing mistake or a tool from an environment that is not active. Reading $? after a failure, or the exit code a scheduler reports, often tells you which of these happened before you open any log.
Key point
Put double quotes around variables ("$f"). Without them, a file name with a space would be split into two arguments.
For example, if your files sit in a folder called My Data, an unquoted variable hands the command two words, My and Data/…, and it looks for two files that do not exist. The quoted version passes the whole path as one argument. Better still, use names without spaces for folders and files you analyse: underscores or hyphens work everywhere.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. What does basename data/s1_R1.fastq.gz _R1.fastq.gz print?
2. What does set -e do in a script?
3. Why does sample = ctrl_1 fail?
Up next
Lesson 7
Installing software with conda