Lesson 2 of 10
Module 3 · The command line
A tidy project folder is the first step of a reproducible analysis. This lesson builds one, and shows how wildcards let one command act on many files.
Build a project
A common layout for an analysis project keeps three kinds of file apart: data (the inputs, never changed), results (everything the analysis produces) and scripts (the commands that turn one into the other). With that split, you can delete and regenerate the results at any time, and anyone reading the project knows where to look.
In The terminal and where to run it you made the folder ~/linux-course and listed the seven course files in it; this lesson organises them. The split matters most when something goes wrong. If a results file looks odd, you can rerun the scripts on the untouched data and compare. If the raw data had been edited in place, there would be nothing clean to go back to. A short plain-text note in the project folder, often called README.txt, saying where the data came from and when, is also worth writing on day one: months later it is the first thing you, or a colleague, will look for.
The braces make three folders in one go. -p creates parent folders as needed and does not complain if they already exist.
cd ~/linux-course
mkdir -p project/{data,results,scripts}
cp *.fastq.gz practice_genome.fa practice_genes.gtf practice_samples.csv project/data/
cd project
ls dataWhat this does
cd ~/linux-coursemoves you into the course folder from The terminal and where to run it, where the course files are.mkdir -p project/{data,results,scripts}creates the project. Beforemkdirruns, the shell rewrites the part in braces as three separate paths, project/data, project/results and project/scripts; this is called brace expansion. Because project does not exist yet,-pcreates it first, then the three folders inside it. Note that there are no spaces inside the braces; a space would split them into separate arguments.cpcopies files. When it is given several files followed by a folder, it copies all of them into that folder; the folder must come last. Here*.fastq.gzmatches the four read files, so seven files in total are copied into project/data/. The originals stay where they were.cd projectmoves you into the new project folder. This is a relative path: it works because you are in ~/linux-course.ls datalists what is in the data folder, to check that the copy worked.
Reading the output
The seven names are the files now in project/data, listed in alphabetical order: the four read files matched by the wildcard, then the three files named one by one. They are the same seven files you saw in The terminal and where to run it, which confirms that nothing was missed. The results and scripts folders exist but are still empty.
Checking after each step like this is a good habit. mkdir and cp print nothing when they succeed, so the only way to be sure the files arrived is to look. A common slip is to forget the destination folder at the end of cp: the last file named is then taken as the destination, so a short list of two files would overwrite the second with the first. Without -p, mkdir project/data would also fail with an error if project did not exist yet, because plain mkdir makes only the last folder in the path.
Wildcard matching is defined in the POSIX standard, in the shell's pattern-matching rules, so * and ? behave the same in every Unix-like shell. Brace expansion is different: it is an extension found in bash and zsh, described in the Bash Reference Manual, and not part of POSIX. A script run with plain sh may leave the braces untouched. On Ubuntu, sh is a smaller shell called dash, which does not expand them, so mkdir {a,b} there makes one folder with braces in its name. This is one reason the scripts in Variables, loops and scripts name bash on their first line. For the data, results and scripts layout, a widely cited guide is Wilson et al. (2017), 'Good enough practices in scientific computing', PLOS Computational Biology.
Wildcards
The shell expands wildcards into matching file names before the command runs. * matches any characters; ? matches exactly one.
A wildcard is a character that stands for other characters in a file name. "Any characters" includes none at all, so a * can match an empty stretch of the name too. The important part is the first sentence: it is the shell, not ls or cp, that looks at the files and replaces the pattern with the list of names. The command never sees the *; it receives the finished list. This is why wildcards work in the same way with every command.
ls data/*_R1.fastq.gz # every read-1 file ls data/ctrl_?_R?.fastq.gz # control samples, both reads
What this does
ls data/*_R1.fastq.gzlists every file in data whose name ends in _R1.fastq.gz. The*can stand for ctrl_1, treat_1, or anything else, so every read-1 file matches, whatever the sample is called.ls data/ctrl_?_R?.fastq.gzuses two?, each standing for exactly one character. The first matches the sample number, the second the read number. Only names that start with ctrl_ can match, so the treated files are left out.
A third kind of wildcard is the character set, written in square brackets. It matches exactly one character from the list inside, so data/*_R[12].fastq.gz matches names ending in _R1.fastq.gz or _R2.fastq.gz, and nothing else. It is stricter than ?, which would also accept _RX or _R9. Choosing the narrowest pattern that does the job is the safest habit, because a broad pattern can catch files you did not have in mind.
Reading the output
The two commands print one after the other with nothing between them, so it helps to split the output by hand. The first two lines come from the first command: the read-1 files of the control and the treated sample. The last two come from the second command: both reads of the control sample. ctrl_1_R1.fastq.gz appears twice because both patterns match it. Each name starts with data/ because the pattern did; ls prints names the way it was given them.
If a pattern matches nothing, bash passes it to the command unchanged, so ls reports that no file with that literal name exists. zsh, the default shell on newer Macs, stops with a "no matches found" message instead. Either message means the same thing: check the pattern.
The order of the names matters later, when a loop works through files one by one. Wildcards return names sorted as text, character by character, not as numbers. With samples called sample_1 to sample_10, the name sample_10 sorts before sample_2, because the character 1 comes before 2. Writing numbers with a leading zero, sample_01, sample_02 and so on, keeps text order and number order the same.
Go deeper: hidden files, and braces versus wildcards
Names that start with a dot, such as .bashrc, are hidden files, usually holding settings. ls does not show them and * does not match them, so a wildcard can quietly leave them out. ls -a lists everything, including the two special entries . and .. from The terminal and where to run it.
Braces and wildcards look alike but work differently. A wildcard is matched against files that already exist, so it can only name files that are there. Braces simply generate text, whether or not any such file exists. That is why mkdir -p project/{data,results,scripts} can create new folders, while ls data/*.txt in this project finds nothing, because no file ends in .txt. Bash expands braces first and wildcards last, so the two can be combined in one pattern.
Copy, move, remove
These three commands manage files. Each takes the file to act on first, and for cp and mv the destination second.
The difference between copying and moving is worth keeping in mind. A copy leaves the original in place and uses as much disk space again; a move within the same disk only changes where the file is listed, so it is instant even for a large file and uses no extra space. For raw data the rule is simple: copy it into a project, never move the only copy.
cp data/practice_samples.csv results/samples_copy.csv # copy mv results/samples_copy.csv results/samples_backup.csv # move, or rename ls results rm results/samples_backup.csv # remove: no undo ls results | wc -l # count what is left
What this does
cpcopies data/practice_samples.csv to a new file, results/samples_copy.csv. You now have two identical files in two folders.mvmoves a file. When the source and destination are in the same folder, moving is the same as renaming, so samples_copy.csv becomes samples_backup.csv. There is no separate rename command for this.ls resultslists the results folder, which now holds only the renamed file.rmremoves the file. It asks no question and prints nothing.ls results | wc -lcounts what is left. The|, called a pipe, sends the output of the command on its left into the command on its right instead of to the screen.wccounts words, lines and characters; with-lit counts only lines. Sincelsprints one name per line here, the number of lines is the number of files.
A caution: cp and mv overwrite an existing file at the destination without asking. If samples_backup.csv had already existed, its old contents would be gone.
A few options change this behaviour, and they work the same on Linux and macOS. -i ("interactive") makes cp, mv and rm ask before they overwrite or delete; answer y to go ahead. Folders need one more option: cp -r copies a folder with everything inside it, and rm -r removes a folder and everything inside it, which makes it the most dangerous command in this lesson. rmdir removes a folder only if it is empty, so it cannot take files with it by accident. If the destination of mv or cp is an existing folder, the file goes inside it and keeps its name.
Reading the output
Only two of the five lines print anything. samples_backup.csv comes from the first ls results: after the copy and the rename, it is the one file in the folder. The 0 comes from the last line: after rm, the folder is empty, so ls printed no lines and wc -l counted zero. The cp, mv and rm lines printed nothing, because they succeeded.
On a Mac, wc pads its number with spaces on the left, so the last line may look indented; the count is the same. Counting files this way is a quick check before and after any step that should create or remove a known number of them: four read files in, four results out.
Worked example · what a wildcard expands to
so
ls data/*_R1.fastq.gz runs as ls data/ctrl_1_R1.fastq.gz data/treat_1_R1.fastq.gzCheck a wildcard with ls (or echo) before you use it with rm or mv.
Step by step. The shell takes the pattern apart. The fixed part data/ says which folder to look in. Then it goes through every name in that folder and asks whether it ends in _R1.fastq.gz, with anything at all in front. ctrl_1_R1.fastq.gz: yes, with ctrl_1 in front. ctrl_1_R2.fastq.gz: no, it ends in _R2. practice_genes.gtf: no. And so on through all seven. Two names pass, and they replace the pattern on the command line, in alphabetical order.
What the answer means. Any command given the pattern gets exactly the same list. ls only prints it, which is harmless; rm would delete every file on it. echo is a command that prints its arguments, so it shows the expanded list without doing anything to the files. Checking first costs a second and shows you exactly what the destructive command would touch.
Key point
rm deletes permanently: there is no recycle bin. Keep raw data in its own folder, never edit it, and keep a copy somewhere else.
A second example of how quickly this goes wrong: a stray space typed between a * and the rest of a pattern turns one pattern into two arguments. The first, a lone *, matches every file in the folder. Given to rm, that deletes everything, and the second argument then only produces an error. This is the reason for the project layout above: if data holds only raw files that you never change, and a copy is kept elsewhere, a mistake in results costs you a rerun, not your data.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. Which files does *.gtf match?
2. How do you rename a.txt to b.txt ?
3. What does ls data/s?.txt match?
Up next
Lesson 3
Looking inside files