LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 73 lessons

Lesson 2 of 10

Module 3 · The command line

Files, folders and wildcards

Files, folders and wildcards

Files, folders and wildcards

A tidy project folder is the first step of a reproducible analysis. This lesson builds one, and shows how wildcards let one command act on many files.

⏱ 11 min readModule 3 · The command lineFree
Lesson at a glance3 steps · about 11 min
  1. Build a project
  2. Wildcards
  3. Copy, move, remove

Build a project

A common layout for an analysis project keeps three kinds of file apart: data (the inputs, never changed), results (everything the analysis produces) and scripts (the commands that turn one into the other). With that split, you can delete and regenerate the results at any time, and anyone reading the project knows where to look.

In The terminal and where to run it you made the folder ~/linux-course and listed the seven course files in it; this lesson organises them. The split matters most when something goes wrong. If a results file looks odd, you can rerun the scripts on the untouched data and compare. If the raw data had been edited in place, there would be nothing clean to go back to. A short plain-text note in the project folder, often called README.txt, saying where the data came from and when, is also worth writing on day one: months later it is the first thing you, or a colleague, will look for.

The braces make three folders in one go. -p creates parent folders as needed and does not complain if they already exist.

Terminal
cd ~/linux-course
mkdir -p project/{data,results,scripts}
cp *.fastq.gz practice_genome.fa practice_genes.gtf practice_samples.csv project/data/
cd project
ls data

What this does

  1. cd ~/linux-course moves you into the course folder from The terminal and where to run it, where the course files are.
  2. mkdir -p project/{data,results,scripts} creates the project. Before mkdir runs, the shell rewrites the part in braces as three separate paths, project/data, project/results and project/scripts; this is called brace expansion. Because project does not exist yet, -p creates it first, then the three folders inside it. Note that there are no spaces inside the braces; a space would split them into separate arguments.
  3. cp copies files. When it is given several files followed by a folder, it copies all of them into that folder; the folder must come last. Here *.fastq.gz matches the four read files, so seven files in total are copied into project/data/. The originals stay where they were.
  4. cd project moves you into the new project folder. This is a relative path: it works because you are in ~/linux-course.
  5. ls data lists what is in the data folder, to check that the copy worked.
ctrl_1_R1.fastq.gz ctrl_1_R2.fastq.gz practice_genes.gtf practice_genome.fa practice_samples.csv treat_1_R1.fastq.gz treat_1_R2.fastq.gz

Reading the output

The seven names are the files now in project/data, listed in alphabetical order: the four read files matched by the wildcard, then the three files named one by one. They are the same seven files you saw in The terminal and where to run it, which confirms that nothing was missed. The results and scripts folders exist but are still empty.

Checking after each step like this is a good habit. mkdir and cp print nothing when they succeed, so the only way to be sure the files arrived is to look. A common slip is to forget the destination folder at the end of cp: the last file named is then taken as the destination, so a short list of two files would overwrite the second with the first. Without -p, mkdir project/data would also fail with an error if project did not exist yet, because plain mkdir makes only the last folder in the path.

Files, folders and wildcards · LSMA free course A folder called project contains three folders: data, results and scripts. The data folder holds the seven course files; results and scripts are empty. THE PROJECT AFTER THE COPY project/data/results/scripts/ emptyempty ctrl_1_R1.fastq.gz ctrl_1_R2.fastq.gz practice_genes.gtf practice_genome.fa practice_samples.csv treat_1_R1.fastq.gz treat_1_R2.fastq.gz
One mkdir made the three folders inside project, and one cp filled data with copies of the seven course files. Schematic.

Wildcard matching is defined in the POSIX standard, in the shell's pattern-matching rules, so * and ? behave the same in every Unix-like shell. Brace expansion is different: it is an extension found in bash and zsh, described in the Bash Reference Manual, and not part of POSIX. A script run with plain sh may leave the braces untouched. On Ubuntu, sh is a smaller shell called dash, which does not expand them, so mkdir {a,b} there makes one folder with braces in its name. This is one reason the scripts in Variables, loops and scripts name bash on their first line. For the data, results and scripts layout, a widely cited guide is Wilson et al. (2017), 'Good enough practices in scientific computing', PLOS Computational Biology.

Wildcards

The shell expands wildcards into matching file names before the command runs. * matches any characters; ? matches exactly one.

A wildcard is a character that stands for other characters in a file name. "Any characters" includes none at all, so a * can match an empty stretch of the name too. The important part is the first sentence: it is the shell, not ls or cp, that looks at the files and replaces the pattern with the list of names. The command never sees the *; it receives the finished list. This is why wildcards work in the same way with every command.

Terminal
ls data/*_R1.fastq.gz          # every read-1 file
ls data/ctrl_?_R?.fastq.gz     # control samples, both reads

What this does

  • ls data/*_R1.fastq.gz lists every file in data whose name ends in _R1.fastq.gz. The * can stand for ctrl_1, treat_1, or anything else, so every read-1 file matches, whatever the sample is called.
  • ls data/ctrl_?_R?.fastq.gz uses two ?, each standing for exactly one character. The first matches the sample number, the second the read number. Only names that start with ctrl_ can match, so the treated files are left out.

A third kind of wildcard is the character set, written in square brackets. It matches exactly one character from the list inside, so data/*_R[12].fastq.gz matches names ending in _R1.fastq.gz or _R2.fastq.gz, and nothing else. It is stricter than ?, which would also accept _RX or _R9. Choosing the narrowest pattern that does the job is the safest habit, because a broad pattern can catch files you did not have in mind.

data/ctrl_1_R1.fastq.gz data/treat_1_R1.fastq.gz data/ctrl_1_R1.fastq.gz data/ctrl_1_R2.fastq.gz

Reading the output

The two commands print one after the other with nothing between them, so it helps to split the output by hand. The first two lines come from the first command: the read-1 files of the control and the treated sample. The last two come from the second command: both reads of the control sample. ctrl_1_R1.fastq.gz appears twice because both patterns match it. Each name starts with data/ because the pattern did; ls prints names the way it was given them.

If a pattern matches nothing, bash passes it to the command unchanged, so ls reports that no file with that literal name exists. zsh, the default shell on newer Macs, stops with a "no matches found" message instead. Either message means the same thing: check the pattern.

The order of the names matters later, when a loop works through files one by one. Wildcards return names sorted as text, character by character, not as numbers. With samples called sample_1 to sample_10, the name sample_10 sorts before sample_2, because the character 1 comes before 2. Writing numbers with a leading zero, sample_01, sample_02 and so on, keeps text order and number order the same.

Files, folders and wildcards · LSMA free course Three boxes from left to right. You type ls data/*_R1.fastq.gz. The shell compares the pattern with the file names in data. The ls command receives two names: data/ctrl_1_R1.fastq.gz and data/treat_1_R1.fastq.gz. 1 · YOU TYPE 2 · SHELL EXPANDS 3 · LS RECEIVES ls data/*_R1.fastq.gz * matches any characters; every name in data/ is checked data/ctrl_1_R1.fastq.gz data/treat_1_R1.fastq.gz ls never sees the *: the shell hands it the finished list of names.
The expansion happens before the command starts. This is why checking a pattern with ls first is safe: you see exactly the list another command would receive. Schematic.
Go deeper: hidden files, and braces versus wildcards

Names that start with a dot, such as .bashrc, are hidden files, usually holding settings. ls does not show them and * does not match them, so a wildcard can quietly leave them out. ls -a lists everything, including the two special entries . and .. from The terminal and where to run it.

Braces and wildcards look alike but work differently. A wildcard is matched against files that already exist, so it can only name files that are there. Braces simply generate text, whether or not any such file exists. That is why mkdir -p project/{data,results,scripts} can create new folders, while ls data/*.txt in this project finds nothing, because no file ends in .txt. Bash expands braces first and wildcards last, so the two can be combined in one pattern.

Copy, move, remove

These three commands manage files. Each takes the file to act on first, and for cp and mv the destination second.

The difference between copying and moving is worth keeping in mind. A copy leaves the original in place and uses as much disk space again; a move within the same disk only changes where the file is listed, so it is instant even for a large file and uses no extra space. For raw data the rule is simple: copy it into a project, never move the only copy.

Terminal
cp data/practice_samples.csv results/samples_copy.csv          # copy
mv results/samples_copy.csv results/samples_backup.csv        # move, or rename
ls results
rm results/samples_backup.csv                                  # remove: no undo
ls results | wc -l                                             # count what is left

What this does

  1. cp copies data/practice_samples.csv to a new file, results/samples_copy.csv. You now have two identical files in two folders.
  2. mv moves a file. When the source and destination are in the same folder, moving is the same as renaming, so samples_copy.csv becomes samples_backup.csv. There is no separate rename command for this.
  3. ls results lists the results folder, which now holds only the renamed file.
  4. rm removes the file. It asks no question and prints nothing.
  5. ls results | wc -l counts what is left. The |, called a pipe, sends the output of the command on its left into the command on its right instead of to the screen. wc counts words, lines and characters; with -l it counts only lines. Since ls prints one name per line here, the number of lines is the number of files.

A caution: cp and mv overwrite an existing file at the destination without asking. If samples_backup.csv had already existed, its old contents would be gone.

A few options change this behaviour, and they work the same on Linux and macOS. -i ("interactive") makes cp, mv and rm ask before they overwrite or delete; answer y to go ahead. Folders need one more option: cp -r copies a folder with everything inside it, and rm -r removes a folder and everything inside it, which makes it the most dangerous command in this lesson. rmdir removes a folder only if it is empty, so it cannot take files with it by accident. If the destination of mv or cp is an existing folder, the file goes inside it and keeps its name.

samples_backup.csv 0

Reading the output

Only two of the five lines print anything. samples_backup.csv comes from the first ls results: after the copy and the rename, it is the one file in the folder. The 0 comes from the last line: after rm, the folder is empty, so ls printed no lines and wc -l counted zero. The cp, mv and rm lines printed nothing, because they succeeded.

On a Mac, wc pads its number with spaces on the left, so the last line may look indented; the count is the same. Counting files this way is a quick check before and after any step that should create or remove a known number of them: four read files in, four results out.

Worked example · what a wildcard expands to

data/*_R1.fastq.gz → data/ctrl_1_R1.fastq.gz data/treat_1_R1.fastq.gz
so ls data/*_R1.fastq.gz runs as ls data/ctrl_1_R1.fastq.gz data/treat_1_R1.fastq.gz

Check a wildcard with ls (or echo) before you use it with rm or mv.

Step by step. The shell takes the pattern apart. The fixed part data/ says which folder to look in. Then it goes through every name in that folder and asks whether it ends in _R1.fastq.gz, with anything at all in front. ctrl_1_R1.fastq.gz: yes, with ctrl_1 in front. ctrl_1_R2.fastq.gz: no, it ends in _R2. practice_genes.gtf: no. And so on through all seven. Two names pass, and they replace the pattern on the command line, in alphabetical order.

What the answer means. Any command given the pattern gets exactly the same list. ls only prints it, which is harmless; rm would delete every file on it. echo is a command that prints its arguments, so it shows the expanded list without doing anything to the files. Checking first costs a second and shows you exactly what the destructive command would touch.

Key point

rm deletes permanently: there is no recycle bin. Keep raw data in its own folder, never edit it, and keep a copy somewhere else.

A second example of how quickly this goes wrong: a stray space typed between a * and the rest of a pattern turns one pattern into two arguments. The first, a lone *, matches every file in the folder. Given to rm, that deletes everything, and the second argument then only produces an error. This is the reason for the project layout above: if data holds only raw files that you never change, and a copy is kept elsewhere, a mistake in results costs you a rerun, not your data.

Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course

Check your understanding

1. Which files does *.gtf match?

2. How do you rename a.txt to b.txt ?

3. What does ls data/s?.txt match?

Up next

Lesson 3

Looking inside files

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.