Lesson 1 of 10
Module 3 · The command line
The terminal is a window where you type commands instead of clicking. It looks bare, but it is how you run almost every bioinformatics tool, on your laptop or on a server with hundreds of cores.
This lesson opens Module 3. Module 2 introduced the file formats and databases that biological data comes in: FASTA, GenBank, GFF, VCF and the archives that hold them. Real files of these kinds are often far too large to open in a word processor or a spreadsheet, and the servers that store them usually have no screen to click on. The command line handles files of any size, one line at a time, and every step you type can be saved and rerun. That is why this module turns to it: it teaches the commands you will use to look at, search and reshape those files.
Three ways to get a Linux terminal
A terminal is the window you type into. The program running inside it, which reads each command and carries it out, is called the shell; the most common one is called bash. Linux is a family of operating systems, the software that runs a computer, like Windows or macOS. Almost every server used for bioinformatics runs Linux, which is why the tools are built for it. You need one of the three set-ups below; pick whichever matches your computer.
Why type commands at all, when most programs have menus and buttons? There are three reasons. First, size: a FASTQ file from one sequencing run can hold tens of millions of reads, and command-line tools read it line by line instead of loading it all at once. Second, record: a typed command is text, so it can be saved, checked by someone else and run again on new data in exactly the same way. Third, reach: most bioinformatics software is written to be run from a shell, and the large computers it runs on are reached through one. Clicking leaves no record of what you did; a command does.
When the terminal is ready for a command it shows a prompt: a short line of text, often ending in $ (or % in zsh), followed by the cursor. You type after the prompt and press Enter to run the line. In this course the prompt is not shown in the code boxes, so you can copy the lines directly; do not type the $ yourself if you see it in other guides.
- Open the Terminal app; nothing to install
- macOS is not Linux, but almost every command here works the same
- Install WSL (Windows Subsystem for Linux): open PowerShell and run wsl --install
- Then open “Ubuntu” from the Start menu
- A Linux machine in your browser, free with a Google account
- Start each cell with %%bash
- Each cell starts in /content, so begin cells with a cd
- macOS. macOS and Linux share a common ancestor, Unix, which is why the commands carry over. Newer Macs use a shell called zsh by default instead of bash; for the commands in this lesson the two behave the same.
- Windows. WSL runs a real Linux system inside Windows. PowerShell is Windows' own command window, used here only once to install it. Ubuntu is a widely used version of Linux, and the one WSL installs by default. After the first start you will be asked to choose a Linux user name and password; these belong to the Linux system only.
- Google Colab. Colab is a notebook: a page of cells, each a box you type into and run. Normally the cells run Python. The first line %%bash tells Colab to run the rest of that cell as shell commands instead. Each cell starts afresh in the folder /content, so a cd at the start of each cell takes you back to your course folder. The machine is temporary: files you leave on it are deleted when the session ends.
The behaviour of a Unix-like shell and its basic tools is set out in POSIX (IEEE Std 1003.1), a standard developed jointly by the IEEE and The Open Group. Bash is GNU software, and its full reference is the Bash Reference Manual from the Free Software Foundation, also available as man bash. Tools from different families can differ in their options: GNU tools on Linux, BSD tools on macOS. Apple still ships an old bash, version 3.2, and has made zsh the default shell since macOS Catalina (2019). When a script matters for a result, note the operating system and the shell version, which bash --version prints.
Where am I? What is here?
Every command is a program name, sometimes followed by options (starting with -) and arguments (what to act on). Create a folder for the course, go into it, and look around. Put the course files in it.
Take the first line below as an example: mkdir is the program, -p is an option that changes how it behaves, and ~/linux-course is the argument, the folder it should create. The parts are separated by spaces. A directory is the same thing as a folder; the terminal world uses the older word, which is why so many commands contain "dir".
Commands are case-sensitive: ls works, but LS or Ls is a different name, and the shell replies that the command is not found. The same message appears when a name is mistyped or the program is not installed, so "command not found" is first a reason to check your spelling.
mkdir -p ~/linux-course # make a folder in your home directory cd ~/linux-course # change directory into it pwd # print working directory: where am I? ls # list what is here
What this does
mkdir -p ~/linux-coursemakes a directory called linux-course inside your home directory. The-poption means "no error if it already exists", so running the line twice does no harm. It prints nothing when it succeeds.cd ~/linux-coursemoves you into that directory. From now on, commands act on files here unless told otherwise. It also prints nothing.pwdprints your working directory, the folder you are in now, as a full path.lslists the files and folders in the working directory.
Many commands print nothing when they work, and print a message only when something goes wrong. Silence is usually good news.
Reading the output
The first line comes from pwd: the full address of the folder you are in. The other seven lines come from ls, which lists names in alphabetical order. They are the course files, and their names already tell you what they hold:
- ctrl_1_R1.fastq.gz and ctrl_1_R2.fastq.gz: reads from control sample 1, in FASTQ format (Files, folders and wildcards). R1 and R2 are the two reads of each pair from paired-end sequencing. The .gz ending means the file is compressed with a program called gzip, to save space.
- treat_1_R1.fastq.gz and treat_1_R2.fastq.gz: the same for treated sample 1.
- practice_genome.fa: a reference genome sequence in FASTA format, a plain-text format of a name line starting with > followed by the sequence.
- practice_genes.gtf: a GTF (Gene Transfer Format) file, a table saying where each gene lies on that genome.
- practice_samples.csv: a CSV (comma-separated values) file, a small table describing the samples.
If ls printed nothing, the course files are not in this folder yet; move them in before you go on.
Your pwd shows your own user name; on Colab it is /root/linux-course. Everything after # is a comment, ignored by the computer.
On Colab you work as the administrator account, called root, whose home directory is /root. On a Mac, home directories live under /Users instead of /home. Comments are notes for people: they are there so you can read later what each line was for, and you can copy the lines with or without them.
Go deeper: how the shell finds a program
When you type ls, the shell has to find a program with that name. It looks through a list of folders stored in a variable called PATH, in order, and runs the first match. echo $PATH shows the list: folder names separated by colons. type ls tells you which file the shell will run, or whether the name is an alias, a short name standing for a longer command.
A few commands are not separate programs at all but are built into the shell; these are called builtins. cd has to be one. Every program runs as its own process, a running copy with its own working directory, so a separate cd program could only change its own folder and would then exit, leaving yours as it was. type cd reports that it is a shell builtin. Activating a conda environment in Installing software with conda works through PATH too: it puts the environment's folder at the front of the list.
Paths
A path is the address of a file. Paths starting with / are absolute: from the top of the system. Others are relative: from where you are now.
The file system is a tree. At the top is a single folder called the root, written /. Every other folder sits inside it, one level below another. An absolute path lists every folder on the way down from the root, so it means the same thing wherever you are. A relative path starts from your working directory, so the same relative path can point to different files depending on where you run it. In /home/you/linux-course, the relative path practice_genome.fa and the absolute path /home/you/linux-course/practice_genome.fa name the same file.
Which kind should you use? Relative paths are shorter and keep working if the whole project folder is moved or copied to another computer, as long as you start from the right place. Absolute paths always point to the same file, whoever runs the command and from wherever, but they break as soon as the folder is moved, and they often contain a user name that differs between computers. A common pitfall is a relative path that worked yesterday and fails today, because the terminal was opened in a different folder. Two short forms help you move around: cd on its own takes you back to your home directory, and cd - returns to the folder you were in just before.
| Symbol | Means | Example |
|---|---|---|
| ~ | Your home directory | cd ~ |
| . | The current directory | ls . |
| .. | The directory one level up | cd .. |
| / | The top (root) of the file system, and the separator between folders | ls /home |
How to read this table
Each symbol is a short way of writing part of a path, and each can be combined with others. Your home directory is your own folder, where your files live; ~ is replaced by its full path, such as /home/you, before the command runs. A single dot is the folder you are in, so ls . lists the same as ls on its own. Two dots mean the parent folder, the one that contains the current one. The slash has two jobs: at the start of a path it means the root, and between names it separates one folder from the next, so in /home/you the first slash is the root and the second is a separator.
Getting help
Most tools explain themselves: ls --help prints the options, and man ls opens the full manual (press q to quit). The up arrow recalls earlier commands, and Tab completes file names as you type, which saves time and typing errors.
man is short for manual. Inside a manual page, the arrow keys and the space bar scroll through it. Tab completion is worth using from the first day: type prac and press Tab, and the shell fills in as much of the name as is shared by the files that start that way. Pressing Tab twice lists the possible matches. If nothing appears at all, no file starts that way, which often reveals a typing mistake before it causes an error. One difference to know about: on macOS, some built-in tools such as ls come from a different family of Unix tools and may not accept --help; man works on all three set-ups.
Help pages follow a few conventions that make them easier to read. Parts in square brackets, such as [OPTION], are optional. Single-letter options start with one dash and can usually be joined, so ls -l -h and ls -lh mean the same; long options start with two dashes and are spelled out, like --help. Two keys are worth learning now as well. Ctrl+C stops the command that is running, which is the way out when something prints far more than you expected or seems to hang. history lists the commands you have typed, so you can see what you ran earlier. When an error message makes no sense, searching the web for its exact wording, without your own file names, usually finds someone who has met it before.
Worked example · following a path
cd .. → /home/you
cd linux-course/../.. → /home
Step by step. Start in /home/you/linux-course. The first cd .. climbs one level, removing the last folder from the path: you are now in /home/you. The second command starts from there, because a relative path always starts from where you are. Read it one piece at a time: linux-course goes down into /home/you/linux-course; the first .. climbs back to /home/you; the second .. climbs again to /home.
What the answer means. A path is a set of steps, followed from left to right, and the result depends on the starting point. The same cd linux-course/../.. typed in another folder would end somewhere else, or fail if there is no linux-course folder there. When you are unsure where you are, run pwd.
Key point
Avoid spaces in file and folder names. Use sample_1.fastq.gz, not sample 1.fastq.gz: a space separates arguments, so the terminal would see two files.
A second example: a folder named my results. A command meant to act on it would instead act on two names, my and results: at best you get an error that neither exists, and at worst it acts on a different file called results. Upper and lower case matter too. On Linux, Data and data are two different folders, while a Mac normally treats them as the same, so a script that works on one computer can fail on another. Lower-case names joined with underscores avoid both problems.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. What does pwd print?
2. You are in /home/you/project/data . Where does cd .. take you?
3. Which path is absolute?
Up next
Lesson 2
Files, folders and wildcards