Lesson 7 of 10
Module 3 · The command line
Bioinformatics tools depend on specific versions of other software. Conda installs them, with everything they need, into separate environments you can record and rebuild exactly.
Install conda once
Most bioinformatics programs are built on other software: a particular version of Python, Java or a compression library, for example. These are the program's dependencies. Installing tools by hand means finding and installing every dependency yourself, and two tools may need different versions of the same one. Conda is a package manager: you name the tools you want, and it works out a set of versions that fit together, downloads them and installs them. A package is one tool or library in the ready-to-install form conda uses, and a channel is an online collection of packages.
Use Miniforge, a small conda installer that uses the community conda-forge channel. Download the installer for your system from github.com/conda-forge/miniforge, run it, and open a new terminal. On Colab, run !pip install -q condacolab and then import condacolab; condacolab.install() in a Python cell; the session restarts once.
The installer puts everything in one folder in your home directory, normally miniforge3, and needs no administrator rights, which matters on shared servers where you cannot install software system-wide. Opening a new terminal afterwards lets the shell pick up the conda command. Google Colab is a hosted notebook service; in a notebook cell, a line starting with ! is sent to the shell instead of to Python, and the restart lets the notebook find the newly installed conda.
Why not the other installers you may have heard of? A system package manager such as apt on Ubuntu needs administrator rights and usually offers one, often older, version of each tool. pip installs Python packages, but many bioinformatics tools are written in C, C++ or Java and depend on libraries that pip does not manage. Conda handles programs in any language, keeps several versions side by side and works without administrator rights, which is why it became the usual choice for command-line bioinformatics.
During installation, the installer asks whether to set up your shell for conda; answering yes adds a few lines to your shell's start-up file so that the conda command is found in every new terminal. Check that it worked with conda --version, which should print a version number. If the shell instead reports command not found, the usual cause is a terminal that was opened before the installation finished; open a new one. If that does not help, conda init adds the start-up lines afterwards.
Bioconda is described in Grüning et al. (2018), 'Bioconda: sustainable and comprehensive software distribution for the life sciences', Nature Methods 15:475–476, doi:10.1038/s41592-018-0046-7. The Bioconda and conda-forge documentation pages are the place to check current installation advice, which changes from time to time. In your methods, cite the tools themselves, not only the installer. seqkit, for example, is Shen et al. (2016), 'SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation', PLOS ONE 11:e0163962. Give each tool's exact version, and share the environment.yml, or a full conda env export, with your code so a reader can rebuild the same set-up.
One environment per project
An environment is a separate folder holding its own copy of a set of programs and their dependencies. Tools in one environment cannot see or disturb tools in another. Activating an environment tells this terminal to look in that folder first when you type a command.
In Variables, loops and scripts you met PATH, the list of folders the shell searches for programs. Activating an environment works by putting that environment's folder of programs at the front of PATH, so its versions are found before any others. Deactivating takes it out again.
conda create -n ngs -c conda-forge -c bioconda seqkit fastqc fastp conda activate ngs # your prompt now starts with (ngs) seqkit version seqkit stats data/*_R1.fastq.gz
What this does
conda create -n ngsmakes a new environment named ngs (short for next-generation sequencing). The two-coptions name the channels to search, in order of priority. The last three words are the packages to install: seqkit, a fast toolkit for FASTA and FASTQ files; FastQC, which writes quality reports; and fastp, which trims reads. You will use the last two in Module 5. Conda lists everything it is about to install, including dependencies such as the Java that FastQC runs on, and asks you to confirm.conda activate ngsswitches this terminal to the new environment. The (ngs) at the start of your prompt reminds you which environment is active.seqkit versionchecks that the tool is found and records which version you have.seqkit statssummarises every R1 file in one table, reading the compressed files directly.
Reading the output
- The first line is the installed version, 2.14.0. Write it down; it goes in your methods.
- In the table, num_seqs is the number of reads and sum_len the total number of bases. min_len, avg_len and max_len are the shortest, mean and longest read length.
- For ctrl_1: 9,000 reads × 100 bases = 900,000 bases, the same counts you worked out with
wc -land awk in Looking inside files and Text processing with awk and sed. Every read is exactly 100 bases, as expected for reads straight off the sequencer that have not yet been trimmed.
The most common mistake with environments is forgetting to activate one. A new terminal starts outside your project's environment, so seqkit is either not found or, worse, an older copy elsewhere on the computer runs instead. which seqkit prints the full path of the program the shell will use; if that path contains envs/ngs, you are using the environment's copy. The same applies to scripts: activate the environment before you run them.
Environments can change after they are created. conda install -n ngs -c conda-forge -c bioconda samtools adds a tool to the existing ngs environment, and conda env remove -n ngs deletes an environment you no longer need. Because each environment is self-contained, removing one never affects the others.
Bioconda is the channel with thousands of bioinformatics tools; conda-forge supplies the general software they rely on. Your version numbers may be newer.
Bioconda's own instructions list conda-forge first, as the command above does, so that the general software comes from one consistent source. Newer versions appear regularly, which is exactly why you record the versions you used: a later version may fix a bug or change a default, and then give slightly different numbers on the same data.
Go deeper: why conda can be slow, and strict channel priority
When you ask for packages, conda has to find one version of every package and dependency that fits all the others. The part that solves this puzzle is called the solver, and older versions of conda could take many minutes over it. Since late 2023, conda has used a much faster solver taken from mamba, a reimplementation of conda that Miniforge also installs. The mamba command accepts the same options as conda.
Bioconda's set-up instructions add one more setting: conda config --set channel_priority strict. With it, if a package exists in a higher-priority channel such as conda-forge, conda never takes a version of it from a lower one, which avoids clashes between builds. Not every package is built for every system, so a tool that installs on Linux may be missing for Macs with Apple processors.
Record the environment
An environment.yml file lists what a project needs. Keep it with your scripts, and anyone can rebuild the same environment.
The file is written in YAML, a plain-text format of key: value lines in which items of a list start with a dash and indentation shows what belongs to what. People can read it without special software, but the spaces matter: an item indented differently from its neighbours is read as something else.
One rule catches many people: YAML does not allow tab characters for indentation, only spaces. Some editors insert a tab when you press the Tab key, and conda then refuses the file with an error about its format. If a file that looks right will not load, check the indentation first.
name: ngs channels: - conda-forge - bioconda dependencies: - seqkit=2.14 - fastqc - fastp
Reading the file
nameis the environment's name, the one you give toconda activate.channelslists where to look, in priority order.dependencieslists the packages.seqkit=2.14is pinned: conda will install a 2.14 release (2.14.0, or a later 2.14.x bug-fix release) and nothing else.fastqcandfastpare not pinned, so a rebuild picks the newest versions that fit. Pin every tool whose version could change your results.
conda env create -f environment.yml # build it on another computer conda env export --from-history # list what you asked for conda env list # all your environments conda deactivate
What this does
conda env create -f environment.ymlbuilds a new environment from the file, with the name written inside it.conda env export --from-historyprints the packages you asked for when you built the environment, in the same layout as the file above. Without--from-historyit lists every installed package, dependencies included, with exact version and build codes: the most precise record, but often tied to one operating system.conda env listshows all your environments and where they live; the active one is marked with a star.conda deactivateleaves the current environment and returns the terminal to the one it was in before.
You do not have to write the file by hand. As in Searching, sorting and pipes, the > redirection saves printed output to a file: conda env export --from-history > environment.yml writes the record straight into your project. Check the result before you share it, because the exported file may name packages without the version you want pinned, and it may include a prefix: line with a folder path that only exists on your computer.
Update the file whenever you add or change a tool, and keep it next to your scripts. An environment file that no longer matches the environment you actually used is worse than none, because it looks like a complete record.
Worked example · why separate environments
one shared installation → one project breaks
two environments → each has its own version, side by side
Step by step. Each environment is its own folder inside your conda installation, such as miniforge3/envs/ followed by the environment's name. Each folder holds its own copy of samtools. When you activate project A's environment, the terminal finds samtools 1.9; when you activate project B's, it finds 1.20. Neither installation knows about the other.
What the answer means. You can keep an old pipeline running exactly as it was published while using current tools for new work. The cost is disk space, since shared dependencies may be stored more than once, and that is usually a fair price.
The same reasoning applies to anything with versions. A tool that needs an old version of Python and one that needs a new version cannot share an environment, but each can have its own.
Key point
Do not install everything into the base environment. Create a small environment per project, and write its versions in your methods.
The base environment holds conda itself. If you fill it with tools, a later install can force changes to the packages conda depends on, and conda itself may stop working properly. A methods sentence can be as short as "Reads were summarised with seqkit 2.14.0", with one such statement per tool and each version taken from the tool itself or from conda env export.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. Where do most bioinformatics conda packages come from?
2. What does conda activate ngs do?
3. Why keep an environment.yml with a project?
Up next
Lesson 8
Bigger jobs, servers and clusters