Lesson 8 of 10
Module 3 · The command line
Real datasets need more memory and time than a laptop has. This lesson covers long-running jobs, working on a remote server, moving data, and submitting work to a cluster scheduler.
Long jobs that survive logging out
A real alignment can run for hours. Normally a command runs in the foreground: the terminal waits for it to finish before you can type again, and if you close the terminal or lose the connection, the command is stopped. A job in the background runs while you carry on typing. The commands below start a job that way and then check on the computer it is running on.
nohup ./scripts/count_reads.sh data/*.fastq.gz > results/counts.tsv 2> logs.txt & jobs # jobs started from this terminal top # what is running, and how much CPU and memory it uses (q to quit) df -h . # free disk space here du -sh data # space a folder uses
What this does
- The first line runs the read-counting script from Variables, loops and scripts on every FASTQ file, R1 and R2. The results go into results/counts.tsv, any error messages into logs.txt, and the final
&sends the whole job to the background. jobslists the background jobs started from this terminal, each with a number and its state, such as running or done.topshows a live table of the programs running on the computer, busiest first, with the share of CPU and memory each one uses.df -h .("disk free") shows the size and free space of the disk that holds the current folder; the dot means "here".du -sh data, from Looking inside files, shows how much of that space one folder takes.
Check free space before a large job. Aligned reads and other intermediate files can add up quickly, and a job that fills the disk fails part-way through, sometimes leaving a half-written file that looks complete.
The script is the count_reads.sh you wrote in Variables, loops and scripts. Because it stops at the first error and reports through its exit code, it is already safe to leave running unwatched: if something goes wrong, the reason ends up in the log rather than as a quiet zero in the results.
& runs the command in the background; nohup keeps it running if you disconnect. > saves normal output and 2> saves error messages to a separate file. For interactive work you want to return to, tmux keeps a whole terminal session alive on the server.
Every command has two output channels. Standard output carries the results; standard error carries warnings and error messages. Keeping them in separate files means your results table stays clean, while the log tells you what went wrong. nohup ("no hang-up") protects the job from the signal a terminal sends to its programs when it closes. tmux works differently: you start a session, run commands in it, detach from it and log out, and later log in again and reattach to find everything as you left it, including commands still running.
The & on its own is not enough for long jobs. A background job still belongs to the terminal that started it, and when that terminal closes the job may be stopped with it. That is the gap nohup fills. To see how a running job is doing, look at its files: tail logs.txt shows the latest error messages, and tail -f logs.txt keeps printing new lines as they arrive until you press Ctrl+C. Pressing Ctrl+C there stops only tail, not the job.
A typical tmux session looks like this. tmux new -s align starts a session named align, and you run your commands inside it as usual. Pressing Ctrl+B, then D, detaches; you can now log out. Later, tmux attach -t align brings the session back. Many people start every long piece of server work inside tmux, because a dropped Wi-Fi connection then costs nothing.
Servers: ssh, scp and rsync
A server is a computer you use over the network, usually one with far more memory, processors and disk than a laptop. You reach it with ssh ("secure shell"), which gives you a terminal on the remote machine through an encrypted connection. Everything you type after logging in runs on the server, not on your laptop.
ssh you@server.example.ac.uk # log in scp data/ctrl_1_R1.fastq.gz you@server.example.ac.uk:~/data/ # copy one file up rsync -av --progress project/ you@server.example.ac.uk:~/project/ # copy a folder; resumes and skips unchanged files tar -czf results_backup.tar.gz results # pack a folder into one file
What this does
ssh you@server.example.ac.uklogs in as the user you on the computer server.example.ac.uk (an example address; your institution gives you the real one). Many servers ask you to set up an ssh key, a matched pair of files that proves who you are without typing a password each time.scp("secure copy") copies a file between computers. The part after the colon is a path on the server;~means your home folder there. Run it from your laptop, not from inside the ssh session.rsync -av --progresscopies a whole folder.-a("archive") copies everything inside, keeping dates and permissions;-vlists each file;--progressshows how far each transfer has got. Run it again after an interruption and it sends only what is missing or changed.tar -czfpacks a folder into one compressed archive file:-ccreates an archive,-zcompresses it with gzip and-fnames the file. One file is easier to copy, check and store than thousands.
One detail of rsync catches many people: the slash at the end of the source folder matters. project/ means "the contents of project"; without the slash, rsync copies the folder itself, and you can end up with a folder inside a folder of the same name.
After ssh logs you in, the prompt usually changes to show the server's name; that is the sign that your commands now run there. Type exit to log out and return to your laptop's terminal. Losing track of which computer a terminal is connected to is a common source of confusion, so it is worth glancing at the prompt before deleting or copying anything.
An ssh key is made on your laptop with ssh-keygen. It creates two files: a private key, which stays on your laptop and is never shared, and a public key, ending in .pub, which you give to the server. The server can then confirm that you hold the matching private key without a password crossing the network.
Copies work in both directions: put the server path first and the local path second to bring results back to your laptop. Before a large rsync, adding -n (a dry run) lists what would be copied without copying anything, which is a cheap way to check that the trailing slash and the destination are what you meant.
Go deeper: checking that a copy arrived intact
A transfer that ends without an error is not proof that the file is complete and unchanged. The standard check is a checksum: a short code calculated from every byte of a file, which changes if even one byte differs. Sequencing centres usually send an MD5 checksum with each file. On Linux, md5sum data/ctrl_1_R1.fastq.gz prints the code and the file name; on macOS the command is md5.
If the centre provides a file of codes, md5sum -c checksums.md5 checks every file it lists and prints OK or FAILED for each. Compare the codes after every large copy, and keep the checksum file with the raw data. For gzip files, gzip -t file.gz is a second quick test: it reads the whole file and reports an error if it is cut short or damaged, and prints nothing if all is well.
Clusters: submit, do not run
On a shared cluster you log in to a login node and send heavy work to compute nodes through a scheduler, most often Slurm. You describe the resources at the top of a script and submit it.
A cluster is many computers, called nodes, linked together and shared by many users. The login node is where everyone arrives; it is meant for editing files, moving data and submitting work. The compute nodes do the heavy calculations. The scheduler keeps a queue of submitted jobs and starts each one when a compute node with the requested CPUs and memory is free, so that users do not slow each other down. Slurm is the most common scheduler; others, such as PBS and SGE, work on the same idea with different commands.
The script below is the read-counting job from the start of this lesson, rewritten for Slurm.
#!/usr/bin/env bash #SBATCH --job-name=count_reads #SBATCH --cpus-per-task=1 #SBATCH --mem=2G #SBATCH --time=00:30:00 #SBATCH --output=logs/%x_%j.out set -euo pipefail source ~/miniforge3/etc/profile.d/conda.sh conda activate ngs ./scripts/count_reads.sh data/*.fastq.gz > results/counts.tsv
What this does, line by line
- The shebang is the same as in any bash script.
- Lines starting with
#SBATCHare comments to bash but instructions to Slurm.--job-namegives the job a readable name.--cpus-per-task=1asks for one processor,--mem=2Gfor 2 gigabytes of memory and--time=00:30:00for at most 30 minutes, written as hours:minutes:seconds. A job that goes past its time or memory is stopped by the scheduler. --output=logs/%x_%j.outnames the file for everything the job prints. Slurm fills in%xwith the job name and%jwith the job number, so every run gets its own log file.set -euo pipefailstops the job at the first error, as in Variables, loops and scripts.- The
sourceline loads conda's set-up file. A job runs in a fresh, non-interactive shell that has not read your usual start-up settings, so without this lineconda activatewould not work. - The last two lines activate the environment and run the script, exactly as you would by hand.
mkdir -p logs sbatch scripts/count_reads.slurm # submit; prints a job number squeue -u $USER # your queued and running jobs scancel 123456 # cancel a job by its number
What this does
mkdir -p logsmakes the folder for the log files. It must exist before you submit: Slurm does not create it, and a job that cannot write its log may fail without telling you why. The-pmeans "no error if it already exists".sbatchhands the script to the scheduler and returns at once with the job's number. The job itself runs later, on a compute node.squeue -u $USERlists your jobs.$USERis a variable that always holds your user name. The state column shows PD for pending (waiting in the queue) and R for running.scancelcancels a job; the number here is an example, so use the onesbatchgave you.
When the job disappears from squeue, it has finished, successfully or not. Read its log in logs/ first: the file name contains the job number, and any error message from the script ends up there. Slurm starts the job in the folder you were in when you ran sbatch, so the relative paths in the script, such as data/ and results/, only work if you submit from the project folder. Submitting from the wrong folder is one of the most common reasons a job fails within seconds.
For testing commands on a compute node, many clusters let you ask for an interactive session, for example with srun --pty bash, which gives you a terminal on a compute node once the scheduler finds room. The exact command and its limits differ between clusters, so check your local guide.
Worked example · asking for the right resources
a Slurm job array of 48 tasks, 1 CPU each → about 20 minutes if 48 CPUs are free
Step by step. Run one after another, 48 samples × 20 minutes = 960 minutes, and 960 ÷ 60 = 16 hours. A job array submits one script once but runs it as many separate tasks, each given its own index number, which the script uses to pick one sample. If 48 CPUs are free, all 48 tasks run at the same time, and the whole batch takes about as long as one sample, 20 minutes.
What the answer means. On a busy cluster not every task starts at once. If only 12 CPUs are free, the tasks run in waves of 12: 48 ÷ 12 = 4 waves × 20 minutes = 80 minutes, still far less than 16 hours. Work that splits into independent pieces, one per sample, is exactly what a cluster is good at.
Ask for a little more time and memory than a test run needed. Too little and the job is killed; far too much and it waits longer in the queue.
Slurm was first described in Yoo, Jette and Grondona (2003), 'SLURM: Simple Linux Utility for Resource Management', in Job Scheduling Strategies for Parallel Processing, Lecture Notes in Computer Science 2862 (Springer). It is now developed by SchedMD, whose documentation at slurm.schedmd.com describes every sbatch option. Treat the script in this lesson as a template, not a recipe: each cluster sets its own partitions (groups of nodes), accounts, limits and ways of loading software, and its local guide comes first. After a job ends, sacct can report how long it ran and how much memory it used, which helps you size the next request. Many institutions ask you to acknowledge their computing facility in papers.
Key point
Never run heavy jobs on a cluster's login node; it is shared by everyone. Test on a small subset, then submit.
The practice files in this course are a ready-made small subset: each holds about 10,000 reads, so a test run takes seconds. Time the test, check the log for errors, then submit the full data with a time limit scaled up from the test plus a margin. On many clusters, administrators stop programs that use a lot of CPU on the login node, so a heavy job started there may be killed anyway.
Further reading · Module 3The Unix Shell (The Carpentries): a second explanation of every idea in this module, with exercises. · All reading for this course
Check your understanding
1. What does & at the end of a command do?
2. Which tool copies a large folder and can resume if interrupted?
3. On a Slurm cluster, how do you run a heavy job?
Up next
Lesson 9
Workflows, the cloud and reproducible pipelines