Lesson 7 of 8
Module 1 · Python basics
Copying the same function into every notebook does not scale. This lesson moves your functions into a module you import, handles bad input with clear errors, and processes a whole folder of files.
Write a module
A module is just a .py file of functions. In Colab, the first line %%writefile seqtools.py saves the cell as a file; on your own computer, create the file in your editor. The text in triple quotes is a docstring: it says what each function does.
Keeping your functions in one file solves a real problem. If gc_content lives in five notebooks and you find a bug, you must fix it five times and hope you miss none. In a module, you fix it once and every notebook that imports it gets the fix. The line starting %% is a cell magic: an instruction to the notebook itself, not to Python, which is why it only works as the first line of a notebook cell.
Choose the module's name with care. When you write import seqtools, Python looks first in the notebook's own folder, so a file there named after an existing library, such as pandas.py or random.py, is imported instead of the real library and breaks code that has nothing to do with it. A distinctive name such as seqtools avoids the clash.
%%writefile seqtools.py
"""Small tools for DNA sequences."""
COMPLEMENT = {"A": "T", "T": "A", "G": "C", "C": "G", "N": "N"}
def gc_content(seq):
"""Share of G and C bases, ignoring N."""
seq = seq.upper()
known = len(seq) - seq.count("N")
if known == 0:
raise ValueError("sequence has no A, C, G or T bases")
return (seq.count("G") + seq.count("C")) / known
def reverse_complement(seq):
"""Reverse complement; N stays N."""
try:
return "".join(COMPLEMENT[base] for base in reversed(seq.upper()))
except KeyError as err:
raise ValueError(f"not a DNA base: {err.args[0]!r}") from None
def read_fasta(path):
"""Return a dictionary of {id: sequence} from a FASTA file."""
records, name = {}, None
with open(path) as handle:
for line in handle:
line = line.strip()
if line.startswith(">"):
name = line[1:].split()[0]
records[name] = ""
elif line:
records[name] += line
return recordsWhat this does, function by function
- The top. The first docstring describes the whole module.
COMPLEMENTis the base-pairing dictionary from Strings, lists and functions with one new pair, N to N. Its name is in capitals by convention, to signal a fixed value that the code should not change. gc_content.knownis the length minus the number of Ns: the bases whose identity is known. If there are none,raise ValueError(…)stops the function with a clear message, instead of letting a division by zero fail with a less helpful one. Otherwise the G and C count is divided byknown, not by the full length.reverse_complement. Thetryblock attempts the same one-line reverse complement as before. If a letter is missing from the dictionary, Python raises aKeyError, and theexcept KeyError as errblock catches it.err.args[0]is the letter that failed;!rinside the f-string prints it with quotes, so a space or an odd character is visible.from Nonehides the originalKeyErrorfrom the error report, so the reader sees one clear message.read_fasta.records, name = {}, Nonesets up two names at once: an empty dictionary andNone, Python's value for "nothing yet".with open(path) as handleopens the file; looping overhandlegives one line at a time.line.strip()removes the line-ending character and any stray spaces. A line starting with>is a header:line[1:]drops the>,.split()breaks the rest into words, and[0]keeps the first word, the identifier. It starts an empty sequence for that name. Any other non-empty line (elif line:; an empty string counts as false) is added to the current sequence, so sequences wrapped over several lines are joined back together.
Two improvements on Strings, lists and functions: N, the letter for an unknown base, is handled, and anything else gives a clear ValueError instead of a confusing KeyError. with open(...) closes the file for you when the block ends.
A ValueError is Python's standard error for "this value is the right kind of thing, but not acceptable": here, a string that is not valid DNA. Raising it with a message that names the bad letter tells the user what to fix. Closing files matters because an operating system limits how many files a program may have open at once, and on some systems an open file cannot be moved or deleted; the with block closes it even if an error happens inside.
read_fasta makes two assumptions worth knowing. It expects the file to start with a header: a sequence line before the first > would try to add text to records[None], which does not exist, and stop with a KeyError. And it keys the dictionary by identifier, so if two records share an identifier, the second silently starts again from an empty sequence and the first is lost. Both are rare in files from databases but common in files assembled by hand, and both are good candidates for the checks at the end of this lesson.
Wilson et al. (2014), "Best practices for scientific computing", PLOS Biology 12(1): e1001745, recommends the habits in this lesson: write code once and reuse it through functions and modules, test it with assertions and automated tests, and let the computer repeat tasks. Its follow-up, Wilson et al. (2017), "Good enough practices in scientific computing", PLOS Computational Biology 13(6): e1005510, gives a shorter list aimed at beginners. Python's own conventions are written up as Python Enhancement Proposals: PEP 8 is the style guide, which is where capitals for fixed values such as COMPLEMENT come from, and PEP 257 covers docstrings. The two papers give advice drawn from experience, not results of controlled trials, so cite them as guidance.
Import and use it
import seqtools
seqs = seqtools.read_fasta("practice_sequences.fasta")
print(list(seqs))
print(round(seqtools.gc_content("ATGNNGC"), 3))
print(seqtools.reverse_complement("ATGNC"))
try:
seqtools.reverse_complement("ATGX")
except ValueError as err:
print("Error:", err)What this does
import seqtoolsfinds seqtools.py in the notebook's working folder and loads it. You leave off the .py. Its functions are then reached withseqtools.in front, so it is always clear where a function came from.read_fastareads the practice file into a dictionary, andlist(seqs)lists its keys.- The next two lines test the new handling of N: GC content with two unknown bases, and a reverse complement that contains one.
- The
tryblock deliberately passes an invalid letter, X. Theexcept ValueError as errblock catches the error, stores it aserr, and prints its message instead of stopping the notebook.
Reading the output
The identifiers come out in the order they appear in the file, because Python dictionaries keep the order in which keys were added. 0.6 is worked out in the example below. GNCAT is ATGNC reversed (CNGTA) and complemented, with the N left as N. The last line is the message written in the module, with the bad letter in quotes.
There is a second way to import. from seqtools import gc_content brings in just that function, so you can call it as gc_content(…) without the prefix. It is shorter, but in a long notebook it can be less clear which gc_content you are using, the one from Strings, lists and functions or the module's. Keeping the seqtools. prefix removes the doubt.
The docstrings now pay off. help(seqtools.gc_content) prints the function's name, its parameter and its docstring, and most editors show the same text when you hover over the name. A one-line docstring that says what goes in and what comes out is the cheapest documentation you can write.
Catching an error with try and except is useful when you expect some inputs to be bad and want to note them and carry on, for example skipping a damaged file in a folder of hundreds and listing it at the end. Catching every error without looking, with a bare except:, is a trap: it also hides mistakes in your own code, so the notebook runs to the end and produces wrong results silently. Catch the specific error you expect, as here with ValueError.
Worked example · GC content with unknown bases
G + C among them: G, G, C = 3
GC content = 3 ÷ 5 = 0.6 (dividing by 7 would wrongly give 0.43)
Step by step. The letters are A, T, G, N, N, G, C. Removing the two Ns leaves A, T, G, G, C: five known bases. Of those, three are G or C. 3 ÷ 5 = 0.6. The function from Strings, lists and functions would have divided by all seven letters: 3 ÷ 7 = 0.4286, rounded to 0.43.
What the answer means. An N could be any base, so it should not count either for or against GC. Dividing by 7 treats every unknown base as if it were A or T, which pulls the answer down. In real data this matters: sequencing reads and genome assemblies can contain runs of Ns where a base could not be read, and an honest GC content leaves them out.
A whole folder of files
pathlib handles file paths on any operating system. First split the practice file into one file per sequence, then loop over the folder and collect a summary table.
A path is the address of a file, such as seqs/seq1.fasta. Windows writes paths with backslashes and macOS and Linux with forward slashes; pathlib builds the right form for whichever system the code runs on, so the same notebook works for a colleague on another machine.
from pathlib import Path
import pandas as pd
out = Path("seqs")
out.mkdir(exist_ok=True)
for name, s in seqs.items():
(out / f"{name}.fasta").write_text(f">{name}\n{s}\n")
print(sorted(p.name for p in out.glob("*.fasta")))
rows = []
for path in sorted(out.glob("*.fasta")):
for name, s in seqtools.read_fasta(path).items():
rows.append({"file": path.name, "id": name, "length": len(s),
"gc": round(seqtools.gc_content(s), 3)})
summary = pd.DataFrame(rows)
print(summary)
summary.to_csv("sequence_summary.csv", index=False)What this does
Path("seqs")names a folder, andout.mkdir(exist_ok=True)creates it.exist_ok=Truemeans "no error if it is already there", so the cell can be run twice.- The first loop writes one file per sequence.
out / f"{name}.fasta"joins the folder and a file name with/, whichpathlibunderstands as "inside".write_textwrites a header line and the sequence;\nis the character that ends a line. out.glob("*.fasta")finds every file in the folder whose name ends in .fasta;*stands for any characters.p.nameis the file name without the folder.sortedgives a fixed order, since the operating system does not promise one.- The second loop reads each file with your
read_fastaand appends one dictionary per sequence to the listrows: file name, identifier, length and GC content. pd.DataFrame(rows)turns the list into a table, with the dictionary keys as column names.to_csv(…, index=False)saves it without the 0 to 3 row numbers, which are not data.
Reading the output
The first line confirms four files were written. The table has one row per file, and its lengths and GC values are the same as Biopython's in Sequences with Biopython: 36, 23, 21 and 27 bases, and 0.444, 0.087, 0.952 and 0.481. Splitting and re-reading has lost nothing.
A Path also knows the parts of its own name. For seqs/seq1.fasta, path.name is seq1.fasta, path.stem is seq1 and path.suffix is .fasta, which is a tidy way to turn file names into sample names. pandas and most other libraries accept a Path wherever they accept a file name as text.
Paths like seqs are relative: they are read from the notebook's current working folder, which Path.cwd() shows. If code that worked yesterday now finds no files, a different working folder is the usual cause. And glob("*.fasta") looks only in the folder itself; rglob("*.fasta") also searches every folder inside it, which suits results that are filed one folder per sample.
Building a list of dictionaries and turning it into a DataFrame at the end is the standard pattern for collecting results in a loop.
The reason is speed and simplicity. Adding a row to a list is quick; adding a row to a DataFrame makes pandas copy the whole table each time, which becomes slow for thousands of files. The same pattern works for anything you collect in a loop, such as one quality-control summary per sequencing sample.
Check your functions
assert stops with an error if a statement is false. A few checks against answers you worked out by hand catch mistakes early.
assert seqtools.reverse_complement("AAGC") == "GCTT"
assert seqtools.gc_content("GGCC") == 1.0
print("all checks passed")What this does
Each assert line compares a function's result with an answer worked out by hand. AAGC reversed is CGAA, and complemented is GCTT; GGCC is all G and C, so its GC content is 1.0. If either comparison were false, Python would stop with an AssertionError at that line and the final print would never run. Seeing all checks passed therefore means every check above it held. Good checks include the awkward cases too: a sequence with N, an empty result, an invalid letter. These small checks are called tests, and keeping them next to your functions means every later change is checked automatically.
Be careful when a test compares decimals. Computers store most decimal fractions approximately, so 0.1 + 0.2 == 0.3 is False in Python: the sum is 0.30000000000000004. A test that compares a calculated float with == can therefore fail even when the code is right. math.isclose(a, b) asks whether two numbers are equal to within a tiny tolerance, and is the safer comparison for anything that involved division. The two checks above are safe, because both answers are exact.
Add a test each time you find a bug, one that fails on the old code and passes on the fixed version. That way the same bug cannot quietly return. Testing AI-written code, in Module 3, takes this idea further, with tests kept in their own file and run all at once.
Go deeper: from assert lines to a test suite
As a project grows, the checks move into their own file and a test runner runs them all. A popular runner is pytest, installed with pip install pytest. You write a file whose name starts with test_, such as test_seqtools.py, and inside it functions whose names also start with test_, each holding one or more plain assert lines. Typing pytest in a terminal finds every such function, runs it, and reports which passed and which failed, showing the values that differed.
pytest can also check that an error is raised. Inside a test function, the line with pytest.raises(ValueError): followed by an indented seqtools.reverse_complement("ATGX") passes only if that call raises a ValueError; if the function quietly returns a value, the test fails. That is exactly the check the Module 1 project asks for, written in two lines.
Key point
If you edit seqtools.py after importing it, restart the notebook (or use importlib.reload(seqtools)) so Python loads the new version.
This happens because Python imports each module only once per session and remembers it; a second import seqtools does nothing. So if you fix a bug in the file and rerun your tests without restarting, they still run the old code, and a test can pass or fail for the wrong reason. Restarting is the safer habit, because it also clears any old values left in memory, as in Setting up Python and Jupyter.
Further reading · Module 1The Python Tutorial: the official tutorial, for anything you want explained a second way. · All reading for this course
Check your understanding
1. What is a Python module?
2. What is the GC content of "GCNN" with the new function?
3. What does try … except ValueError do?
Up next
Project
A sequence and sample report