LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Lesson 7 of 8

Module 1 · Python basics

Your own toolkit: modules, files and errors

Your own toolkit: modules, files and errors

Your own toolkit: modules, files and errors

Copying the same function into every notebook does not scale. This lesson moves your functions into a module you import, handles bad input with clear errors, and processes a whole folder of files.

⏱ 14 min readModule 1 · Python basicsFree
Lesson at a glance4 steps · about 14 min
  1. Write a module
  2. Import and use it
  3. A whole folder of files
  4. Check your functions
Files for this lessonSave each file under the name shown here, next to your notebook, or upload it to Colab (folder icon › Upload). Your browser may give the download a different name; rename it before you run the code.
practice_sequences.fastaFour short practice sequences (not real genes)

Write a module

A module is just a .py file of functions. In Colab, the first line %%writefile seqtools.py saves the cell as a file; on your own computer, create the file in your editor. The text in triple quotes is a docstring: it says what each function does.

Keeping your functions in one file solves a real problem. If gc_content lives in five notebooks and you find a bug, you must fix it five times and hope you miss none. In a module, you fix it once and every notebook that imports it gets the fix. The line starting %% is a cell magic: an instruction to the notebook itself, not to Python, which is why it only works as the first line of a notebook cell.

Choose the module's name with care. When you write import seqtools, Python looks first in the notebook's own folder, so a file there named after an existing library, such as pandas.py or random.py, is imported instead of the real library and breaks code that has nothing to do with it. A distinctive name such as seqtools avoids the clash.

Python · seqtools.py
%%writefile seqtools.py
"""Small tools for DNA sequences."""

COMPLEMENT = {"A": "T", "T": "A", "G": "C", "C": "G", "N": "N"}


def gc_content(seq):
    """Share of G and C bases, ignoring N."""
    seq = seq.upper()
    known = len(seq) - seq.count("N")
    if known == 0:
        raise ValueError("sequence has no A, C, G or T bases")
    return (seq.count("G") + seq.count("C")) / known


def reverse_complement(seq):
    """Reverse complement; N stays N."""
    try:
        return "".join(COMPLEMENT[base] for base in reversed(seq.upper()))
    except KeyError as err:
        raise ValueError(f"not a DNA base: {err.args[0]!r}") from None


def read_fasta(path):
    """Return a dictionary of {id: sequence} from a FASTA file."""
    records, name = {}, None
    with open(path) as handle:
        for line in handle:
            line = line.strip()
            if line.startswith(">"):
                name = line[1:].split()[0]
                records[name] = ""
            elif line:
                records[name] += line
    return records

What this does, function by function

  • The top. The first docstring describes the whole module. COMPLEMENT is the base-pairing dictionary from Strings, lists and functions with one new pair, N to N. Its name is in capitals by convention, to signal a fixed value that the code should not change.
  • gc_content. known is the length minus the number of Ns: the bases whose identity is known. If there are none, raise ValueError(…) stops the function with a clear message, instead of letting a division by zero fail with a less helpful one. Otherwise the G and C count is divided by known, not by the full length.
  • reverse_complement. The try block attempts the same one-line reverse complement as before. If a letter is missing from the dictionary, Python raises a KeyError, and the except KeyError as err block catches it. err.args[0] is the letter that failed; !r inside the f-string prints it with quotes, so a space or an odd character is visible. from None hides the original KeyError from the error report, so the reader sees one clear message.
  • read_fasta. records, name = {}, None sets up two names at once: an empty dictionary and None, Python's value for "nothing yet". with open(path) as handle opens the file; looping over handle gives one line at a time. line.strip() removes the line-ending character and any stray spaces. A line starting with > is a header: line[1:] drops the >, .split() breaks the rest into words, and [0] keeps the first word, the identifier. It starts an empty sequence for that name. Any other non-empty line (elif line:; an empty string counts as false) is added to the current sequence, so sequences wrapped over several lines are joined back together.

Two improvements on Strings, lists and functions: N, the letter for an unknown base, is handled, and anything else gives a clear ValueError instead of a confusing KeyError. with open(...) closes the file for you when the block ends.

A ValueError is Python's standard error for "this value is the right kind of thing, but not acceptable": here, a string that is not valid DNA. Raising it with a message that names the bad letter tells the user what to fix. Closing files matters because an operating system limits how many files a program may have open at once, and on some systems an open file cannot be moved or deleted; the with block closes it even if an error happens inside.

read_fasta makes two assumptions worth knowing. It expects the file to start with a header: a sequence line before the first > would try to add text to records[None], which does not exist, and stop with a KeyError. And it keys the dictionary by identifier, so if two records share an identifier, the second silently starts again from an empty sequence and the first is lost. Both are rare in files from databases but common in files assembled by hand, and both are good candidates for the checks at the end of this lesson.

Wilson et al. (2014), "Best practices for scientific computing", PLOS Biology 12(1): e1001745, recommends the habits in this lesson: write code once and reuse it through functions and modules, test it with assertions and automated tests, and let the computer repeat tasks. Its follow-up, Wilson et al. (2017), "Good enough practices in scientific computing", PLOS Computational Biology 13(6): e1005510, gives a shorter list aimed at beginners. Python's own conventions are written up as Python Enhancement Proposals: PEP 8 is the style guide, which is where capitals for fixed values such as COMPLEMENT come from, and PEP 257 covers docstrings. The two papers give advice drawn from experience, not results of controlled trials, so cite them as guidance.

Import and use it

Python
import seqtools

seqs = seqtools.read_fasta("practice_sequences.fasta")
print(list(seqs))
print(round(seqtools.gc_content("ATGNNGC"), 3))
print(seqtools.reverse_complement("ATGNC"))

try:
    seqtools.reverse_complement("ATGX")
except ValueError as err:
    print("Error:", err)
['seq1', 'seq2', 'seq3', 'seq4'] 0.6 GNCAT Error: not a DNA base: 'X'

What this does

  • import seqtools finds seqtools.py in the notebook's working folder and loads it. You leave off the .py. Its functions are then reached with seqtools. in front, so it is always clear where a function came from.
  • read_fasta reads the practice file into a dictionary, and list(seqs) lists its keys.
  • The next two lines test the new handling of N: GC content with two unknown bases, and a reverse complement that contains one.
  • The try block deliberately passes an invalid letter, X. The except ValueError as err block catches the error, stores it as err, and prints its message instead of stopping the notebook.

Reading the output

The identifiers come out in the order they appear in the file, because Python dictionaries keep the order in which keys were added. 0.6 is worked out in the example below. GNCAT is ATGNC reversed (CNGTA) and complemented, with the N left as N. The last line is the message written in the module, with the bad letter in quotes.

There is a second way to import. from seqtools import gc_content brings in just that function, so you can call it as gc_content(…) without the prefix. It is shorter, but in a long notebook it can be less clear which gc_content you are using, the one from Strings, lists and functions or the module's. Keeping the seqtools. prefix removes the doubt.

The docstrings now pay off. help(seqtools.gc_content) prints the function's name, its parameter and its docstring, and most editors show the same text when you hover over the name. A one-line docstring that says what goes in and what comes out is the cheapest documentation you can write.

Catching an error with try and except is useful when you expect some inputs to be bad and want to note them and carry on, for example skipping a damaged file in a folder of hundreds and listing it at the end. Catching every error without looking, with a bare except:, is a trap: it also hides mistakes in your own code, so the notebook runs to the end and produces wrong results silently. Catch the specific error you expect, as here with ValueError.

Worked example · GC content with unknown bases

ATGNNGC → 7 letters, 2 unknown (N) → 5 known bases
G + C among them: G, G, C = 3
GC content = 3 ÷ 5 = 0.6 (dividing by 7 would wrongly give 0.43)

Step by step. The letters are A, T, G, N, N, G, C. Removing the two Ns leaves A, T, G, G, C: five known bases. Of those, three are G or C. 3 ÷ 5 = 0.6. The function from Strings, lists and functions would have divided by all seven letters: 3 ÷ 7 = 0.4286, rounded to 0.43.

What the answer means. An N could be any base, so it should not count either for or against GC. Dividing by 7 treats every unknown base as if it were A or T, which pulls the answer down. In real data this matters: sequencing reads and genome assemblies can contain runs of Ns where a base could not be read, and an honest GC content leaves them out.

A whole folder of files

pathlib handles file paths on any operating system. First split the practice file into one file per sequence, then loop over the folder and collect a summary table.

A path is the address of a file, such as seqs/seq1.fasta. Windows writes paths with backslashes and macOS and Linux with forward slashes; pathlib builds the right form for whichever system the code runs on, so the same notebook works for a colleague on another machine.

Python
from pathlib import Path
import pandas as pd

out = Path("seqs")
out.mkdir(exist_ok=True)
for name, s in seqs.items():
    (out / f"{name}.fasta").write_text(f">{name}\n{s}\n")
print(sorted(p.name for p in out.glob("*.fasta")))

rows = []
for path in sorted(out.glob("*.fasta")):
    for name, s in seqtools.read_fasta(path).items():
        rows.append({"file": path.name, "id": name, "length": len(s),
                     "gc": round(seqtools.gc_content(s), 3)})

summary = pd.DataFrame(rows)
print(summary)
summary.to_csv("sequence_summary.csv", index=False)
['seq1.fasta', 'seq2.fasta', 'seq3.fasta', 'seq4.fasta'] file id length gc 0 seq1.fasta seq1 36 0.444 1 seq2.fasta seq2 23 0.087 2 seq3.fasta seq3 21 0.952 3 seq4.fasta seq4 27 0.481

What this does

  • Path("seqs") names a folder, and out.mkdir(exist_ok=True) creates it. exist_ok=True means "no error if it is already there", so the cell can be run twice.
  • The first loop writes one file per sequence. out / f"{name}.fasta" joins the folder and a file name with /, which pathlib understands as "inside". write_text writes a header line and the sequence; \n is the character that ends a line.
  • out.glob("*.fasta") finds every file in the folder whose name ends in .fasta; * stands for any characters. p.name is the file name without the folder. sorted gives a fixed order, since the operating system does not promise one.
  • The second loop reads each file with your read_fasta and appends one dictionary per sequence to the list rows: file name, identifier, length and GC content.
  • pd.DataFrame(rows) turns the list into a table, with the dictionary keys as column names. to_csv(…, index=False) saves it without the 0 to 3 row numbers, which are not data.

Reading the output

The first line confirms four files were written. The table has one row per file, and its lengths and GC values are the same as Biopython's in Sequences with Biopython: 36, 23, 21 and 27 bases, and 0.444, 0.087, 0.952 and 0.481. Splitting and re-reading has lost nothing.

A Path also knows the parts of its own name. For seqs/seq1.fasta, path.name is seq1.fasta, path.stem is seq1 and path.suffix is .fasta, which is a tidy way to turn file names into sample names. pandas and most other libraries accept a Path wherever they accept a file name as text.

Paths like seqs are relative: they are read from the notebook's current working folder, which Path.cwd() shows. If code that worked yesterday now finds no files, a different working folder is the usual cause. And glob("*.fasta") looks only in the folder itself; rglob("*.fasta") also searches every folder inside it, which suits results that are filed one folder per sample.

Your own toolkit: modules, files and errors · LSMA free course The practice FASTA file with four records is split into four files in a folder called seqs. A loop reads each file and adds one row to a list; the list becomes the summary DataFrame, which is saved as sequence_summary.csv. 1 · ONE FILE 2 · SPLIT 3 · LOOP 4 · COLLECT practice_sequences.fasta seqs/ seq1.fasta seq2.fasta seq3.fasta seq4.fasta rows seq1, 36, 0.444 seq2, 23, 0.087 seq3, 21, 0.952 seq4, 27, 0.481 summary a 4 × 4 table then saved as sequence_summary.csv
The pattern you will reuse for any folder of results: find the files, read each one, add one row per item to a list, and build the table once at the end. Schematic, not to scale.

Building a list of dictionaries and turning it into a DataFrame at the end is the standard pattern for collecting results in a loop.

The reason is speed and simplicity. Adding a row to a list is quick; adding a row to a DataFrame makes pandas copy the whole table each time, which becomes slow for thousands of files. The same pattern works for anything you collect in a loop, such as one quality-control summary per sequencing sample.

Check your functions

assert stops with an error if a statement is false. A few checks against answers you worked out by hand catch mistakes early.

Python
assert seqtools.reverse_complement("AAGC") == "GCTT"
assert seqtools.gc_content("GGCC") == 1.0
print("all checks passed")
all checks passed

What this does

Each assert line compares a function's result with an answer worked out by hand. AAGC reversed is CGAA, and complemented is GCTT; GGCC is all G and C, so its GC content is 1.0. If either comparison were false, Python would stop with an AssertionError at that line and the final print would never run. Seeing all checks passed therefore means every check above it held. Good checks include the awkward cases too: a sequence with N, an empty result, an invalid letter. These small checks are called tests, and keeping them next to your functions means every later change is checked automatically.

Be careful when a test compares decimals. Computers store most decimal fractions approximately, so 0.1 + 0.2 == 0.3 is False in Python: the sum is 0.30000000000000004. A test that compares a calculated float with == can therefore fail even when the code is right. math.isclose(a, b) asks whether two numbers are equal to within a tiny tolerance, and is the safer comparison for anything that involved division. The two checks above are safe, because both answers are exact.

Add a test each time you find a bug, one that fails on the old code and passes on the fixed version. That way the same bug cannot quietly return. Testing AI-written code, in Module 3, takes this idea further, with tests kept in their own file and run all at once.

Go deeper: from assert lines to a test suite

As a project grows, the checks move into their own file and a test runner runs them all. A popular runner is pytest, installed with pip install pytest. You write a file whose name starts with test_, such as test_seqtools.py, and inside it functions whose names also start with test_, each holding one or more plain assert lines. Typing pytest in a terminal finds every such function, runs it, and reports which passed and which failed, showing the values that differed.

pytest can also check that an error is raised. Inside a test function, the line with pytest.raises(ValueError): followed by an indented seqtools.reverse_complement("ATGX") passes only if that call raises a ValueError; if the function quietly returns a value, the test fails. That is exactly the check the Module 1 project asks for, written in two lines.

Key point

If you edit seqtools.py after importing it, restart the notebook (or use importlib.reload(seqtools)) so Python loads the new version.

This happens because Python imports each module only once per session and remembers it; a second import seqtools does nothing. So if you fix a bug in the file and rerun your tests without restarting, they still run the old code, and a test can pass or fail for the wrong reason. Restarting is the safer habit, because it also clears any old values left in memory, as in Setting up Python and Jupyter.

Further reading · Module 1The Python Tutorial: the official tutorial, for anything you want explained a second way. · All reading for this course

Check your understanding

1. What is a Python module?

2. What is the GC content of "GCNN" with the new function?

3. What does try … except ValueError do?

Up next

Project

A sequence and sample report

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.