Lesson 4 of 9
Module 3 · Coding with AI assistants
Tests turn your checks into code that runs every time anything changes. pytest finds every function whose name starts with test_ and reports which pass.
From checks to tests
In Read before you run you checked the draft by printing four results and comparing them with the answers in the comments, by eye. A test is the same check written as code, so the computer does the comparing and can repeat it every time anything changes. pytest is the most widely used Python package for running tests. Each test is a short function containing an assert statement: assert followed by something that should be True. If it is True, nothing happens and the test passes; if it is False, the test fails and pytest reports what it found.
Why bother, when printing worked? Because code keeps changing. You will tidy a function, speed it up, or ask an assistant to add a feature, and each change can break something that used to work. A bug that returns after it was fixed, or appears in code that used to be right, is called a regression. Printed checks catch regressions only if you remember to rerun them and reread the output every time. Tests rerun in a second and say plainly which check failed.
%%writefile test_seqtools.py
import pytest
from seqtools import gc_content, reverse_complement, mean_quality
def test_gc_content_simple():
assert gc_content("GGCCAT") == pytest.approx(66.67, abs=0.01)
def test_gc_content_lowercase():
assert gc_content("ggccat") == pytest.approx(66.67, abs=0.01)
def test_reverse_complement():
assert reverse_complement("AAGC") == "GCTT"
def test_mean_quality_phred33():
assert mean_quality("IIII") == 40 # "I" is ASCII 73; 73 - 33 = 40
def test_empty_sequence_is_an_error():
with pytest.raises(ValueError):
gc_content("")What this does
%%writefile test_seqtools.pysaves the cell as a file. The name matters: pytest looks for files whose names start with test_, and inside them for functions whose names start with test_.import pytestloads pytest's helpers, and the next line imports the three functions to be tested from seqtools.py.test_gc_content_simplechecks the upper-case example.pytest.approx(66.67, abs=0.01)means "66.67, give or take 0.01". The true answer is 66.666…, which never equals 66.67 exactly, so an exact comparison would fail even for correct code.test_gc_content_lowercaseis the same check on "ggccat", the edge case that broke the draft.test_reverse_complementcompares the result with the string "GCTT" exactly; text either matches or it does not.test_mean_quality_phred33checks that "IIII" gives 40, with the reason written in the comment so a later reader knows where 40 came from.test_empty_sequence_is_an_errorworks the other way round.with pytest.raises(ValueError):says "the code inside this block should stop with a ValueError". The test passes only ifgc_content("")raises exactly that error.
Each test checks one behaviour and has a name that says which. When a test fails, its name alone tells you what is broken.
A good test reads like a small claim about the code: given this input, expect this output. Keep each test that short. If a single test checked case, the reverse complement and the quality score together, its failure would tell you only that something somewhere was wrong. Short, separate tests also make it easy to see what has not been tested yet: there is no test for N in this file, for example, even though the prompt in Asking well: prompts for code asked for N to stay N.
First, run the tests against the draft from Read before you run, saved as seqtools.py. The first cell below makes that copy.
In Colab, a line that starts with ! is passed to the computer's command line instead of to Python. cp copies a file, so this line makes a copy of the draft under the name the tests import. Then run pytest on the test file. The options shorten the report. -q (quiet) prints less detail; --tb=no turns off the traceback, the long listing of where each error happened; and -rf adds a one-line summary for each failed test at the end.
Running pytest with ! also starts a fresh Python process each time, so it always imports the file as it is now saved. That avoids a common notebook trap: once a module has been imported in a notebook, importing it again in the same session does not reload it, so you can edit seqtools.py and still be running the old version. In the notebook itself, restart the runtime, or use importlib.reload, after changing a module.
cp ai_draft.py seqtools.py
pytest -q --tb=no -rf test_seqtools.py
Reading the output
- The first line has one character per test, in the order they appear in the file: . for a pass and F for a failure. .FFFF means the first test passed and the other four failed. [100%] says all tests in the file have run.
- Each FAILED line names the file, then the test after ::, then the reason. For the lower-case test, the code returned 0.0 where 66.67 ± 0.01 was expected. For the reverse complement, it returned 'TTCG' where 'GCTT' was expected. For the quality, 73.0 instead of 40.
- The empty-sequence test failed differently. The code did stop with an error, but with ZeroDivisionError, not the ValueError the test asked for, so the test counts it as a failure.
- The last line gives the totals and the time taken. These are exactly the four bugs you found by reading in Read before you run, now found automatically.
Go deeper: many cases, one test
Writing a separate test function for every input gets repetitive. pytest can run one test on a whole table of cases with @pytest.mark.parametrize. Put @pytest.mark.parametrize("seq, expected", [("GGCCAT", 66.67), ("ggccat", 66.67), ("ATAT", 0.0), ("GGGG", 100.0)]) on the line above def test_gc(seq, expected):, and make the body assert gc_content(seq) == pytest.approx(expected, abs=0.01).
pytest then runs the test four times, once per pair, and reports each case separately, so a failure tells you exactly which input broke. Adding a new case is now one more pair in the list. Notice the two extra cases: 0% and 100% are boundary values, the extremes of what the function can return, and boundaries are where off-by-one and division mistakes tend to show up.
Fix, then test again
Each fix below is small, and each answers one failed test.
%%writefile seqtools.py
"""Small, tested tools for sequences and FASTQ quality strings."""
COMPLEMENT = {"A": "T", "T": "A", "G": "C", "C": "G", "N": "N"}
def gc_content(seq: str) -> float:
"""Percentage of G and C in a DNA sequence (case-insensitive)."""
if not seq:
raise ValueError("empty sequence")
seq = seq.upper()
return (seq.count("G") + seq.count("C")) / len(seq) * 100
def reverse_complement(seq: str) -> str:
"""Reverse complement of a DNA sequence; N stays N."""
return "".join(COMPLEMENT[base] for base in reversed(seq.upper()))
def mean_quality(qual: str) -> float:
"""Mean Phred quality of a FASTQ quality string (Phred+33 encoding)."""
return sum(ord(c) - 33 for c in qual) / len(qual)What this does
- The text in triple quotes at the top is a module docstring: one line saying what the file is for.
- COMPLEMENT is the base-pairing dictionary, now including "N": "N" so unknown bases pass through. Writing its name in capitals is a Python convention for a value that should not change.
gc_contentfirst checksif not seq:. An empty string counts as False in Python, so for empty input the function raisesValueError("empty sequence")before it can divide by zero. Thenseq.upper()turns the sequence into capitals, so "g" and "G" are counted alike.reverse_complementalso usesseq.upper(), andreversed(...)walks through the sequence from the last base to the first, so the complements are joined in reverse order: the missing step.mean_qualitysubtracts 33 from every character code before adding them up: the Phred+33 offset.- Each function now has type hints (seq: str, -> float) and a docstring, as the prompt in Asking well: prompts for code asked.
pytest -q test_seqtools.py
Reading the output
Five dots, five passes, and no failure summary because there is nothing to report. The test file did not change between the two runs; only the code did. That is the point of keeping tests: they are a fixed standard that any new version of the code must meet, whether you, a colleague or an assistant wrote it.
Passing tests show that the code works for the cases you tested, not for every case. For example, mean_quality("") would still divide by zero, because no test asks what an empty quality string should do, and a sequence containing a letter other than A, C, G, T or N would still stop reverse_complement with an error. Whether those need handling is a decision about your data; if they do, the next step is a new test.
Suppose you decide that an empty quality string, like an empty sequence, should be an error. Write the test first: a function test_empty_quality_is_an_error containing with pytest.raises(ValueError): and then mean_quality(""), the same shape as the empty-sequence test. Run pytest and watch it fail with a ZeroDivisionError. Only then add an if not qual: check to mean_quality, and run the tests again. Seeing the new test fail first matters: it proves that the test can fail, and so that its later pass means something.
This order, a failing test, then the smallest change that makes it pass, then a rerun of every test, is the same loop you followed above. It works just as well when an assistant writes the fix. You give it the failing test and its output, and the rerun of the whole file tells you whether the fix broke anything else.
Worked example · tests written by AI need checking too
assert mean_quality("IIII") == 73that test passes with the buggy code, because it encodes the same mistake
so: you supply the expected values, worked out by hand or from a trusted source
Step by step. If you ask an assistant to write the code and the tests together, it may make the same mistake in both. Suppose it forgets the Phred offset. Its code returns 73 for "IIII", and its test, worked out the same wrong way, expects 73. pytest compares 73 with 73, finds them equal and reports a pass. The test has checked only that the code agrees with itself.
What the answer means. A test is useful only if its expected value comes from somewhere independent of the code: your own working, a published example, or a trusted tool. Reading AI-written tests is quicker than reading code, so check every expected value in them before you rely on a green result.
The lesson’s warning that passing tests cover only the cases you tested is an old idea. Edsger Dijkstra put it in his Notes on Structured Programming (1970): testing can show that bugs are present, but never that they are absent.
One way to test more inputs is property-based testing, where you state a rule that must hold for every input and a tool generates hundreds of inputs to try. The Hypothesis library does this for Python (MacIver et al. 2019, Journal of Open Source Software). For example, the reverse complement of a reverse complement should give back the original sequence. Properties add to hand-worked examples; they do not replace them, because a property can hold even when every individual answer is wrong. Official pytest documentation: docs.pytest.org.
Key point
A test is only as good as its expected answer. Write tests for the cases most likely to break: empty input, lower case, unexpected characters, and the examples from your prompt.
A second example. For a function that converts gene IDs to gene symbols, a good test set would include one ID you looked up yourself in a database, one ID that does not exist, an ID with a version suffix such as .5 at the end, and an empty list. Each covers a different way the function could fail.
Further reading · Module 3Wilson et al. 2014: testing and review habits that apply to AI-written code too. · All reading for this course
Check your understanding
1. Which functions does pytest run?
2. Why use pytest.approx for 66.67?
3. An AI-written test passes with buggy code. How can that happen?
Up next
Lesson 5
Debugging with AI