Guided project
Module 1 · Python basics · Self-checked
Bring the course together in one notebook of your own, using the course files. Work through the six tasks, check each against its checkpoint, and tick it off. Nothing is submitted.
Research question
Is the airway sample sheet complete and balanced, and what can you say about the four practice sequences: their length, GC content and longest open reading frame?
How to approach this
The question has two halves, and each half reuses work from earlier lessons. "Complete and balanced" is about the sample sheet: every sample present once, and every donor with one treated and one untreated sample. "Length, GC content and longest open reading frame" is about the sequences. Work through the tasks in order, because later ones build on earlier ones: your seqtools module grows in Task 3, is tested in Task 4 and feeds the figure in Task 5.
This is the shape of most real analyses, small or large: check the samples, measure something about each one, test the code that did the measuring, show the result, and write down what it means. The sample sheet comes first for a reason. If the sheet is wrong, every later number is attached to the wrong sample, however good the code that produced it.
- Start a fresh notebook and put the course files in its working folder. Write a short text cell at the top stating the question in your own words.
- Keep one step per cell, with a text cell above each saying what it does. This is the write-up you will need in Task 6, built as you go.
- Check against each checkpoint before moving on. If a number differs, the cause is almost always in the step you have just done, so it is cheapest to find now.
- Restart and run all at the end of each task, not only at the end of the project, so a cell that depends on something you deleted is caught early.
One new idea appears in Task 3. An open reading frame, or ORF, is a stretch of DNA that starts with the start codon ATG and runs, codon by codon, to the first stop codon in the same frame. It is a candidate for a stretch that codes for a protein. Because codons are read in threes, the same letters can be grouped in three different ways, depending on whether you start at the first, second or third letter. These are the three reading frames of the strand, shown below.
An ORF is a candidate, not proof of a gene. In a long stretch of random DNA, an ATG followed a few codons later by a stop occurs by chance all the time, because three of the 64 codons are stops. That is why genome annotation tools usually ignore ORFs below a minimum length and look for other evidence, such as a similar protein in a database. In this project, the sequences are short and made up, so every ORF counts; in Task 6, it is worth saying that a real analysis would set a minimum.
Clean the sample sheet
NumPy, pandas and files · Cleaning and combining data
- Load airway_samples_messy.csv and list every problem you find.
- Fix them in code, with a comment for each fix.
Hint. Cleaning and combining data used four short checks to find the problems: the table's shape, a count of each value with missing values included, the distinct values of a text column, and a count of duplicated rows. Write down what each check shows before you change anything. Then fix the problems in one chained step, and confirm the result with the crosstab from NumPy, pandas and files. Fill a missing value only from a record you can cite.
Why it matters. Task 1 is the only task that changes data rather than measuring it, so it is where a careless step does the most damage. Keep messy untouched and build clean from it, as in Cleaning and combining data, so the two can always be compared.
Checkpoint: 8 rows remain; every donor has one treated and one untreated sample; clean.equals(pd.read_csv("airway_samples.csv")) is True.
Summarise the sequences
Sequences with Biopython · Your own toolkit: modules, files and errors
- Use your seqtools module to build a table of id, length and GC content.
- Check your GC values against Biopython's
gc_fraction.
Hint. The "list of dictionaries, then one DataFrame" pattern from Your own toolkit: modules, files and errors gives the table. For the comparison, Sequences with Biopython printed both GC values side by side; here you could add Biopython's value as a second column and compare the two columns. Round both to the same number of places before comparing.
A pitfall. Your seqtools.read_fasta returns plain strings, while Biopython's records hold Seq objects. gc_fraction accepts either, but if you mix the two readers, make sure you are comparing the same sequences in the same order: building both columns inside one loop, keyed by identifier, is the simplest way.
Checkpoint: lengths 36, 23, 21 and 27; GC 0.444, 0.087, 0.952 and 0.481; both methods agree.
Find the longest open reading frame
Strings, lists and functions · Sequences with Biopython
- Add a
longest_orf(seq)function to seqtools.py: in each of the three reading frames, find runs from ATG to the first stop codon (TAA, TAG or TGA), and return the longest. - Return an empty string when there is none.
Hint. Break the problem into small steps. A frame is defined by its starting position, 0, 1 or 2, and a slice of three letters at a time (Strings, lists and functions) gives you each codon in that frame. Walk along the codons; when you meet ATG, keep reading until you reach a stop codon, and keep that stretch as a candidate. Compare each candidate's length with the longest found so far. Decide two details before you write code, and check them against the checkpoint: whether the stop codon is included in the result, and what happens to an ATG that never reaches a stop before the sequence ends. Real ORF finders also search the reverse complement; this task does not ask for that.
Test as you build. Before running longest_orf on the course file, try it on sequences small enough to check by eye: ATGTAA should give the whole six letters; CCATGTAA should find the same ORF in a later frame; ATGCCC, which never reaches a stop, tests the rule you chose for an ATG with no stop; most ORF finders return nothing in that case. If those three behave, the course sequences usually will too.
Checkpoint: seq1 gives the whole 36-base sequence; seq4 gives 18 bases, ATGCGTACGTTAGCCTAG; seq2 and seq3 have none.
Test your toolkit
Your own toolkit: modules, files and errors
- Write at least five
assertchecks, including an empty ORF result and a sequence with N. - Check that a sequence with an X raises a
ValueError.
Hint. Good tests use answers you can work out by hand: short sequences whose GC content, reverse complement or ORF you can see at a glance. The checkpoints of Tasks 2 and 3 give you several more. For the X case, recall the try and except ValueError pattern from Your own toolkit: modules, files and errors: the test should pass only when the error is raised, and fail if the function quietly returns a value.
Why both kinds of test. Checks on ordinary input show the code gives the right answer when all is well. Checks on awkward input, such as an empty ORF, an N or an X, show it fails clearly when all is not well. Code that returns a plausible number for nonsense input is more dangerous than code that stops, because nobody notices.
Checkpoint: every check passes after “Restart and run all”.
One clear figure
Figures with matplotlib and seaborn
- A bar chart of GC content per sequence, with a dashed line at 0.5 and labelled axes.
- Save it at 300 dpi.
Hint. GC content is a proportion, one value per sequence, so a bar starting at zero is a fair way to show it. Use the summary table from Task 2 as the data, and the figure-and-axes pattern from Figures with matplotlib and seaborn. matplotlib can draw a horizontal reference line across an axes, with a dashed line style. Label the y axis with what the number is, and save before you show the figure.
Reading your figure. The dashed line at 0.5 marks equal shares of G and C on one side and A and T on the other, so it turns four bars into an immediate comparison: which sequences are GC-rich and which AT-rich, the same labels as in Strings, lists and functions. Order the bars by identifier so the figure matches the table; sorting by height is also fair, as long as the caption says so.
Checkpoint: seq3 is the tallest bar and seq2 the shortest; the file opens and reads well at a small size.
Write it up
Setting up Python and Jupyter · Your own toolkit: modules, files and errors
- Add text cells: what you did, what you found, and what you would check next.
- Export the summary table to CSV.
Hint. Answer the research question directly, in two or three sentences per half, using the numbers from your own outputs. "What you would check next" can be a limitation you noticed, such as searching only one strand. For the export, Your own toolkit: modules, files and errors saved a table without its row numbers. Then restart and run all one last time.
What a good write-up contains. State the answer first, then the evidence: for example, the number of samples and donors after cleaning, and the sequence with the longest ORF and its length. Name every fix you made to the sample sheet and where the missing value came from. Keep claims to what the outputs show: four short, made-up sequences cannot tell you anything about real genes, and saying so is part of a careful report.
Checkpoint: someone else could rerun your notebook from top to bottom and get the same table and figure.
Go deeper: why short ORFs mean little
In random DNA with all four bases equally common, 3 of the 64 codons are stops, so each codon has a 3 ÷ 64 ≈ 4.7% chance of ending a reading frame. On average a stop turns up about every 64 ÷ 3 ≈ 21 codons. The chance of reading 100 codons in a row without meeting one is 61 ÷ 64 multiplied by itself 100 times: about 0.008, under 1%.
That is why short ORFs appear everywhere by chance, and why real gene finders ignore ORFs below a minimum length and weigh other evidence, such as similarity to known proteins. The 36-base ORF in seq1 would never count as a gene on its own evidence; the practice sequences are short on purpose, so that you can check every answer by hand. Real tools also search all six frames, three on each strand, and must decide what to report when a shorter ORF sits inside a longer one and shares its stop codon.
Take it further
Put the notebook and seqtools.py on GitHub with a short README. Then try the same functions on a FASTA file of real sequences downloaded from NCBI, or go on to Module 4: machine learning.
GitHub is a website for storing and sharing code, which keeps a history of every change you make. A README is a short text file shown on the project's front page: say what the project does, which files it needs and how to run it. Do not upload data you are not allowed to share; the course files here are made up or already public.
By the end of this project you can load, clean, combine, measure, test and plot data in Python, and you have a small module of your own. Module 2, Figures that tell the truth, takes the plotting step from Figures with matplotlib and seaborn and Task 5 and makes it the subject. It starts with choosing the right chart for a question, then shows distributions, comparisons with their uncertainty, colour that every reader can see, honest axes and multi-panel figures, before ending with omics figures and export for publication. The tables you can now build are the data those figures are drawn from.
Further reading · Module 1The Python Tutorial: the official tutorial, for anything you want explained a second way. · All reading for this course
Up next
Lesson 1
Choosing the right chart