Lesson 6 of 12
Module 8 · AI for biology
See what a convolutional network does with one-hot DNA. Train a small network on synthetic sequences with a planted motif, and read the motif back from its top filter.
From features you choose to features the model learns
In Turning sequences into numbers: features for machine learning and A first sequence classifier, evaluated honestly you decided what the model would see: composition, a window on the N-terminus, k-mer counts. Deep learning takes a different route. You give the network the raw sequence, one-hot encoded, and it learns its own features during training. This lesson opens up the simplest such network for DNA and shows that what it learns can be read: its first layer turns out to hold motifs.
A neural network is a chain of layers. Each layer takes numbers in, multiplies them by its weights, adds them up, and passes the result on, usually after a simple non-linear step. Training shows the network examples with known labels, measures how wrong its predictions are with a loss (a single number that is small when predictions are good), and nudges every weight a little in the direction that lowers the loss. Repeating this over all the training data once is called an epoch. Here we focus on what one kind of layer does with DNA.
The non-linear step matters more than it looks. Without it, a chain of layers that each multiply and add would collapse into a single weighted sum, no more powerful than the logistic regression of A first sequence classifier, evaluated honestly. The usual choice, ReLU (rectified linear unit), simply sets negative numbers to zero. That small bend lets later layers combine signals in ways a weighted sum cannot, such as "motif A present and motif B absent".
What convolution does on one-hot DNA
One-hot DNA is a grid with 4 rows (A, C, G, T) and one column per base, with a single 1 in each column. A convolutional filter is a small grid of weights with the same 4 rows and a fixed width, say 7 bases. The network slides the filter along the sequence one base at a time. At each position it multiplies the filter's weights by the one-hot values beneath them and adds everything up. Because only one row in each column is 1, that sum is simply the filter's weight for whichever base is actually there, added over the 7 positions. The result is a match score at every position: high where the sequence looks like what the filter "wants", low elsewhere.
This is the same idea as a position weight matrix (PWM), the classic way to describe a transcription factor binding site: a score for each base at each position of the motif. The difference is that nobody writes the filter. It starts as random numbers and training shapes it. A filter that ends up scoring a useful pattern highly is, in effect, a learned motif.
Because one filter is reused at every position, it needs very few weights: 4 × 7 = 28, plus one bias. The network below has 8 such filters, 232 numbers in all, and a final layer of 9, so it learns from just 241 parameters. A layer that gave every one of the 400 one-hot values its own weight for every output would need far more, and would have to learn GATTACA separately at each of the 94 positions where it could start. Sharing the filter across positions builds in the fact that a motif means the same wherever it sits, which is why convolution suits sequence.
After the filter has scored every position, max pooling keeps only the highest score. That turns the question into "does this pattern occur anywhere in the sequence?", which is what you want when a motif can sit at any position.
How to read this figure
Read the top panel as a table: the letters across the top are the bases, and in each column a single square is filled in the row of that base. The orange box is the filter, here placed over GATTACA. The bottom panel has one bar for each place the filter's left edge can start; the bar's height is the match score there. It is low almost everywhere and tall at one position, the window over GATTACA. Max pooling reduces the whole row of bars to that single tallest value.
Worked example · scoring two windows with a learned filter
Take the filter the network learns in block 3 below. Its weights for the bases of GATTACA are G 0.90, A 0.82, T 0.96, T 0.88, A 0.84, C 0.83, A 0.74. Score the window GATTACA, then the one-base change GATTAGA.
GATTAGA: position 6 is G, weight −0.67 instead of C's 0.83
5.97 − 0.83 + (−0.67) = 4.47
Step by step. In each column of one-hot DNA only the row of the base present is 1, so each position contributes just one weight: the filter's weight for that base at that position. For GATTACA, add the seven weights for G, A, T, T, A, C, A, which gives 5.97. For GATTAGA, six positions are the same; at position 6 the base is G, whose weight there is −0.67. Take away C's 0.83 and add −0.67: 5.97 − 0.83 − 0.67 = 4.47. The filter also adds a constant, its bias, which is the same for every window, so we leave it out.
What the answer means. One wrong base cost 1.5 points, a quarter of the score. A random 7-base stretch would get mostly negative weights and score far lower. So the filter behaves like a motif scanner: it gives the highest scores to an exact match, a little less to a near match, and much less to anything else. This is how the network can learn motifs that tolerate some variation, as real binding sites do.
Go deeper: log-odds scores and how rare a motif is
A classic position weight matrix is usually written as log-odds: at each position, the score for a base is the logarithm to base 2 of (its frequency in real sites ÷ 0.25), where 0.25 is its chance in random DNA. A base that is always present scores log2(4) = 2 bits. A perfectly fixed 7-base motif therefore has a top score of 7 × 2 = 14 bits.
That number has a meaning. A 7-base word turns up by chance once in 4 × 4 × 4 × 4 × 4 × 4 × 4 = 16,384 positions on one strand. In a 100-base sequence there are 94 starting positions, so a random sequence contains GATTACA on the plus strand about 94 ÷ 16,384 times, roughly 0.6%. This is why the lesson's negatives had to be checked. Real binding sites are less strict, carry fewer bits and turn up far more often, which is why a motif match alone predicts binding poorly.
The experiment: a planted motif
To see whether a network really learns motifs, we need data where we know the answer. So the data here is synthetic, made up by the code, and is not from any genome. We make 2,000 random DNA sequences, each 100 bases long. In half of them, the positives, we plant the made-up motif GATTACA at a random position. The other half, the negatives, are random and checked to contain no GATTACA. A network that tells them apart must have found the motif.
Synthetic data play the role of a positive control, the known-answer sample you run alongside a laboratory experiment. If the method cannot find a motif you planted yourself, it will not find a real one. If it can, you have checked the code and the reading-out step before trusting them on data where the answer is unknown.
The code uses PyTorch, a deep learning library that is already installed in Colab. It runs in a few seconds on an ordinary processor, so you do not need a GPU.
import random
import torch
random.seed(4)
torch.manual_seed(4)
MOTIF = "GATTACA" # made-up motif, planted for teaching
BASES = "ACGT"
def random_dna(n=100):
return "".join(random.choice(BASES) for _ in range(n))
def make_example(positive):
s = random_dna()
while MOTIF in s: # keep negatives truly motif-free
s = random_dna()
if positive:
p = random.randrange(len(s) - len(MOTIF))
s = s[:p] + MOTIF + s[p + len(MOTIF):]
return s
def one_hot(s): # 4 rows (A, C, G, T) x length
return torch.tensor([[1.0 if b == x else 0.0 for b in s] for x in BASES])
labels = [i % 2 for i in range(2000)] # half positive
seqs = [make_example(l) for l in labels]
X = torch.stack([one_hot(s) for s in seqs]) # shape 2000 x 4 x 100
y = torch.tensor(labels, dtype=torch.float32)
print(X.shape, seqs[1][:40])What this does
random.seed(4)andtorch.manual_seed(4)fix the random numbers, so the sequences and the network's starting weights are the same on every run.random_dnajoins 100 bases picked at random.make_examplekeeps drawing until the sequence has no GATTACA, then, for a positive, pastes the motif over 7 bases at a random start positionp.one_hotbuilds the 4-row grid: for each of A, C, G and T, a row with 1.0 where the sequence has that base.labelsalternates 0, 1, 0, 1, so exactly half are positive.torch.stackpiles the 2,000 grids into one block of numbers called a tensor.
Reading the output
torch.Size([2000, 4, 100]) is the shape of the data: 2,000 sequences, 4 rows each, 100 positions long. The letters are the first 40 bases of the second sequence, a positive. Its GATTACA is somewhere further along; nothing in the first 40 bases gives it away, which is the point: the network has to search.
import torch.nn as nn
model = nn.Sequential(
nn.Conv1d(4, 8, kernel_size=7), # 8 filters, each 7 bases wide
nn.ReLU(),
nn.AdaptiveMaxPool1d(1), # best match anywhere in the sequence
nn.Flatten(),
nn.Linear(8, 1)) # one score per sequence
Xtr, ytr, Xte, yte = X[:1600], y[:1600], X[1600:], y[1600:]
opt = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.BCEWithLogitsLoss()
for epoch in range(5):
for i in range(0, 1600, 64):
opt.zero_grad()
loss = loss_fn(model(Xtr[i:i + 64]).squeeze(1), ytr[i:i + 64])
loss.backward()
opt.step()
with torch.no_grad():
acc = ((model(Xte).squeeze(1) > 0) == (yte == 1)).float().mean()
print(f"epoch {epoch + 1} loss {loss.item():.3f} test accuracy {acc:.3f}")What this does
nn.Sequentialchains the layers.nn.Conv1d(4, 8, kernel_size=7)is the convolution: it reads 4 rows and has 8 filters, each 7 bases wide, so it learns 8 possible motifs at once.nn.ReLU()sets negative scores to zero.nn.AdaptiveMaxPool1d(1)keeps each filter's best score along the sequence.nn.Linear(8, 1)weighs the 8 best scores into one number per sequence; above 0 means "positive".- The first 1,600 sequences are for training and the last 400 for testing.
Adamis the routine that nudges the weights;lr=0.01sets how big each nudge is.BCEWithLogitsLossis the usual loss for yes/no labels. - The inner loop takes 64 sequences at a time (a batch): predict, measure the loss, work out how each weight should change (
loss.backward()), and change them (opt.step()). After each epoch the test accuracy is measured with training switched off (torch.no_grad()).
Reading the output
Each line is one pass through the 1,600 training sequences. The loss is from the last batch of the epoch, and it falls from 0.518 to 0.037 as the network gets better at the training data. The test accuracy, on 400 sequences it never trained on, rises from 95.7% to 98.8%, so about 395 of the 400 are classified correctly after five epochs. Your numbers may differ slightly with a different PyTorch version; the pattern should not.
Watch the two numbers together on real data. A training loss that keeps falling while the test score stalls or drops is the usual sign of overfitting: the network has started to learn its training examples rather than the pattern. Note also that this code looks at the test set after every epoch. That is fine for watching, but if you used it to choose when to stop training, the test set would be helping to tune the model, the trap from Measuring a model honestly: confusion matrix, ROC, precision, recall and imbalance. In real work, hold back a separate validation set from the training data for decisions like this, and touch the test set once at the end.
Looking inside: is the top filter the motif?
The network gets the answers right, but has it found GATTACA, or some shortcut? Each of the 8 filters feeds the final layer with its own weight. The filter with the largest positive weight is the one whose high score most pushes a sequence towards "positive". For each of its 7 positions we ask which base has the highest weight.
w_out = model[4].weight[0] # how much each filter pushes "positive"
top = int(w_out.argmax())
filt = model[0].weight[top].detach() # 4 x 7 grid of weights
print("top filter", top, "output weight", round(w_out[top].item(), 2))
print("strongest base at each position:",
"".join(BASES[int(filt[:, j].argmax())] for j in range(7)))
for b, row in zip(BASES, filt):
print(b, " ".join(f"{v:5.2f}" for v in row.tolist()))What this does
model[4]is the fifth layer, the finalLinear; itsweight[0]holds one weight per filter.argmax()picks the filter with the largest.model[0].weight[top]is that filter's 4 × 7 grid of weights.detach()takes a plain copy, without the training bookkeeping.- For each column
j,filt[:, j].argmax()finds the row with the highest weight, andBASES[...]turns it into a letter. The loop then prints the whole grid, one row per base.
Reading the output
Filter 7 has the largest output weight, 1.38. Reading the strongest base at each of its positions spells GATTACA: the planted motif, letter for letter. In the grid, each column has exactly one clearly positive weight (between 0.74 and 0.96), and the other three are negative. The network was never told the motif; it found it because that was the only thing separating the two classes. The figure below draws this grid.
How to read this figure
Columns are the 7 positions of the filter and rows are the 4 bases. Each cell holds the weight the filter gives that base at that position. Find the orange cell in each column and read the letters from left to right: G, A, T, T, A, C, A. Every other cell is negative, so any other base at that position lowers the score. This is a learned position weight matrix.
Reading first-layer filters as motifs became standard with DeepBind (Alipanahi et al. 2015, Nature Biotechnology) and DeepSEA (Zhou & Troyanskaya 2015, Nature Methods), both trained on measured binding and chromatin data. To name a learned filter, researchers compare it with a motif database such as JASPAR (Castro-Mondragon et al. 2022, Nucleic Acids Research) using a tool such as Tomtom (Gupta et al. 2007, Genome Biology). Report such a match as a resemblance with a score, not as proof that the network learned that factor. Filters are often partial or shifted, and several can share one motif.
What this does and does not show
The experiment shows the principle: a convolutional network's first-layer filters can learn sequence motifs without being told them, and you can read them back. Real models of this kind are trained on measured data, such as which DNA regions are open or bound by a protein in a cell type, and their filters often resemble known transcription factor motifs.
Real life is messier than a planted motif, in four ways:
- Filters are not always tidy. When this code was run with other seeds, the top filter sometimes came out shifted, spelling ATTACAT: six bases of the motif plus one weak, uninformative position. Several filters may also split one motif between them.
- Real motifs are fuzzy. Binding sites vary, so learned filters show preferences rather than one perfect word, and must be compared with motif databases to be named.
- Deeper layers are harder to read. Later layers combine motifs into patterns of spacing and order, and they do not line up with sequence as neatly as the first layer.
- Shortcuts are possible. If positives and negatives differ in something unintended, such as GC content or length, a network will happily learn that instead. Here we built the negatives to differ only in the motif; with real data you must check.
One more simplification is built into this code: it only reads one strand. In a cell, a transcription factor can bind its site on either strand, so a motif may appear as GATTACA on one strand and as its reverse complement, TGTAATC, on the other. Models for real genomic data handle this either by scanning both strands or by adding the reverse complement of every training sequence, with the same label. You met reverse complements in Module 5, in Trimming and filtering, where read 2 of a pair comes from the opposite strand to read 1.
Key point
A convolutional filter is a learned motif scanner. On one-hot DNA, each filter scores every window, and max pooling asks whether the pattern occurs anywhere. Reading the top filters is a direct way to check what the network has learned.
A second example: imagine training the same network on promoters versus random DNA and finding that the top filter spells TATAAA. That matches the TATA box, a well-known promoter element, which is reassuring. If instead the top filter were just a run of Gs and Cs, it would suggest the network had learned that promoters are GC-rich, which is true but is a much weaker kind of knowledge, and a sign to check the negatives.
Further reading · Module 8PyTorch documentation: Conv1d: the convolution layer used here, with its options. · PyTorch tutorials: learn the basics: tensors, models and training loops, step by step. · All reading for this course
Check your understanding
1. A convolutional filter 7 bases wide slides along one-hot DNA. What does it produce at each position?
2. Why does the network use max pooling after the convolution?
3. A network trained on promoters versus random DNA has a top filter that is just a run of Gs and Cs. What should you suspect?
Up next
Lesson 7
Protein language models and embeddings