Lesson 3 of 8
Module 1 · Python basics
NumPy adds fast arrays of numbers and pandas adds tables. Almost every data notebook starts by importing both, then reading a file.
NumPy arrays
An array is like a list of numbers, but maths applies to every element at once.
With a plain Python list, you would write a loop to double each number. With an array, one operation does it, and it runs fast because NumPy hands the work to compiled code underneath. This "whole array at once" style is called vectorised code. It matters when a table holds expression values for 20,000 genes: one line replaces a loop of 20,000 steps.
In Strings, lists and functions you met lists, which can hold anything: text, numbers, even other lists. An array is stricter. Every element has the same type, usually all whole numbers or all floats, and that uniformity is what lets NumPy work so quickly. If you build an array from a mix of numbers and text, NumPy turns everything into text, and arithmetic on it then fails, so a stray word in a column of numbers is worth catching early.
import numpy as np import pandas as pd counts = np.array([120, 340, 56, 980]) print(counts * 2) print(counts[counts > 100])
What this does
import numpy as npandimport pandas as pdload the two libraries under short nicknames. np and pd are a convention almost every Python user follows, so code you find elsewhere will use them too.np.array([120, 340, 56, 980])turns a list of four read counts into an array.counts * 2multiplies every element by 2. With a plain list,*means something different: it repeats the list, so doubling a list gives eight items, not four doubled numbers. That difference catches many beginners.counts > 100compares every element with 100 and gives an array of True and False, one per element. Putting that inside square brackets,counts[…], keeps only the elements where the answer was True. This is called boolean indexing, or using a mask.
Reading the output
NumPy prints arrays in square brackets without commas, lined up in columns, which is why there are extra spaces in the first line. [ 240 680 112 1960] is each count doubled. The comparison with 100 gave True, True, False, True, so the second line keeps 120, 340 and 980 and drops 56. In real data the same one-line pattern removes genes with too few reads to analyse.
Arrays also come with summary methods. counts.sum() gives 1496, the total of the four counts, and counts.mean() gives 374.0, their average. NumPy's functions work element by element in the same way as the arithmetic: np.log2(counts) takes the base-2 logarithm of each count, the scale you met in the first lesson. Real count tables contain zeros, and the logarithm of 0 is minus infinity, so analysts usually add a small constant first, written np.log2(counts + 1). The added 1 is called a pseudocount; it keeps zeros at 0 on the log scale and barely changes large counts.
Go deeper: when a slice is not a copy
Slicing a list makes a new list, but slicing a NumPy array makes a view: a new name that looks at the same numbers in memory. Changing the view changes the original. After first_two = counts[:2], the line first_two[0] = 0 also changes counts, which now prints as [ 0 340 56 980]. NumPy works this way to save memory and time when arrays hold millions of values.
When you want an independent copy, ask for one: first_two = counts[:2].copy(). Boolean indexing, as in counts[counts > 100], always makes a copy, so changing its result leaves counts alone. A related point holds for every Python object: b = a never copies anything. It gives the same list or array a second name, so a change made through either name shows up in both.
pandas DataFrames
A DataFrame is a table with named columns. Read the course sample sheet (upload airway_samples.csv to Colab first):
The file is a CSV, short for comma-separated values: a plain text file in which each line is one row and commas separate the columns. The first line holds the column names. Any spreadsheet program can save one, and almost every analysis tool can read one.
Why a table rather than separate lists? Because each row describes one sample, and the columns must stay lined up. If you sorted a list of sample names and forgot to sort the matching list of treatments, every label would silently shift to the wrong sample. A DataFrame moves whole rows together, so filtering or sorting can never pull one column out of step with the others.
If the file is not in the notebook's working folder, read_csv stops with a FileNotFoundError. In Colab, uploaded files sit in the session's folder and disappear when the session ends, so you upload again next time. On your own computer, the simplest habit is to keep the notebook and its data files in the same folder.
samples = pd.read_csv("airway_samples.csv")
print(samples.shape) # (rows, columns)
print(samples.head(3))
print(pd.crosstab(samples["cell"], samples["dex"]))What this does
pd.read_csv("airway_samples.csv")reads the file from the notebook's working folder into a DataFrame calledsamples.samples.shapegives the size of the table as a pair of numbers: rows first, then columns. There are no brackets aftershapebecause it is a stored property of the table, not a function to run.samples.head(3)shows the first three rows. With no number,head()shows five.pd.crosstab(…)builds a cross-tabulation: a table that counts how many rows have each combination of two columns, here donor (cell) against treatment (dex).samples["cell"]picks out one column by name.
Reading the output
(8, 3): eight rows and three columns. In the head output, the column names are on top and the numbers 0, 1 and 2 down the left are the index, the row labels pandas adds, counted from 0 like string positions. Each SRR… code is a run accession: the identifier of one sequencing run in a public archive. In the cross-tabulation, each row is a donor and each column a treatment; every cell holds 1, so each donor has exactly one treated and one untreated sample. pandas sorts the labels as text, character by character, which is why N052611 (whose second character is 0) comes before N61311 (whose second character is 6).
Two more quick looks are worth running. samples.columns.tolist() lists the column names, here ['sample', 'cell', 'dex'], which helps when a name contains an unexpected space or capital letter. samples.dtypes shows the type pandas chose for each column. Text columns are shown as object; a column you expected to be numbers that shows as object usually contains a stray word or symbol somewhere.
These are the eight samples of the airway experiment, a published RNA-seq study of how cells respond to a steroid: airway smooth muscle cells from four donors (cell), each treated with the steroid dexamethasone (trt) or left untreated (untrt). Every donor has one of each, a paired design.
A paired design is valuable because people differ. Cells from one donor may simply express a gene more highly than cells from another. When every donor provides both a treated and an untreated sample, the analysis can compare each treated sample with its own donor's untreated one, so the donor-to-donor differences cancel out and the treatment effect is easier to see. That is why the crosstab check matters: a missing or duplicated sample would break the pairs.
Both libraries have papers you can cite. NumPy: Harris et al. (2020), "Array programming with NumPy", Nature 585, 357–362 (doi:10.1038/s41586-020-2649-2). pandas: McKinney (2010), "Data structures for statistical computing in Python", Proceedings of the 9th Python in Science Conference. The airway experiment was published by Himes et al. (2014) in PLOS ONE, a study of how airway smooth muscle cells respond to glucocorticoids such as dexamethasone. Cite the original paper for the data and the software papers for the tools. Also give the version of each library (pd.__version__, np.__version__), because defaults change between releases; pandas, for example, changed several behaviours in its 2.0 release in 2023.
Columns, filters and categories
treated = samples[samples["dex"] == "trt"] # keep matching rows print(len(treated)) samples["dex"] = pd.Categorical(samples["dex"], categories=["untrt", "trt"]) print(samples["dex"].cat.categories.tolist())
What this does
samples["dex"] == "trt"compares every value in thedexcolumn with trt, giving True or False for each of the eight rows. The double==asks "is it equal?"; a single=would try to store a value.samples[…]with that True/False column inside keeps only the True rows: the same mask idea as with the NumPy array above, now applied to whole rows. The result is a new, smaller DataFrame calledtreated;samplesitself is unchanged.len(treated)counts its rows.pd.Categorical(…, categories=["untrt", "trt"])converts the column to the categorical type with the groups in the order given, and the result replaces the old column..cat.categories.tolist()reads back the group order as a plain list.
Reading the output
4: four of the eight samples were treated, one per donor, as the design says. ['untrt', 'trt'] confirms the order: untreated first.
A categorical column has a fixed order of groups. Putting untreated first makes it the reference, so comparisons read “treated versus untreated”.
Without this step, pandas would order the groups alphabetically, and because t comes before u, trt would become the reference. Results would then describe untreated versus treated, and every change would have the opposite sign: a gene that rises with dexamethasone would appear to fall. Setting the reference on purpose avoids that confusion.
Filters can combine conditions. To keep the treated sample from one donor, write samples[(samples["dex"] == "trt") & (samples["cell"] == "N61311")], which keeps exactly one row. The & means "and" for whole columns, and | means "or". Each condition needs its own round brackets, because & is applied before ==. Writing the English word and here does not work: pandas stops with a ValueError, since it cannot reduce a whole column of True and False to one answer. To keep rows matching any of several values, samples["cell"].isin(["N61311", "N052611"]) is shorter than a chain of |.
Missing values
pandas marks missing data as NaN and, unlike R, skips it by default: pd.Series([4, None, 8]).mean() returns 6.0. Count missing values with df.isna().sum() so you know what is being skipped.
NaN stands for "not a number". In the example, a Series is a single column of values, which is what each column of a DataFrame is. The mean is worked out from the two values that are present: (4 + 8) ÷ 2 = 6.0. The missing value is left out of both the total and the count. In R the same calculation returns NA unless you ask for missing values to be removed. Skipping is convenient, but it is silent: if half a column were missing, the mean would rest on half the data and nothing would warn you. df.isna() marks every cell as True if missing, and .sum() adds them up column by column, because True counts as 1 and False as 0.
Once you know where the gaps are, you choose what to do, and the choice depends on why the values are missing. df.dropna() removes every row that has a missing value, which is safe when only a few rows are affected and they are missing for unrelated reasons. df.fillna(0) replaces gaps with 0, which is right only when missing truly means zero, for example a gene with no reads recorded. Filling a missing measurement with 0 or with the mean, when it was simply not taken, invents data and makes groups look more alike than they are.
A related pitfall is how missing values are written in the file. By default, read_csv treats empty cells and common markers such as NA, N/A and NaN as missing. Other markers, such as a dash or the word missing, are read as ordinary text, so a numeric column turns into text and isna() reports nothing. If you know a file uses such a marker, pass it to read_csv with na_values=["-"].
Key point
Check shape, head() and a quick crosstab every time you load data. Most problems show up in these three lines.
Each check catches a different kind of problem. shape catches a file that was cut short or read with the wrong separator, which often shows up as a single column. head() catches column names that were read as data, or numbers that arrived as text. The crosstab catches problems in the design itself: if one donor showed 2 treated and 0 untreated, the labels would be wrong somewhere, and the paired comparison would fail. The next lesson deliberately loads a sheet with exactly these kinds of mistakes.
Further reading · Module 1The Python Tutorial: the official tutorial, for anything you want explained a second way. · All reading for this course
Check your understanding
1. What does np.array([1, 2, 3]) * 10 return?
2. What does df.shape return for the sample sheet?
3. What does pd.Series([2, None, 4]).mean() return?
Up next
Lesson 4
Cleaning and combining data