LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Lesson 3 of 8

Module 1 · Python basics

NumPy, pandas and files

NumPy, pandas and files

NumPy, pandas and files

NumPy adds fast arrays of numbers and pandas adds tables. Almost every data notebook starts by importing both, then reading a file.

⏱ 11 min readModule 1 · Python basicsFree
Lesson at a glance4 steps · about 11 min
  1. NumPy arrays
  2. pandas DataFrames
  3. Columns, filters and categories
  4. Missing values
Files for this lessonSave each file under the name shown here, next to your notebook, or upload it to Colab (folder icon › Upload). Your browser may give the download a different name; rename it before you run the code.
airway_samples.csvThe airway sample sheet: 8 samples, donor and treatment

NumPy arrays

An array is like a list of numbers, but maths applies to every element at once.

With a plain Python list, you would write a loop to double each number. With an array, one operation does it, and it runs fast because NumPy hands the work to compiled code underneath. This "whole array at once" style is called vectorised code. It matters when a table holds expression values for 20,000 genes: one line replaces a loop of 20,000 steps.

In Strings, lists and functions you met lists, which can hold anything: text, numbers, even other lists. An array is stricter. Every element has the same type, usually all whole numbers or all floats, and that uniformity is what lets NumPy work so quickly. If you build an array from a mix of numbers and text, NumPy turns everything into text, and arithmetic on it then fails, so a stray word in a column of numbers is worth catching early.

Python
import numpy as np
import pandas as pd

counts = np.array([120, 340, 56, 980])
print(counts * 2)
print(counts[counts > 100])
[ 240 680 112 1960] [120 340 980]

What this does

  • import numpy as np and import pandas as pd load the two libraries under short nicknames. np and pd are a convention almost every Python user follows, so code you find elsewhere will use them too.
  • np.array([120, 340, 56, 980]) turns a list of four read counts into an array.
  • counts * 2 multiplies every element by 2. With a plain list, * means something different: it repeats the list, so doubling a list gives eight items, not four doubled numbers. That difference catches many beginners.
  • counts > 100 compares every element with 100 and gives an array of True and False, one per element. Putting that inside square brackets, counts[…], keeps only the elements where the answer was True. This is called boolean indexing, or using a mask.

Reading the output

NumPy prints arrays in square brackets without commas, lined up in columns, which is why there are extra spaces in the first line. [ 240 680 112 1960] is each count doubled. The comparison with 100 gave True, True, False, True, so the second line keeps 120, 340 and 980 and drops 56. In real data the same one-line pattern removes genes with too few reads to analyse.

Arrays also come with summary methods. counts.sum() gives 1496, the total of the four counts, and counts.mean() gives 374.0, their average. NumPy's functions work element by element in the same way as the arithmetic: np.log2(counts) takes the base-2 logarithm of each count, the scale you met in the first lesson. Real count tables contain zeros, and the logarithm of 0 is minus infinity, so analysts usually add a small constant first, written np.log2(counts + 1). The added 1 is called a pseudocount; it keeps zeros at 0 on the log scale and barely changes large counts.

Go deeper: when a slice is not a copy

Slicing a list makes a new list, but slicing a NumPy array makes a view: a new name that looks at the same numbers in memory. Changing the view changes the original. After first_two = counts[:2], the line first_two[0] = 0 also changes counts, which now prints as [ 0 340 56 980]. NumPy works this way to save memory and time when arrays hold millions of values.

When you want an independent copy, ask for one: first_two = counts[:2].copy(). Boolean indexing, as in counts[counts > 100], always makes a copy, so changing its result leaves counts alone. A related point holds for every Python object: b = a never copies anything. It gives the same list or array a second name, so a change made through either name shows up in both.

pandas DataFrames

A DataFrame is a table with named columns. Read the course sample sheet (upload airway_samples.csv to Colab first):

The file is a CSV, short for comma-separated values: a plain text file in which each line is one row and commas separate the columns. The first line holds the column names. Any spreadsheet program can save one, and almost every analysis tool can read one.

Why a table rather than separate lists? Because each row describes one sample, and the columns must stay lined up. If you sorted a list of sample names and forgot to sort the matching list of treatments, every label would silently shift to the wrong sample. A DataFrame moves whole rows together, so filtering or sorting can never pull one column out of step with the others.

If the file is not in the notebook's working folder, read_csv stops with a FileNotFoundError. In Colab, uploaded files sit in the session's folder and disappear when the session ends, so you upload again next time. On your own computer, the simplest habit is to keep the notebook and its data files in the same folder.

Python
samples = pd.read_csv("airway_samples.csv")
print(samples.shape)                 # (rows, columns)
print(samples.head(3))
print(pd.crosstab(samples["cell"], samples["dex"]))
(8, 3) sample cell dex 0 SRR1039508 N61311 untrt 1 SRR1039509 N61311 trt 2 SRR1039512 N052611 untrt dex trt untrt cell N052611 1 1 N061011 1 1 N080611 1 1 N61311 1 1

What this does

  • pd.read_csv("airway_samples.csv") reads the file from the notebook's working folder into a DataFrame called samples.
  • samples.shape gives the size of the table as a pair of numbers: rows first, then columns. There are no brackets after shape because it is a stored property of the table, not a function to run.
  • samples.head(3) shows the first three rows. With no number, head() shows five.
  • pd.crosstab(…) builds a cross-tabulation: a table that counts how many rows have each combination of two columns, here donor (cell) against treatment (dex). samples["cell"] picks out one column by name.

Reading the output

(8, 3): eight rows and three columns. In the head output, the column names are on top and the numbers 0, 1 and 2 down the left are the index, the row labels pandas adds, counted from 0 like string positions. Each SRR… code is a run accession: the identifier of one sequencing run in a public archive. In the cross-tabulation, each row is a donor and each column a treatment; every cell holds 1, so each donor has exactly one treated and one untreated sample. pandas sorts the labels as text, character by character, which is why N052611 (whose second character is 0) comes before N61311 (whose second character is 6).

Two more quick looks are worth running. samples.columns.tolist() lists the column names, here ['sample', 'cell', 'dex'], which helps when a name contains an unexpected space or capital letter. samples.dtypes shows the type pandas chose for each column. Text columns are shown as object; a column you expected to be numbers that shows as object usually contains a stray word or symbol somewhere.

NumPy, pandas and files · LSMA free course The airway sample sheet drawn as a grid with three named columns, sample, cell and dex, and eight rows numbered 0 to 7 by the index. The first three rows show their values. The dex column is outlined as one column, a Series. A box notes that samples.shape is (8, 3). 1 · A DATAFRAME: ROWS, COLUMNS AND AN INDEX index sample cell dex 0 SRR1039508 N61311 untrt 1 SRR1039509 N61311 trt 2 SRR1039512 N052611 untrt 3 4 5 6 7 Column names sample · cell · dex samples["dex"] one column, a Series 8 rows, one per sample numbered 0 to 7 by the index samples.shape (8, 3) · rows, columns
Every DataFrame has the same three parts: named columns, rows, and an index of row labels down the left. Rows 3 to 7 are drawn empty here only to save space. Schematic, not to scale.

These are the eight samples of the airway experiment, a published RNA-seq study of how cells respond to a steroid: airway smooth muscle cells from four donors (cell), each treated with the steroid dexamethasone (trt) or left untreated (untrt). Every donor has one of each, a paired design.

A paired design is valuable because people differ. Cells from one donor may simply express a gene more highly than cells from another. When every donor provides both a treated and an untreated sample, the analysis can compare each treated sample with its own donor's untreated one, so the donor-to-donor differences cancel out and the treatment effect is easier to see. That is why the crosstab check matters: a missing or duplicated sample would break the pairs.

Both libraries have papers you can cite. NumPy: Harris et al. (2020), "Array programming with NumPy", Nature 585, 357–362 (doi:10.1038/s41586-020-2649-2). pandas: McKinney (2010), "Data structures for statistical computing in Python", Proceedings of the 9th Python in Science Conference. The airway experiment was published by Himes et al. (2014) in PLOS ONE, a study of how airway smooth muscle cells respond to glucocorticoids such as dexamethasone. Cite the original paper for the data and the software papers for the tools. Also give the version of each library (pd.__version__, np.__version__), because defaults change between releases; pandas, for example, changed several behaviours in its 2.0 release in 2023.

Columns, filters and categories

Python
treated = samples[samples["dex"] == "trt"]          # keep matching rows
print(len(treated))

samples["dex"] = pd.Categorical(samples["dex"], categories=["untrt", "trt"])
print(samples["dex"].cat.categories.tolist())
4 ['untrt', 'trt']

What this does

  • samples["dex"] == "trt" compares every value in the dex column with trt, giving True or False for each of the eight rows. The double == asks "is it equal?"; a single = would try to store a value.
  • samples[…] with that True/False column inside keeps only the True rows: the same mask idea as with the NumPy array above, now applied to whole rows. The result is a new, smaller DataFrame called treated; samples itself is unchanged.
  • len(treated) counts its rows.
  • pd.Categorical(…, categories=["untrt", "trt"]) converts the column to the categorical type with the groups in the order given, and the result replaces the old column.
  • .cat.categories.tolist() reads back the group order as a plain list.

Reading the output

4: four of the eight samples were treated, one per donor, as the design says. ['untrt', 'trt'] confirms the order: untreated first.

A categorical column has a fixed order of groups. Putting untreated first makes it the reference, so comparisons read “treated versus untreated”.

Without this step, pandas would order the groups alphabetically, and because t comes before u, trt would become the reference. Results would then describe untreated versus treated, and every change would have the opposite sign: a gene that rises with dexamethasone would appear to fall. Setting the reference on purpose avoids that confusion.

Filters can combine conditions. To keep the treated sample from one donor, write samples[(samples["dex"] == "trt") & (samples["cell"] == "N61311")], which keeps exactly one row. The & means "and" for whole columns, and | means "or". Each condition needs its own round brackets, because & is applied before ==. Writing the English word and here does not work: pandas stops with a ValueError, since it cannot reduce a whole column of True and False to one answer. To keep rows matching any of several values, samples["cell"].isin(["N61311", "N052611"]) is shorter than a chain of |.

Missing values

pandas marks missing data as NaN and, unlike R, skips it by default: pd.Series([4, None, 8]).mean() returns 6.0. Count missing values with df.isna().sum() so you know what is being skipped.

NaN stands for "not a number". In the example, a Series is a single column of values, which is what each column of a DataFrame is. The mean is worked out from the two values that are present: (4 + 8) ÷ 2 = 6.0. The missing value is left out of both the total and the count. In R the same calculation returns NA unless you ask for missing values to be removed. Skipping is convenient, but it is silent: if half a column were missing, the mean would rest on half the data and nothing would warn you. df.isna() marks every cell as True if missing, and .sum() adds them up column by column, because True counts as 1 and False as 0.

Once you know where the gaps are, you choose what to do, and the choice depends on why the values are missing. df.dropna() removes every row that has a missing value, which is safe when only a few rows are affected and they are missing for unrelated reasons. df.fillna(0) replaces gaps with 0, which is right only when missing truly means zero, for example a gene with no reads recorded. Filling a missing measurement with 0 or with the mean, when it was simply not taken, invents data and makes groups look more alike than they are.

A related pitfall is how missing values are written in the file. By default, read_csv treats empty cells and common markers such as NA, N/A and NaN as missing. Other markers, such as a dash or the word missing, are read as ordinary text, so a numeric column turns into text and isna() reports nothing. If you know a file uses such a marker, pass it to read_csv with na_values=["-"].

Key point

Check shape, head() and a quick crosstab every time you load data. Most problems show up in these three lines.

Each check catches a different kind of problem. shape catches a file that was cut short or read with the wrong separator, which often shows up as a single column. head() catches column names that were read as data, or numbers that arrived as text. The crosstab catches problems in the design itself: if one donor showed 2 treated and 0 untreated, the labels would be wrong somewhere, and the paired comparison would fail. The next lesson deliberately loads a sheet with exactly these kinds of mistakes.

Further reading · Module 1The Python Tutorial: the official tutorial, for anything you want explained a second way. · All reading for this course

Check your understanding

1. What does np.array([1, 2, 3]) * 10 return?

2. What does df.shape return for the sample sheet?

3. What does pd.Series([2, None, 4]).mean() return?

Up next

Lesson 4

Cleaning and combining data

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.