LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 73 lessons

Lesson 1 of 7

Module 2 · Biological data and databases

File formats you will meet

File formats you will meet

File formats you will meet

Biological data travels as plain-text files with agreed layouts. Learn to recognise the five you will meet most often, what each one holds, and how to read a line of each.

⏱ 15 min readModule 2 · Biological data and databasesFree
Lesson at a glance6 steps · about 15 min
  1. Why formats matter
  2. FASTA: just the sequence
  3. GenBank: a sequence with its story
  4. GFF: annotation on its own
  5. VCF: the differences
  6. PDB and mmCIF: the shape

Module 1, Understanding omics, showed what each omics layer measures, from the genome to the metabolome. This module is about where that data is kept and how to read it: the files it travels in, the public databases that hold it, the identifiers that link one record to another, and the papers that describe it. Module 3 then teaches the command line, for handling those files at scale. Knowing the files comes first, because every database and every tool reads and writes them.

Why formats matter

Databases and tools pass data around as files, and each kind of data has an agreed layout, called a file format: rules for what goes on each line and what each symbol means. Because the rules are shared, a file written in one lab can be read by any program anywhere. Almost all the formats here are plain text, so you can open them in a text editor. You do not need every rule; you need to recognise each format, know what it holds, and know where mistakes happen.

You have already met one format. FASTQ, from Module 1, holds raw sequencing reads with a quality score for every base. This lesson covers the five you will meet next: one for sequences, two for annotation, one for variants and one for 3D structures.

Being plain text does not mean being small. A human genome in FASTA is about three billion letters, so most large files are stored compressed, usually with gzip, and end in .gz. The content is unchanged; it only needs unpacking, or a tool that reads compressed files directly. A file name is a hint, not a guarantee: a file called .txt may be FASTA, and a .vcf file may be truncated or empty. Looking at the first few lines is the quickest way to know what you really have, and it catches most mix-ups before they reach an analysis.

File formats you will meet: FASTA, GenBank, GFF, VCF, PDB · LSMA free course Four cards. Sequence is held in FASTA, with FASTQ for raw reads. Annotation, meaning where the genes and other features lie, is held in GFF and GTF. Variants, the differences from a reference, are held in VCF. 3D structure, the position of every atom, is held in PDB and mmCIF. A bar under the first two cards shows that a GenBank record holds both a sequence and its annotation in one file. WHICH FORMAT HOLDS WHAT Sequence Annotation Variants 3D structure the letters themselves where features lie what differs in a sample where every atom sits ATGGCTTCAGGT CTGAAAGCCATC ref ACGTTGCA alt ACGCTGCA FASTA GFF · GTF VCF PDB · mmCIF FASTQ for raw reads one line per feature one line per variant one line per atom GenBank · sequence and annotation in one file All of these are plain text.
Four kinds of biological data, and the formats that hold them. A GenBank record bundles the first two together. Schematic.

How to read this figure

Read each card from top to bottom: the kind of data, a cartoon of it, then the format names. The cartoons show raw letters, a line of DNA with two features marked, a sample differing from the reference at one base, and a chain of atoms in space. The bar spans two cards because a GenBank record holds both a sequence and its annotation. Whenever you meet a file, first ask which card it belongs to.

FASTA: just the sequence

FASTA is the simplest and most common format. Each record has two parts. The first line, the header, starts with the symbol >, followed by an identifier and, optionally, a free-text description. Every line after it, up to the next >, is sequence. The same format holds DNA, RNA or protein; only the letters change.

An example record · FASTA

>example_1 invented example gene, mRNA (teaching record)
ATGGCTTCAGGTCTGAAAGCCATCGAGCTTCGTAACCTGGAAGCTGTGCCAGAGTTCAAG
GACCTGTTCAGCGAGATCCGTAAAGCCTTCGAGGTGCTGAACCAGGCTGATTCCACCAAG

One header line, then sequence wrapped at 60 letters. A file can hold millions of records.

An identifier is the short label that names the record; most programs use the first word after >. DNA uses A, C, G and T, with N for an unknown base; protein uses the one-letter amino-acid codes, such as M for methionine. FASTA carries no quality scores and no information about where genes lie, which is why other formats exist.

In How sequencing works you saw that FASTQ stores four lines per read; FASTA is what remains when the quality line is dropped, and it is the usual format for reference genomes, transcript sets and protein collections. A file holding many records is sometimes called multi-FASTA; the layout is the same, one header after another. Two pitfalls are common. First, identifiers should be unique: if two records share the same first word, many tools keep only one, or stop with an error. Second, because programs cut the header at the first space, a long description is often lost on the way through a pipeline, so put anything a program must keep into the identifier itself. Line length also varies between sources, 60 or 80 letters or one long line; well-written tools accept all of them.

Go deeper: the small print of a FASTA file

The format takes its name from FASTA, a sequence-search program published by Pearson and Lipman in 1988; the program's input layout outlived it. Three conventions are worth knowing before you meet real files.

First, letters beyond A, C, G, T and N are allowed. The IUPAC codes stand for uncertain bases: R means A or G, Y means C or T, and so on. Some tools reject them, so check before running. Second, many genome FASTA files mix capital and small letters. Small letters mark repeats found by masking software, a style called soft-masking; the bases are real, only flagged. Some tools ignore the case, others skip small letters entirely. Third, in protein FASTA a * often marks a stop codon at the end of a translated sequence. None of this is visible in the header, so look at a few lines of sequence too.

GenBank: a sequence with its story

A GenBank flat file is the full record behind a sequence in the GenBank database (The main databases: sequences, genomes, proteins, structures). "Flat file" means one plain-text page per record. It opens with labelled lines such as LOCUS, DEFINITION and ACCESSION (the record's identifier), then the FEATURES table, which is the annotation, and finally ORIGIN, the sequence itself. The European archive uses a sister layout, the EMBL format, with the same content.

An example record · GenBank feature excerpt

FEATURES         Location/Qualifiers
    gene         1..1200
                 /gene="EXMP1"
    CDS          101..1000
                 /gene="EXMP1"
                 /product="example protein"
                 /translation="MASGLKAIEL..."

An invented gene, EXMP1: the gene spans bases 1 to 1200, the coding part 101 to 1000.

Each feature is a named stretch of the sequence: a gene, an exon, a coding sequence. CDS stands for coding sequence, the part that is translated into protein, from the start codon to the stop codon. Lines beginning with / are qualifiers, notes on a feature such as the gene symbol or the protein sequence. complement(101..1000) would mean the opposite strand, and join(…) joins pieces such as exons. You will read a full record in Reading a gene record and a protein record.

The advantage of a GenBank record is that it keeps a sequence and its meaning together, which suits single genes, mRNAs, plasmids and small genomes such as those of viruses. Two details trip up new readers. Feature positions count from the first base of that record, not from the start of a chromosome, so 101..1000 in one record says nothing about genome coordinates. And the symbols < and > in a location, as in <1..1200, mean the feature is partial: it continues beyond the stretch of sequence that was submitted.

GFF: annotation on its own

A genome is too big for one GenBank page, so its annotation is kept apart from the sequence, in GFF (General Feature Format, now GFF3). The sequence sits in FASTA; the GFF file says where features are, one per line, in nine tab-separated columns.

seqidsourcetypestartendscorestrandphaseattributes
contig_7examplegene1500118200.-.ID=gene01;Name=EXMP1
contig_7exampleexon1780118200.-.Parent=tx01

How to read these lines

This is an example record with invented positions on an invented sequence called contig_7. Read each row left to right:

  • seqid is the chromosome or contig (a stretch of assembled sequence); source is the program or group that made the annotation.
  • type is the kind of feature (gene, mRNA, exon, CDS). start and end count the first base as 1 and include both ends.
  • score is a confidence value; a dot means "no value". strand is + or -, the DNA strand the feature is read from. phase, for CDS lines only, says how many bases to skip before the first full codon.
  • attributes are labelled notes. ID names the feature and Parent links it to a larger one: exon to transcript, transcript to gene. That is how GFF builds a gene from many lines.

GTF is a close relative with the same nine columns; Ensembl offers gene sets in both. BED is a simpler region format, but it counts the first base as 0, so its starts are one lower than in GFF.

Keeping annotation apart from sequence has a practical reason: a genome assembly may stay fixed for years while its gene set is improved several times a year, and each update needs only a new GFF file. The price is that the two files must match. The seqid in column 1 has to be written exactly as the identifier in the FASTA headers; if the FASTA says chr7 and the GFF says 7, a tool finds no features at all. A GFF3 file starts with the line ##gff-version 3, which is a quick way to tell it from GTF. In GTF the attributes are written differently, as pairs such as gene_id "…"; transcript_id "…";, so a tool built for one may misread the other.

VCF: the differences

VCF (Variant Call Format), from Module 1, stores only where a sample differs from a reference genome, the agreed standard sequence for a species. It has meta-information lines starting ##, which define every abbreviation; one header line starting #CHROM; then one line per variant.

An example record · VCF

##fileformat=VCFv4.2
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT SAMPLE1
chr7  1000 .  G   A   50   PASS   DP=40 GT:AD  0/1:22,18

An example record with an invented position. Columns are separated by tabs in a real file.

CHROM and POS give the place, counting from 1. ID holds a known identifier, such as a dbSNP number starting rs, or a dot. REF is the reference base and ALT the base seen instead. QUAL is the caller's confidence on the Phred scale used in FASTQ: 20 means a 1 in 100 chance of error, 30 a 1 in 1,000. FILTER says PASS if the call passed the pipeline's checks, and INFO holds notes such as read depth (DP). FORMAT names the per-sample fields: GT is the genotype (0 is the reference allele, 1 the first alternative) and AD the read count for each allele.

A few more genotype patterns are worth recognising. 1/1 means both copies carry the alternative allele (homozygous), and ./. means no genotype could be called for that sample. A bar instead of a slash, as in 0|1, means the genotype is phased: it is known which copy carries which allele. When a site has more than one alternative, ALT lists them separated by commas, and a genotype of 1/2 means one copy of each. The allele depths link back to Genomics: dividing the alternative reads by the total gives the variant allele fraction you met there. Large VCF files are usually compressed with bgzip and come with a small index file ending in .tbi or .csi, which lets tools jump straight to one region instead of reading the whole file.

Worked example · reading a VCF line

An example line from an invented sample, with an invented position. What change does it describe?

chr7  2045  .  CTT  C  60  PASS  DP=52  GT:AD  0/1:27,25
REF CTT → ALT C  → TT deleted, after the C at 2045
GT 0/1 → heterozygous  ·  25 ÷ 52 = 0.48 of reads carry it

Step by step. Start with REF and ALT. The reference has three bases, CTT, but the sample has only C, so two bases are missing: this is a deletion, one kind of indel. VCF always keeps one unchanged base in front of an indel, called the anchor base, so that neither REF nor ALT is ever empty. The C at position 2045 is that anchor; the deleted bases are the two Ts at 2046 and 2047. Next, genotype 0/1 means one chromosome copy carries the reference and the other the deletion. The allele depths agree: 27 reference reads and 25 deletion reads, 52 in all, matching DP, and 25 ÷ 52 ≈ 0.48 is close to the 0.5 expected for a heterozygous variant (Module 1). QUAL 60 means about a one-in-a-million chance of error.

What the answer means. The sample carries a two-base deletion on one of its two copies. Inside a coding sequence, losing two bases would shift the reading frame, the grouping of bases into codons, and scramble the protein from there on. Compare CFTR F508del, written c.1521_1523del: it removes three bases, a multiple of three, so the frame is kept and one amino acid is lost. The VCF line gives the change; its effect on a gene needs the annotation from GFF or GenBank.

VCF was introduced with the 1000 Genomes Project and described in Danecek and colleagues (2011, Bioinformatics). The current specification is maintained by the Global Alliance for Genomics and Health (GA4GH) alongside the SAM and BAM formats, and has moved on from version 4.2 to later versions; the first line of each file says which one applies.

A caution when comparing files: in repetitive sequence, the same indel can be written at more than one position. Deleting one T from TTTT gives the same result whichever T you remove. Tan, Abecasis and Kang (2015, Bioinformatics) set out the standard fix, called normalisation: trim and shift each indel to its leftmost position. Normalise both files before matching variants.

PDB and mmCIF: the shape

A 3D structure is a list of atoms, each with a position in space. The classic PDB format, named after the Protein Data Bank (The main databases: sequences, genomes, proteins, structures), writes one atom per line in fixed columns.

An example record · PDB and mmCIF

ATOM      1  N   MET A   1      12.104   8.523  -3.311  1.00 20.15       N

An example atom line with invented coordinates. The same atom in mmCIF, shortened:

loop_
_atom_site.group_PDB
_atom_site.id
_atom_site.label_atom_id
_atom_site.label_comp_id
_atom_site.Cartn_x
_atom_site.Cartn_y
_atom_site.Cartn_z
ATOM 1 N MET 12.104 8.523 -3.311

Read the atom line left to right: atom 1, a nitrogen (N) in methionine (MET), chain A, residue 1. A chain is one separate molecule in the structure; a residue is one amino acid in it. Then come x, y and z in ångströms (a ten-billionth of a metre, about the size of an atom), the occupancy, and the B-factor, a measure of how uncertain the atom's position is. In AlphaFold database files this column holds the prediction's confidence score instead, so check where a file came from.

Not every line describes protein. Atoms of other molecules bound to the structure, such as a drug, a metal ion or water, are written on lines starting HETATM instead of ATOM. Many structures solved by X-ray crystallography have no hydrogen atoms listed, because they are too light to be placed reliably, and flexible loops are sometimes missing altogether. A gap in the residue numbers is therefore normal, not a broken file.

The fixed columns run out of room for very large structures, so the archive now uses mmCIF (also called PDBx/mmCIF) as its standard. It is the same atom list, but each column is named by a label such as _atom_site.Cartn_x, so nothing depends on counting characters. You will rarely read these by eye; viewers such as Mol* draw the shape for you.

The limits of the old layout are concrete: an atom number has room for only five digits and a chain name for one character, so a large complex such as a ribosome cannot fit in a single classic PDB file. Many tools still read the old format, and smaller structures are often still offered in both. If a program refuses an mmCIF file, look for a PDB-format version before converting it yourself.

Counting
  • GFF, VCF, GenBank, PDB: from 1
  • BED: from 0
Genome build
  • A build is one release of a reference, such as GRCh37 or GRCh38
  • Never mix builds
Chromosome names
  • chr7 in some files, 7 in others
  • Tools may silently match nothing
Strand
  • VCF alleles are on the + strand
  • GFF gives the strand in column 7

Key point

Each format answers one question: what is the sequence (FASTA), where are the features (GenBank, GFF), how does this sample differ (VCF), or what shape is the molecule (PDB, mmCIF). Know which question a file answers before you open it.

A second example: a colleague sends a GFF file of genes and a VCF file of variants and asks which variants fall in genes. Before any tool is run, check that both files name chromosomes the same way and use the same genome build. If the GFF says chr17 and the VCF says 17, a program may report no overlaps at all, and the empty answer will look like a real result.

Further reading · Module 2NCBI: a sample GenBank record, every field explained. · The GFF3 specification. · The VCF and related format specifications. · PDB-101: guide to understanding PDB data. · All reading for this course

Check your understanding

1. You have a human genome sequence and want a separate file saying where every gene and exon lies on it. Which format fits best?

2. A VCF line reads REF = A, ALT = AGG, GT = 1/1. What does it describe?

3. A GFF file lists genes on 'chr17' and a VCF file lists variants on '17'. A tool reports that no variants fall in any gene. What should you suspect first?

Up next

Lesson 2

The main databases: sequences, genomes, proteins, structures

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.