The first step in nearly every next-gen sequence analysis pipeline is to map sequencing reads to a reference genome. In this tutorial we'll run some common mapping tools on TACC.
The world of read mappers seems to be settling down a bit after being a bioinformatics Wild West where there was a new gun in town every week that promised to be a faster and more accurate shot than the current record holder. Things seem to have reached the point where there is mainly a trade-off between speed, accuracy, and configurability among read mappers that have remained popular.There are over50 read mapping programs listed here. Each mapper has its own set of limitations (on the lengths of reads it accepts, on how it outputs read alignments, on how many mismatches there can be, on whether it produces gapped alignments, on whether it supports SOLiD colorspace data, etc.). As evidence of how things are settling down, we're going to (mainly) stick to just bowtie2 in this course.
Other read mappers
Previous versions of this class and tutorial have covered using bowtie and bwa. Please consult these tutorials for more specific information on each mapping program. The tutorial presented here is a trimmed down version of the bwa tutorial presented last year.
This tutorial covers the commands necessary to use several common read mapping programs.
- Become comfortable with the basic steps of indexing a reference genome, mapping reads, and converting output to
SAM/BAM
format for downstream analysis. - Usebowtie2andBWAto map reads from anE. coliIllumina data set to a reference genome and compare the output.
Please see theIntroduction to mapping presentationfor more details of the theory behind read mapping algorithms and critical considerations for using these tools correctly.
The tutorial currently available on the Lonestar cluster at TACC is as follows:
Modules also exist at the current time for: bwa,bowtie, andSHRiMP.
The following DNA sequencing read data files were downloaded from theNCBI Sequence Read Archivevia the correspondingEuropean Nucleotide Archive record. They are Illumina Genome Analyzer sequencing of a paired-end library from a (haploid)E. coliclone that was isolated from a population of bacteria that had evolved for 20,000 generations in the laboratory as part of a long-term evolution experiment (Barrick et al, 2009). Thereference genomeis the ancestor of thisE. colipopulation (strain REL606), so we expect the read sample to have differences from this reference that correspond to mutations that arose during the evolution experiment.
Transferring Data
We have already downloaded data files for this example and put them in the path:
$BI/gva_course/mapping/data
File Name | Description | Sample |
---|---|---|
| Paired-end Illumina, First of pair, FASTQ format | Re-sequenced E. coli genome |
| Paired-end Illumina, Second of pair, FASTQ format | Re-sequenced E. coli genome |
| Reference Genome in Genbank format | E. coli B strain REL606 |
The easiest way to run the tutorial is to copy this entire directory into a new folder called "bowtie2mapping" on your $SCRATCH space and then run all of the commands from inside that directory. See if you can figure out how to do that. When you're in the right place, you should get output like this from thels
command.
tacc:~$ lsNC_012967.1.gbk SRR030257_1.fastq SRR030257_2.fastq
If you need a little help but don't want the answer yet, click the triangle...
Remember that to copy an entire folder requires the use of the recursive (-r) option.
Still stuck? click here for the correct code
cdscp -r $BI/gva_course/mapping/data bowtie2MappingTutorialcd bowtie2MappingTutorialls
Useful commands
Often you will have general questions about your sequencing files that you want to answer before or after starting your actual analysis. Here we show you some very handy commands after a warning:
Beware the cat command when working with NGS data
NGS data can be quite large, a single lane of an Illumina Hi-Seq run generates 2 files each with 100s of millions of lines. Printing all of that can take an enormous amount of time and may crash your terminal long before it finishes. If you find yourself in a seemingly endless scroll of sequence (or anything else for that matter) remember ctrl+c will kill whatever command you just executed
Linux 1 liners
How to look at the top/bottom of files to determine their type
head * # will show the top 10 lines to give you an idea of the file type/structuretail * # will show the last 10 lines to determine if the file is a repetitive structure
How to count the total number of lines in a file
wc -l * # can be very useful to determine if it can be printed to screen or opened in a text editor
How to determine the total number of sequences in a fastq file
wc -l * # and then divide by 4 using the your knowledge of fastq files# ORgrep ^@SRR030257 SRR030257_1.fastq | wc -l# ORgrep --count ^@SRR030257 SRR030257_1.fastq
How to determine how long the reads are in a fastq file
sed -n 2p SRR030257_1.fastq | awk -F"[ATCGatcg]" '{print NF-1}'
Converting sequence file formats
Occasionally you might download a sequence or have it emailed to you by a collaborator in one format, and then the program that you want to use demands that it be in another format. Why do they have to be so picky?
Thebp_seqconvert.pl
script that is installed as part of Bioperl is one helpful utility for converting between many common sequence formats. On TACC, the Bioperl modules are installed, but the helper script isn't. So, we've put it in a place that you can run it from for your convenience. However, remember that any time that you use the script you must have the bioperl module loaded. We also took care of this for you when we edited your~/.profile_user
file in the Linux introduction.
Run the script without any arguments to get the help message:
bp_seqconvert.pl
Exercises
The fileNC_012967.1.gbk
is inGenbank
format. The filesSRR030257_*.fastq
are inFASTQ
format.
- Convert
NC_012967.1.gbk
toEMBL
format. Call the outputNC_012967.1.embl
.Does EMBL format have sequence features (like genes) annotated?
Click here for a hint
Try reading through the program help when you run the bp_seqconvert.pl without any options to see the syntax required
Sill need help?
bp_seqconvert.pl --from genbank --to embl < NC_012967.1.gbk > NC_012967.1.emblhead -n 100 NC_012967.1.embl
You might get an error or a warning like the following, even if the bp_seqconvert.pl script executed correctly so don't worry.
Use of uninitialized value in substitution (s///) at /opt/apps/bioperl/1.6.901/Bio/SeqIO/embl.pm line 777, <STDIN> line 164674.Use of uninitialized value in concatenation (.) or string at /opt/apps/bioperl/1.6.901/Bio/SeqIO/embl.pm line 779, <STDIN> line 164674.
From the head command, you should see that yes, EMBL files do maintain gene annotation features.
- Convert only the first 10,000 lines of
SRR030257_1.fastq
toFASTA
format.What information was lost by this conversion?
Click here if you need a hint
Remember use the | character to have the output of head feed into the bp_seqconvert.pl script.
Click here for the answer
head -n 10000 SRR030257_1.fastq | bp_seqconvert.pl --from fastq --to fasta > SRR030257_1.fastahead SRR030257_1.fastqhead SRR030257_1.fasta
The line of funny ASCII characters was lost. Remember, those are your "base quality scores". Many mappers will use the base quality scores to improve how the reads are aligned by not placing as much emphasis on poor bases.
Mapping with bowtie2
Bowtie2is a complete rewrite ofbowtie. It is currently the latest and greatest in the eyes of one very picky instructor (and his postdoc/gradstudent) in terms of configurability, sensitivity, and speed.
Create a fresh output directory named bowtie2. We are going to create a specific output directory for the bowtie2 mapperwithinthe directory that has the input files so that you can compare the results of other mappers if you choose to do the other tutorials.
Commands for making a directory and changing into it
mkdir bowtie2
Next, load thebowtie2module (we usemodule spider
to get the current name, which may not bebowtie/2.1.0
if you re-run this tutorial):
Click here for a hint without the answer
Remember in our earlier tutorial we discussed the use of lonestar's module
commands "spider" and "load" to install new functionality
click here for the answer without having to go back through the previous tutorial
module spider bowtiemodule load bowtie/2.1.0
OPTIONAL -- How to check what version of bowtie2 was loaded?
Here are a few of the possibilities that will work.
In this case all of these methods will work, that may not be true of all programs
bowtie2 --versionmodule listwhich bowtie2
Note that which
can be very useful for making sure you are running the executable that you think you are running, especially if you install your own programs. In particular make sure that the path matches up to what you expect. The most common situations arise from wanting to run a simplistically named script in your $HOME directory conflicting with something of the same name in the $BI directories.
Now convert the reference file from GenBank to FASTA using what you learned above. Name the new output fileNC_012967.1.fasta
and put it in the same directory asNC_012967.1.gbk
.
If you're stuck and want a hint without the answer...
Use the bp_seqconvert.pl script
Click here to get the answer or to check your command
bp_seqconvert.pl --from genbank --to fasta < NC_012967.1.gbk > NC_012967.1.fasta
Generally speaking, the first step in mapping is quite often indexing the reference file regardless of what mapping program is used. Put the output of this command into thebowtie
directory we created a minute ago. The command you need is:
bowtie2-build
Try typing this alone in the terminal and figuring out what to do from the help show just from typing the command by itself.
If you're stuck click here for an explanation of what arguments the command does need
The command requires 2 arguments. The first argument is the reference FASTA. The second argument is the "base" file name to use for the created index files. It will create a bunch of files beginning bowtie/NC_012967.1*.
Click here to check your work, or for the answer if needed
bowtie2-build NC_012967.1.fasta bowtie2/NC_012967.1
Take a look at your output directory usingls bowtie2
to see what new files have appeared. These files are binary files, so looking at them withhead
or tail isn't instructive and can cause issues with your terminal. If you insist on looking at them and your terminal begins behaving oddly, simply close it and log back into lonestar with a new terminal.
Why do so many different mapping programs create an index as a first step you may be wondering?
Like an index for a book (in the olden days before Kindles and Nooks), creating an index for a computer database allows quick access to any "record" given a short "key". In the case of mapping programs, creating an index for a reference sequence allows it to more rapidly place a read on that sequence at a location where it knows at least a piece of the read matches perfectly or with only a few mismatches. By jumping right to these spots in the genome, rather than trying to fully align the read to every place in the genome, it saves a ton of time.
Indexing is a separate step in running most mapping programs because it can take a LONG time if you are indexing a very large genome (like our own overly complicated human genome). Furthermore, you only need to index a genome sequence once, no matter how many samples you want to map. Keeping it as a separate step means that you can skip it later when you want to align a new sample to the same reference sequence.
Finally, map the reads! The command you need is:
bowtie2
Try reading the help to figure out how to run the command yourself. This command takes a while (~5 minutes). This is longer than we want to run a job on the head node (especially when all of us are doing it at once). In fact, TACC noticed the spike in usage last time we taught the class and we got in trouble.
So, you will want to submit the full job to the cluster like you learned in the introduction.
But first, try to figure out the command and start it in interactive mode. Remember these are paired-end reads.Usecontrol-c
to stop the job once you are sure it is running without an immediate error!Then, submit your command that is working to the TACC queue.
Submit to the TACC queue or run in an idev shell
Create a commands
file and use launcher_creator.py followed by qsub.
click here for a hint before getting the answer
launcher_creator.py -h will give you insight to how to use that command.
commands for the commands file if you can't work them out yourself
bowtie2 -t -x bowtie2/NC_012967.1 -1 SRR030257_1.fastq -2 SRR030257_2.fastq -S bowtie2/SRR030257.sam # the -t command is not required for the mapping, but it can be particularly informative when you begin comparing different mappers
Click here for the specific launcher_creator.py commands
launcher_creator.py -n "bowtie2" -t 00:10:00
Your final output file is in SAM format. It's just a text file, so you can peek at it and see what it's like inside. Two warnings though:
- SAM files can be enormously humongous text files (maybe >1 gigabytes). Attempting to open the entire file at once can cause your computer to lock up or your text editor to crash. You are generally safer only looking at a portion at a time using linux commands like
head
orgrep
or using a viewer like IGV, which we will cover later. - SAM files have some rather complicated information encoded as text, like a binary encoded FLAGS field and CIGAR strings. We'll take a look at some of these later, if we have time.
Still, you should recognize some of the information on a line in a SAM file from the input FASTQ, and some of the other information is relatively straightforward to understand, like the position where the read mapped. Give this a try:
head bowtie2/SRR030257.sam
What do you think the 4th and 8th columns mean(click for answer)?
If you thought the answer was the mapping coordinates of the read pairs you were right!
More reading about SAM files
Multithreaded execution
We have actually massively under-utilized Lonestar in this example. We submitted a job that reserved a single node on the cluster, but that node has 12 processors. Bowtie was only using one of those processors (a single "thread")! For programs that support multithreaded execution (and most mappers do because they are obsessed with speed) we could have sped things up by using all 12 processors for the bowtie process.
See if you can figure out how to re-run this using all 12 processors. Click here for a hint
You need to use the -p
, for "processors" option. Since we had 12 processors available to our job.
click here to check your answer
bowtie2 -t -p 12 -x bowtie2/NC_012967.1 -1 SRR030257_1.fastq -2 SRR030257_2.fastq -S bowtie2/SRR030257.sam
Try it out and compare the speed of execution by looking at the log files.
If you want to launch many processes as part of one job, so that they are distributed one per node and use the maximum number of processors available, then you need to learn about the "wayness" of how you request nodes on Lonestar and possibly edit your *.sge script.
One consequence of using multithreading that might be confusing is that the aligned reads might appear in your output SAM file in a different order than they were in the input FASTQ. This happens because small sets of reads get continuously packaged, "sent" to the different processors, and whichever set "returns" fastest is written first. You can force them to appear in the same order (at a slight cost in speed)by adding the--reorder
flag to your command, but is typically only necessary if the reads are already ordered or you intend to do some comparison between the input and output.
Optional Exercises
In thebowtie2example, we mapped in
--local
mode. Try mapping in--end-to-end
mode (aka global mode).- Do theBWA tutorial so you can compare their outputs.
- Didbowtie2orBWAmap more reads?
- In our examples, we mapped in paired-end mode. Try to figure out how to map the reads in single-end mode and create this output.
- Which aligner took less time to run?Are there any options you can change that:
- Lead to a larger percentage of the reads being mapped? (increase sensitivity)
- Speed up performance without causing many fewer reads to be mapped? (increase performance)
Next steps...
The next steps are often to view the output using a specific viewer on your local machine, or to begin identifying variant locations where the reads differ from the reference sequence. These will be the next things we cover in the course. Here is a link to help you return to the GVA 2015 course schedule.