MGM Workshop Assembly Tutorial Alicia Clum DOE Joint Genome Institute, Walnut Creek, CA

1. MGM Workshop Assembly Tutorial Alicia Clum DOE Joint Genome Institute, Walnut Creek, CA May 14, 2012

2. Contents • Vocabulary introduction • Introduction to short-read genome sequencing and assembly • Practical experience of short read genome assembly • Improving genome assembly using 3rd generation sequencing

3. Contents Vocabulary introduction Introduction to short-read genome sequencing and assembly Practical experience of short read genome assembly Improving genome assembly using 3rd generation sequencing

4. Vocabulary Fragment library:a short insert (270bp) library with overlapping ends. Aka std library Long insert library: A 4-8kb library where only 100 bp on each end are sequenced. Aka CLIP, mate pair library Contig: A contiguous sequence of DNA Scaffold: One or more contigs linked together by unknown sequence Captured gap: A gap within a scaffold. The order and orientation of the contigs spanning the gap is known D E C B A

5. Contents 2. Introduction to short-read genome sequencing and assembly • Short read sequencing and assembly basics • Short read assembly - De Bruijn graph example • Short read assembly – Scaffolding

6. Why sequence genomes using short reads? 20,000 650bp $400 1,000 450bp 1 $15 $0.1 150bp Traditional genome sequencing technology Short-read $$! - We have to figure out how to sequence microbial genomes using only illumina data

7. Short read genome sequencing Genomic DNA Random fragmentation 4-8 kb fragments 270 bp fragments molecular biology Sequencing (Illumina) Paired-end short insert reads (10’s millions) Paired-end long insert reads (10’s millions) How do we assemble this data back into a genome?

8. Assembly outline Reads Contigs Assembly algorithms e.g. Allpaths, Velvet, Meraculous Scaffolds

9. Assembly outline Reads ‘De Bruijn’ assembly Contigs Scaffolds

10. Contents 2. Introduction to short-read genome sequencing and assembly • Short read sequencing and assembly basics • Short read assembly - De Bruijn graph example • Short read assembly – Scaffolding

11. De Bruijn example “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity,.... “ Dickens, Charles. A Tale of Two Cities. 1859. London: Chapman Hall Example courtesy of J. Leipzig 2010

12. De Bruijn example itwasthebestoftimesitwastheworstoftimesitwastheageofwisdomitwastheageoffoolishness… Generate random ‘reads’ How do we assemble? fincreduli geoffoolis Itwasthebe Itwasthebe geofwisdom itwastheep epochofinc timesitwas stheepocho nessitwast wastheageo theepochof stheepocho hofincredu estoftimes eoffoolish lishnessit hofbeliefi pochofincr itwasthewo twastheage toftimesit domitwasth ochofbelie eepochofbe eepochofbe astheworst chofincred theageofwi iefitwasth ssitwasthe astheepoch efitwasthe wisdomitwa ageoffooli twasthewor ochofbelie sdomitwast sitwasthea eepochofbe ffoolishne eofwisdomi hebestofti stheageoff twastheepo eworstofti stoftimesi theepochof esitwasthe heepochofi theepochof sdomitwast astheworst rstoftimes worstoftim stheepocho geoffoolis ffoolishne timesitwas lishnessit stheageoff eworstofti orstoftime fwisdomitw wastheageo heageofwis incredulit ishnessitw twastheepo wasthewors astheepoch heworstoft ofbeliefit wastheageo heepochofi pochofincr heageofwis stheageofw fincreduli astheageof wisdomitwa wastheageo astheepoch olishnessi astheepoch itwastheep twastheage wisdomitwa fbeliefitw bestoftime epochofbel theepochof sthebestof lishnessit hofbeliefi Itwasthebe ishnessitw sitwasthew ageofwisdo twastheage esitwasthe twastheage shnessitwa fincreduli fbeliefitw theepochof mesitwasth domitwasth ochofbelie heageofwis oftimesitw stheepocho bestoftime twastheage foolishnes ftimesitwa thebestoft itwastheag theepochof itwasthewo ofbeliefit bestoftime mitwasthea imesitwast timesitwas orstoftime estoftimes twasthebes stoftimesi sdomitwast wisdomitwa theworstof astheworst sitwasthew theageoffo eepochofbe …etc. to 10’s of millions of reads Traditional all-vs-all assemblers fail due to immense computational resources (scales with number of reads2) A million (106 ) reads requires a trillion (1012) pairwise alignments De Bruijn solution: Represent the data as a graph (scales with genome size)

13. De Bruijn example Step 1: Convert reads into “Kmers” Kmer: a substring of defined length Reads: theageofwi sthebestof astheageof worstoftim imesitwast the sth ast wor ime Kmers : (k=3) hea the sth ors mes eag heb the rst esi age ebe hea sto sit geo bes eag tof itw eof est age oft twa ofw sto geo fti was fwi tof eof tim ast …..etc for all reads in the dataset

14. De Bruijn example Step 2: Build a De-Bruijn graph from the kmers hea eag age geo eof the ast sth the hea eag age geo eof ofw fwi ast sth the heb ebe bes est sto tof sto tof wor ors rst oft fti tim ime mes was esi twa itw sit …..etc for all ‘kmers’ in the dataset

15. De Bruijn example Step 3: Simplify the graph as much as possible: A De Bruijn Graph De Bruijn assemblies ‘broken’ by repeats longer than kmer “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity,.... “

16. Drawback of De Bruijn approach Step 4: Dump graph into consensus (fasta) No single solution! Break graph to produce final assembly

17. Kmer size is an important parameter in De Bruijn assembly The final assembly (k=3) foolishness itwasthe st wor wisdom times incredulity age epoch be of belief Repeat with a longer “kmer” length A better assembly (k=20) itwasthebestoftimesitwastheworstoftimesitwastheageofwisdomitwastheageoffoolis… Why not always use longest ‘k’ possible? Sequencing errors: Mostly unaffected kmers the ent sth ebe tof k=3 heb nto ben sthebentof k=10 sthebentof 100% wrong kmer

18. Contents 2. Introduction to short-read genome sequencing and assembly • Short read sequencing and assembly basics • Short read assembly - De Bruijn graph example • Short read assembly – Scaffolding

19. Scaffolding Reads ‘De Bruijn’ assembly Contigs Align reads to DeBruijn contigs Join contigs using evidence from paired end data Scaffolds (An assembly) “Captured” gaps caused by repeats. Represented by “NNN” in assembly

20. Contents Vocabulary introduction Introduction to short-read genome sequencing and assembly Practical experience of short read genome assembly Improving genome assembly using 3rd generation sequencing

21. Real life assembly is messy! Assembly in theory Uniform coverage, no errors, no contamination Assembly in reality Contaminant reads (-> incorrect + inflated assembly) Chimeric reads (->mis-joins) Sequencing errors (-> fragmented assembly) * * * * * * * Biased coverage (->gaps) * Worse than predicted assemblies!

22. Real life assembly is messy! Coverage (x) Theoretical Fraction of normalized coverage GC% of 100 base windows Reference position (bp)

23. Genome properties can also make assembly difficult High repeat content RESULT: misassemblies / collapsed assemblies Polyploidy RESULT: fragmented assembly r r a’ a r r r Biased sequence abundance Biased sequence composition RESULT: incomplete / fragmented assembly ACTGTCTAGTCAGCGCGCGCGCGCGCGCCCGCGCGCGCGGGCGGCGGCGCGGGCGGGCGCATGTAGTGATC RESULT: Incomplete / fragmented assembly

24. How do we get a good assembly?

25. Key features of the JGI microbe sequencing and assembly pipeline Genomic DNA Fragment Longer inserts Improved scaffolding 270bp 4-8 kb Overlapping paired-end reads for short insert library Allows for long ‘kmer’ Illumina V3 sequencing chemistry Reduced GC bias Sequence AllPaths LG assembler Most complete + most accurate assembler in our hands Internal error correction 2 x 100 bp 2 x 150bp Extensive data QC: -Remove artifacts -Remove contaminants -Reorder libraries QC Assembler Best Assemblies

26. Benefits of ALLPATHS Internal error correction of all data types Simplifies the graph by removing kmers at low coverage caused by errors Given the correct input data, a good assembly can be produced with default options Overlaps the fragment data to use a longer kmer size Produces accurate and highly contiguous assemblies

27. Simulated De Bruijn assembly for six ‘known’microbial genomes What fraction of a genome should we be able to assemble, that is, can be represented in unique kmers? 97% Kmer = 150 A small fraction of the genome remains ‘unassemblable’ Kmer =30, most of the genome CAN be assembled Matt Blow

28. How fragmented are short read assemblies? Goal: genome in 1 fragment per replicon Assemblies from simulated data (7 replicons) Short insert library only Assemblies using only short insert sequencing libraries are acceptable (<100 scaffolds)

29. How good are short read assemblies? Goal: genome in 1 fragment per replicon Assemblies from real data Assemblies from simulated data (7 replicons) Short insert library only Number of scaffolds Assemblies using only short insert sequencing libraries are acceptable (<100 scaffolds)

30. How good are short read assemblies? Goal: genome in 1 fragment per replicon Assemblies from real data Assemblies from simulated data (7 replicons) Short + long insert libraries Number of scaffolds Assemblies using short and long insert sequencing libraries are very good (often a single fragment!)

31. Comparison of assembly results to reference genome annotation Average % known genes identified Average number of fragments 85 97.3% Short insert library only 3 97.4% Short + long insert libraries Near complete and accurate assemblies are now possible using only short read data.

32. But problems remain: Reads ‘De Bruijn’ assembly Contigs Align reads to DeBruijncontigs Join contigs using evidence from paired end data An assembly “Captured” gaps caused by repeats. Represented by “NNN” in assembly

33. Contents Vocabulary introduction Introduction to short-read genome sequencing and assembly Practical experience of short read genome assembly Improving genome assembly using 3rd generation sequencing

34. Pacific Biosciences Sequencer 14,000 Mean read length = 1080bp 12,000 10,000 8,000 6,000 Maximum read length >4kb 4,000 2,000 1 2 3 4 5 6 7 Read length (Kb) Long reads from “3rd generation” Pacific Biosciences sequencer hold promise for improving short-read based assemblies Late April 2012 “Early” data Mean read length = 2743 bp Number of reads High error rate Up to 15% error rate In-del errors Low coverage bias Reduced sensitivity to G+C rich regions compared to illumina chemistry Fraction of normalized coverage GC% of 100 base windows

35. PacBio pilot study For each of 20 Microbes: Genomic DNA + + 2 x 150bp 270bp insert “1000”bp 2 x 100bp 8kb insert (Illumina HiSeq, V3 chemistry, 2x150b) (PacBio) Assembly 1 Assembly 2 Assembler: AllPaths V39750 (PacBio enabled)

36. 3rd Generation sequence data improves genome assembly Assembled using: Illumina only Most improved genome: 53 / 71 (75%) gaps closed Illumina + PacBio Least improved genomes (.. but started out in good shape)

37. PacBio assembly improvement is greatest for genomes with high GC content MOST improved genomes average 66% G+C LEAST improved genomes average 56% G+C Conclusion: 3rd generation sequencing is a promising tool for better assemblies, especially for G+C rich genomes.

38. Summary • High quality genome sequencing using only short-reads is within reach • Short-read microbial genomes assemblies are minimally fragmented and contain the vast majority known genes • Third-generation sequencing may provide an inexpensive path to finished genomes

39. END

40. Assembly Q+A Alex Copeland (Assembly group lead) Alicia Clum (Analyst, Genome assembly) James Han (QC group lead)

41. Metagenome Assembly Soil Acid mine Cow rumen 1 10 100 1000 10000 Species per metagenome -All challenges of isolate genome assembly remain -Extra challenges from diversity and different abundance of constituent genomes - The same strategies as isolate assembly can be used, but many heuristics fail for metagenomes

42. Road map for Long Mate Pair (LMP) library Improvement & Development • Fragmentation • column • Hydroshear • Covaris • Transposon • Circularization • Cre-Lox • Hybridization • Ligation • Purification • Clean up • efficiency • Electro-elution • 2-D Electrophoresis • Paired tag generation • RE enzyme • Nick translation • shearing • Amplification • PCR cycle #, specificity, choice of enzyme --- • Illumina • LFPE • CLIP

43. De Bruijn example The final assembly (k=3) foolishness itwasthe st wor wisdom times incredulity age epoch be of belief Repeat with a longer “kmer” length A better assembly (k=20) itwasthebestoftimesitwastheworstoftimesitwastheageofwisdomitwastheageoffoolis… Why not always use longest ‘k’ possible? Sequencing errors: Mostly unaffected kmers the ent sth ebe tof k=3 heb nto ben sthebentof k=10 sthebentof 100% wrong kmer

44. Short read assemblies are improving over time Sequenced genome properties remain constant But illumina sequence quantity and quality is increasing… …resulting in better microbial genome assemblies Average results from sub-optimal “QC” assemblies

45. Q1. What fraction of kmers are unique in a typical microbial genome? (6 known microbe genomes, 15 kmer lengths) 97% Kmer = 150 A small fraction of the genome remains non-unique Majority of genome contained in unique fragments Kmer = 30, most of the genome is in unique kmers

46. Q2. How do non-unique kmers fragment the assembly? Microbe Assemblies will be fragmented! How do we improve this?

47. Metagenome assembly is an ongoing challenge -All challenges of isolate genome assembly remain -Extra challenges from diversity and different abundance of constituent genomes - The same strategies as isolate assembly can be used, but many heuristics fail for metagenomes

48. Useful Reviews • Miller JR, Koren S, Sutton G. , Assembly algorithms for next-generation sequencing data. Genomics. 2010 Jun;95(6):315-27. • Mihai Pop, Genome assembly reborn: recent computational challenges. Brief Bioinform (2009) 10 (4):354-366.

49. Illumina data qualitySyntrophorhabdus aromaticivorans Genome Properties PASS Read Quality Library Quality Run Quality

50. Illumina data quality Opitutaceae bacterium TAV2 Genome Properties FAIL Read Quality Library Quality Run Quality