1 / 83

Beyond Naïve Bayes: Some Other Efficient Learning Methods

Beyond Naïve Bayes: Some Other Efficient Learning Methods. William W. Cohen. Review: Large-vocab Naïve Bayes. Create a hashtable C For each example id, y, x 1 ,…., x d in train: C(“ Y =ANY”) ++; C(“ Y=y”) ++ For j in 1..d : C(“ Y=y ^ X= x j ”) ++. Large-vocabulary Naïve Bayes.

tatum
Télécharger la présentation

Beyond Naïve Bayes: Some Other Efficient Learning Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Beyond Naïve Bayes: Some Other Efficient Learning Methods William W. Cohen

  2. Review: Large-vocab Naïve Bayes • Create a hashtable C • For each example id, y, x1,….,xdin train: • C(“Y=ANY”) ++; C(“Y=y”) ++ • For j in 1..d: • C(“Y=y ^ X=xj”) ++

  3. Large-vocabulary Naïve Bayes • Create a hashtable C • For each example id, y, x1,….,xdin train: • C(“Y=ANY”) ++; C(“Y=y”) ++ • Print “Y=ANY += 1” • Print “Y=y += 1” • For j in 1..d: • C(“Y=y ^ X=xj”) ++ • Print “Y=y ^ X=xj += 1” • Sort the event-counter update “messages” • Scan the sorted messages and compute and output the final counter values Think of these as “messages” to another component to increment the counters java MyTrainertrain| sort | java MyCountAdder > model

  4. Large-vocabulary Naïve Bayes Y=business += 1 Y=business += 1 … Y=business ^ X =aaa += 1 … Y=business ^ X=zynga += 1 Y=sports ^ X=hat += 1 Y=sports ^ X=hockey += 1 Y=sports ^ X=hockey += 1 Y=sports ^ X=hockey += 1 … Y=sports ^ X=hoe += 1 … Y=sports += 1 … • Create a hashtable C • For each example id, y, x1,….,xdin train: • C(“Y=ANY”) ++; C(“Y=y”) ++ • Print “Y=ANY += 1” • Print “Y=y += 1” • For j in 1..d: • C(“Y=y ^ X=xj”) ++ • Print “Y=y ^ X=xj += 1” • Sort the event-counter update “messages” • We’re collecting together messages about the same counter • Scan and add the sorted messages and output the final counter values

  5. Large-vocabulary Naïve Bayes Scan-and-add: streaming • previousKey = Null • sumForPreviousKey = 0 • For each (event,delta) in input: • If event==previousKey • sumForPreviousKey += delta • Else • OutputPreviousKey() • previousKey = event • sumForPreviousKey = delta • OutputPreviousKey() • define OutputPreviousKey(): • If PreviousKey!=Null • print PreviousKey,sumForPreviousKey Y=business += 1 Y=business += 1 … Y=business ^ X =aaa += 1 … Y=business ^ X=zynga += 1 Y=sports ^ X=hat += 1 Y=sports ^ X=hockey += 1 Y=sports ^ X=hockey += 1 Y=sports ^ X=hockey += 1 … Y=sports ^ X=hoe += 1 … Y=sports += 1 … Accumulating the event counts requires constant storage … as long as the input is sorted.

  6. Distributed Counting  Stream and Sort Counting Hash table1 • example 1 • example 2 • example 3 • …. Machine 1 Hash table2 Machine 2 “C[x] +=D” Message-routing logic Counting logic . . . Hash table2 Machine K Machine 0

  7. Distributed Counting  Stream and Sort Counting • example 1 • example 2 • example 3 • …. • C[x1] += D1 • C[x1] += D2 • …. “C[x] +=D” Logic to combine counter updates Sort Counting logic BUFFER Machine A Machine C Machine B

  8. Review: Large-vocab Naïve Bayes • Create a hashtable C • For each example id, y, x1,….,xdin train: • C.inc(“Y=ANY”); C.inc(“Y=y”) • For j in 1..d: • C.inc(“Y=y ^ X=xj”) class EventCounter { void inc(String event) { // increment the right hashtable slot if (hashtable.size()>BUFFER_SIZE) { for (e,n) in hashtable.entries : print e + “\t” + n hashtable.clear(); } } }

  9. How much does buffering help? small-events.txt: nb.jar time java -cpnb.jarcom.wcohen.SmallStreamNB< RCV1.small_train.txt \ | sort -k1,1 \ | java -cpnb.jarcom.wcohen.StreamSumReducer> small-events.txt test-small: small-events.txtnb.jar time java -cpnb.jarcom.wcohen.SmallStreamNB\ RCV1.small_test.txt MCAT,CCAT,GCAT,ECAT 2000 < small-events.txt \ | cut -f3 | sort | uniq -c

  10. Using Large-vocabulary Naïve Bayes -1 [For assignment] • For each example id, y, x1,….,xdin test: • Sort the event-counter update “messages” • Scan and add the sorted messages and output the final counter values • Initialize a HashSet NEEDED and a hashtable C • For each example id, y, x1,….,xdin test: • Add x1,….,xd to NEEDED • For each event, C(event) in the summed counters • If event involves a NEEDED term x read it into C • For each example id, y, x1,….,xdin test: • For each y’ indom(Y): • Compute log Pr(y’,x1,….,xd) = …. Model size: O(|V|) Time: O(n2), size of test Memory: same Time: O(n2) Memory: same Time: O(n2) Memory: same

  11. Parallelizing Naïve Bayes - 1 • How to implement Naïve Bayes • Time is linear in size of data (one scan!) • We need to count C( X=word ^ Y=label) • Can you parallelize Naïve Bayes? • Trivial solution 1 • Split the data up into multiple subsets • Count and total each subset independently • Add up the counts • Result should be the same

  12. Stream and Sort Counting  Distributed Counting Standardized message routing logic • example 1 • example 2 • example 3 • …. • C[x1] += D1 • C[x1] += D2 • …. “C[x] +=D” Logic to combine counter updates Counting logic Sort BUFFER Machines A1,… Machines C1,.., Machines B1,…, Easy to parallelize! Trivial to parallelize!

  13. More of my Makefile small-events.txt: nb.jar time java -cpnb.jarcom.wcohen.SmallStreamNB< RCV1.small_train.txt \ | sort -k1,1 \ | java -cpnb.jarcom.wcohen.StreamSumReducer> small-events.txt test-small: small-events.txtnb.jar time java -cpnb.jarcom.wcohen.SmallStreamNB\ RCV1.small_test.txt MCAT,CCAT,GCAT,ECAT 2000 < small-events.txt \ | cut -f3 | sort | uniq -c STREAMJAR=/usr/local/sw/hadoop/…/hadoop-0.20.1-streaming.jar small-events-hs: hadoopfs -rmr rcv1/small/events // clear the output directory time hadoopjar $(STREAMJAR) \ -input rcv1/small/sharded -output rcv1/small/events \ -mapper 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamNB' \ -reducer 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamSumReducer' \ -file nb.jar \ -numReduceTasks 10

  14. Parallelizing Naïve Bayes - 2 $ hadoopfs -ls rcv1/small/sharded Found 10 items -rw-r--r-- 3 … 606405 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00000 -rw-r--r-- 3 … 1347611 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00001 -rw-r--r-- 3 … 939307 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00002 -rw-r--r-- 3 … 1284062 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00003 -rw-r--r-- 3 … 1009890 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00004 -rw-r--r-- 3 … 1206196 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00005 -rw-r--r-- 3 … 1384658 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00006 -rw-r--r-- 3 … 1299698 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00007 -rw-r--r-- 3 … 928752 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00008 -rw-r--r-- 3 … 806030 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00009 $ hadoopfs -tail rcv1/small/sharded/part-00005 weak as the arrival of arbitragedcargoes from the West hasput the local market underpressure… M14,M143,MCAT The Brentcrude market on the Singapore International …

  15. Summing Counts Event Counting on Subsets of Documents

  16. Parallelizing Naïve Bayes - 2 $ hadoopfs -ls rcv1/small/sharded Found 10 items -rw-r--r-- 3 … 606405 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00000 -rw-r--r-- 3 … 1347611 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00001 -rw-r--r-- 3 … 939307 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00002 -rw-r--r-- 3 … 1284062 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00003 -rw-r--r-- 3 … 1009890 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00004 -rw-r--r-- 3 … 1206196 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00005 -rw-r--r-- 3 … 1384658 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00006 -rw-r--r-- 3 … 1299698 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00007 -rw-r--r-- 3 … 928752 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00008 -rw-r--r-- 3 … 806030 2013-01-22 16:28 /user/wcohen/rcv1/small/sharded/part-00009 $ hadoopfs -tail rcv1/small/sharded/part-00005 weak as the arrival of arbitragedcargoes from the West hasput the local market underpressure… M14,M143,MCAT The Brentcrude market on the Singapore International …

  17. Parallelizing Naïve Bayes - 2 STREAMJAR=/usr/local/sw/hadoop/…/hadoop-0.20.1-streaming.jar small-events-hs: hadoopfs -rmr rcv1/small/events // clear the output directory time hadoopjar$(STREAMJAR) \ -input rcv1/small/sharded -output rcv1/small/events \ -mapper 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamNB' \ -reducer 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamSumReducer' \ -file nb.jar\ -numReduceTasks 10

  18. Summing Counts Event Counting on Subsets of Documents Your code

  19. Parallelizing Naïve Bayes - 2 STREAMJAR=/usr/local/sw/hadoop/…/hadoop-0.20.1-streaming.jar small-events-hs: hadoopfs -rmr rcv1/small/events // clear the output directory time hadoopjar$(STREAMJAR) \ -input rcv1/small/sharded -output rcv1/small/events \ -mapper 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamNB' \ -reducer 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamSumReducer' \ -file nb.jar\ -numReduceTasks 10

  20. Parallelizing Naïve Bayes - 2 $ make small-events-hs hadoopfs -rmr rcv1/small/events Moved to trash: hdfs://hdfsname.opencloud/user/wcohen/rcv1/small/events time hadoop jar /usr/local/sw/hadoop/contrib/streaming/hadoop-0.20.1-streaming.jar \ -input rcv1/small/sharded -output rcv1/small/events \ -mapper 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamNB' \ -reducer 'java -Xmx512m -cp ./lib/nb.jarcom.wcohen.StreamSumReducer' \ -file nb.jar -numReduceTasks 10 packageJobJar: [nb.jar, … 13/01/30 11:01:18 INFO mapred.FileInputFormat: Total input paths to process : 10 … 13/01/30 11:01:20 INFO streaming.StreamJob: /usr/local/sw/hadoop/bin/hadoop job -Dmapred.job.tracker=hadoopjt.opencloud:8021 -kill job_201301231150_0776 13/01/30 11:01:20 INFO streaming.StreamJob: Tracking URL: http://hadoopjt.opencloud:40030/jobdetails.jsp?jobid=job_201301231150_0776 13/01/30 11:01:21 INFO streaming.StreamJob: map 0% reduce 0% 13/01/30 11:01:39 INFO streaming.StreamJob: map 20% reduce 0% 13/01/30 11:01:42 INFO streaming.StreamJob: map 50% reduce 0% 13/01/30 11:01:45 INFO streaming.StreamJob: map 100% reduce 0% 13/01/30 11:01:48 INFO streaming.StreamJob: map 100% reduce 5% 13/01/30 11:01:51 INFO streaming.StreamJob: map 100% reduce 23% 13/01/30 11:02:00 INFO streaming.StreamJob: map 100% reduce 100% 13/01/30 11:02:03 INFO streaming.StreamJob: Job complete: job_201301231150_0776 13/01/30 11:02:03 INFO streaming.StreamJob: Output: rcv1/small/events 2.01user 0.36system 0:46.82elapsed 5%CPU (0avgtext+0avgdata 218848maxresident)k 0inputs+640outputs (2major+36300minor)pagefaults 0swaps

  21. Parallelizing Naïve Bayes - 2 …. 13/01/30 11:01:45 INFO streaming.StreamJob: map 100% reduce 0% 13/01/30 11:01:48 INFO streaming.StreamJob: map 100% reduce 5% 13/01/30 11:01:51 INFO streaming.StreamJob: map 100% reduce 23% 13/01/30 11:02:00 INFO streaming.StreamJob: map 100% reduce 100% 13/01/30 11:02:03 INFO streaming.StreamJob: Job complete: job_201301231150_0776 13/01/30 11:02:03 INFO streaming.StreamJob: Output: rcv1/small/events 2.01user 0.36system 0:46.82elapsed 5%CPU (0avgtext+0avgdata 218848maxresident)k 0inputs+640outputs (2major+36300minor)pagefaults0swaps $

  22. Parallelizing Naïve Bayes - 2 $ hadoopfs -ls rcv1/small/events Found 10 items -rw-r--r-- 3 … 359473 2013-01-30 10:58 /user/wcohen/rcv1/small/events/part-00000 -rw-r--r-- 3 … 359544 2013-01-30 10:58 /user/wcohen/rcv1/small/events/part-00001 -rw-r--r-- 3 … 364252 2013-01-30 10:58 /user/wcohen/rcv1/small/events/part-00002 … $ hadoopfs -tail rcv1/small/events/part-00006 equot 1 ^volumes 112 ^vomit 1 ^votequot 1 ^voters 118 ^vouch 139 ^vowed 53 ^vulnerablequot 1 ^w4 1 ^wagerelated 2 …

  23. Parallelizing Naïve Bayes • How to implement Naïve Bayes • Time is linear in size of data (one scan!) • We need to count C( X=word ^ Y=label) • Can you parallelize Naïve Bayes? • Trivial solution 1 • Split the data up into multiple subsets • Count and total each subset independently • Add up the counts • Result should be the same • This is unusual for streaming learning algorithms • Today: another algorithm that is similarly fast • …and some theory about streaming algorithms • …and a streaming algorithm that is not so fast

  24. Rocchio’s algorithm • Relevance Feedback in Information Retrieval, SMART Retrieval System Experiments in Automatic Document Processing, 1971, Prentice Hall Inc.

  25. Rocchio’s algorithm Many variants of these formulae …as long as u(w,d)=0 for words not in d! Store only non-zeros in u(d), so size is O(|d| ) But size of u(y) is O(|nV| )

  26. Rocchio’s algorithm Given a table mapping w to DF(w), we can compute v(d) from the words in d…and the rest of the learning algorithm is just adding…

  27. Rocchio v Bayes Imagine a similar process but for labeled documents… Event counts Train data id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … id5 y5 w5,1 w5,2 …. .. X=w1^Y=sports X=w1^Y=worldNews X=.. X=w2^Y=… X=… … 5245 1054 2120 37 3 … Recall Naïve Bayes test process? id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … C[X=w1,1^Y=sports]=5245, C[X=w1,1^Y=..],C[X=w1,2^…] C[X=w2,1^Y=….]=1054,…, C[X=w2,k2^…] C[X=w3,1^Y=….]=… …

  28. Rocchio…. Rocchio: DF counts Train data id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … id5 y5 w5,1 w5,2 …. .. aardvark agent … 12 1054 2120 37 3 … id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … v(w1,1,id1), v(w1,2,id1)…v(w1,k1,id1) v(w2,1,id2), v(w2,2,id2)… … …

  29. Rocchio…. Rocchio: DF counts Train data id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … id5 y5 w5,1 w5,2 …. .. aardvark agent … 12 1054 2120 37 3 … id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … v(id1 ) v(id2 ) … …

  30. Rocchio…. Rocchio: DF counts Train data id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … id5 y5 w5,1 w5,2 …. .. aardvark agent … 12 1054 2120 37 3 … id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … v(id1 ) v(id2 ) … …

  31. Rocchio…. id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … v(w1,1 w1,2 w1,3 …. w1,k1 ), the document vector for id1 v(w2,1 w2,2 w2,3….)= v(w2,1 ,d), v(w2,2 ,d), … … … For each (y, v), go through the non-zero values in v …one for each win the document d…and increment a counter for that dimension of v(y) Message: incrementv(y1)’s weight for w1,1by αv(w1,1 ,d) /|Cy| Message: incrementv(y1)’s weight for w1,2by αv(w1,2 ,d) /|Cy|

  32. Rocchio at Test Time Rocchio: DF counts Train data id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … id5 y5 w5,1 w5,2 …. .. aardvark agent … v(y1,w)=0.0012 v(y1,w)=0.013, v(y2,w)=… .... … id1 y1 w1,1 w1,2 w1,3 …. w1,k1 id2 y2 w2,1 w2,2 w2,3 …. id3 y3 w3,1 w3,2 …. id4 y4 w4,1 w4,2 … v(id1 ), v(w1,1,y1),v(w1,1,y1),….,v(w1,k1,yk),…,v(w1,k1,yk) v(id2 ), v(w2,1,y1),v(w2,1,y1),…. … …

  33. Rocchio Summary • Compute DF • one scan thru docs • Compute v(idi) for each document • output size O(n) • Add up vectors to get v(y) • Classification ~= disk NB • time: O(n), n=corpus size • like NB event-counts • time: O(n) • one scan, if DF fits in memory • like first part of NB test procedure otherwise • time: O(n) • one scan if v(y)’s fit in memory • like NB training otherwise

  34. Rocchio results… Joacchim’98, “A Probabilistic Analysis of the Rocchio Algorithm…” Rocchio’s method (w/ linear TF) Variant TF and IDF formulas

  35. Rocchio results… Schapire, Singer, Singhal, “Boosting and Rocchio Applied to Text Filtering”, SIGIR 98 Reuters 21578 – all classes (not just the frequent ones)

  36. A hidden agenda • Part of machine learning is good grasp of theory • Part of ML is a good grasp of what hacks tend to work • These are not always the same • Especially in big-data situations • Catalog of useful tricks so far • Brute-force estimation of a joint distribution • Naive Bayes • Stream-and-sort, request-and-answer patterns • BLRT and KL-divergence (and when to use them) • TF-IDF weighting – especially IDF • it’s often useful even when we don’t understand why

  37. One more Rocchio observation Rennieet al, ICML 2003, “Tackling the Poor Assumptions of Naïve Bayes Text Classifiers” NB + cascade of hacks

  38. One more Rocchio observation Rennieet al, ICML 2003, “Tackling the Poor Assumptions of Naïve Bayes Text Classifiers” “In tests, we found the length normalization to be most useful, followed by the log transform…these transforms were also applied to the input of SVM”.

  39. One? more Rocchio observation Documents/labels Split into documents subsets Documents/labels – 1 Documents/labels – 2 Documents/labels – 3 Compute DFs DFs -1 DFs - 2 DFs -3 Sort and add counts DFs

  40. One?? more Rocchio observation Documents/labels DFs Split into documents subsets Documents/labels – 1 Documents/labels – 2 Documents/labels – 3 Compute partial v(y)’s v-1 v-3 v-2 Sort and add vectors v(y)’s

  41. O(1) more Rocchio observation Documents/labels Split into documents subsets Documents/labels – 1 Documents/labels – 2 Documents/labels – 3 Compute partial v(y)’s DFs DFs DFs v-1 v-3 v-2 Sort and add vectors v(y)’s We have shared access to the DFs, but only shared read access – we don’t need to share write access. So we only need to copy the information across the different processes.

  42. Review/outline • How to implement Naïve Bayes • Time is linear in size of data (one scan!) • We need to count C( X=word ^ Y=label) • Can you parallelize Naïve Bayes? • Trivial solution 1 • Split the data up into multiple subsets • Count and total each subset independently • Add up the counts • Result should be the same • This is unusual for streaming learning algorithms • Why?

  43. Two fast algorithms • Naïve Bayes: one pass • Rocchio: two passes • if vocabulary fits in memory • Both method are algorithmically similar • count and combine • Thought experiment: what if we duplicated some features in our dataset many times? • e.g., Repeat all words that start with “t” 10 times.

  44. Two fast algorithms This isn’t silly – often there are features that are “noisy” duplicates, or important phrases of different length • Naïve Bayes: one pass • Rocchio: two passes • if vocabulary fits in memory • Both method are algorithmically similar • count and combine • Thought thought thought thought thought thought thought thought thought thought experiment: what if we duplicated some features in our dataset many times times times times times times times times times times? • e.g., Repeat all words that start with “t” “t” “t” “t” “t” “t” “t” “t” “t” “t” ten ten ten ten ten ten ten ten ten ten timestimestimestimestimestimestimestimestimes times. • Result: some features will be over-weighted in classifier

More Related