1 / 41

kNN & Naïve Bayes

kNN & Naïve Bayes. Hongning Wang CS@UVa. Today’s lecture. Instance-based classifiers k nearest neighbors Non-parametric learning algorithm Model-based classifiers Naïve Bayes classifier A generative model Parametric learning algorithm. How to classify this document?. Sports.

lcruz
Télécharger la présentation

kNN & Naïve Bayes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. kNN & Naïve Bayes Hongning Wang CS@UVa

  2. Today’s lecture • Instance-based classifiers • k nearest neighbors • Non-parametric learning algorithm • Model-based classifiers • Naïve Bayes classifier • A generative model • Parametric learning algorithm CS 6501: Text Mining

  3. How to classify this document? Sports Documents by vector space representation Politics ? Finance CS 6501: Text Mining

  4. Let’s check the nearest neighbor Sports Are you confident about this? Politics ? Finance CS 6501: Text Mining

  5. Let’s check more nearest neighbors • Ask k nearest neighbors • Let them vote Sports Politics ? Finance CS 6501: Text Mining

  6. Probabilistic interpretation of kNN • Approximate Bayes decision rule in a subset of data around the testing point • Let be the volume of the dimensional ball around containing the nearest neighbors for , we have Nearest neighbors from class 1 => Total number of instances With Bayes rule: Total number of instances in class 1 Counting the nearest neighbors from class 1 CS 6501: Text Mining

  7. kNN is close to optimal • Asymptotically, the error rate of 1-nearest-neighbor classification is less than twice of the Bayes error rate • Decision boundary • 1NN - Voronoi tessellation A non-parametric estimation of posterior distribution CS 6501: Text Mining

  8. Components in kNN • A distance metric • Euclidean distance/cosine similarity • How many nearby neighbors to look at • k • Instance look up • Efficiently search nearby points CS 6501: Text Mining

  9. Effect of k • Choice of k influences the “smoothness” of the resulting classifier CS 6501: Text Mining

  10. Effect of k • Choice of k influences the “smoothness” of the resulting classifier k=1 CS 6501: Text Mining

  11. Effect of k • Choice of k influences the “smoothness” of the resulting classifier k=5 CS 6501: Text Mining

  12. Effect of k • Large k -> smooth shape for decision boundary • Small k -> complicated decision boundary Error Error on testing set Error on training set Larger k Smaller k CS 6501: Text Mining Model complexity

  13. Efficient instance look-up • Recall MP1 • In Yelp_small data set, there are 629K reviews for training and 174K reviews for testing • Assume we have a vocabulary of 15K • Complexity of kNN Feature size Training corpus size Testing corpus size CS 6501: Text Mining

  14. Efficient instance look-up • Exact solutions • Build inverted index for text documents • Special mapping: word -> document list • Speed-up is limited when average document length is large Postings Dictionary CS 6501: Text Mining

  15. Efficient instance look-up • Exact solutions • Build inverted index for text documents • Special mapping: word -> document list • Speed-up is limited when average document length is large • Parallelize the computation • Map-Reduce • Map training/testing data onto different reducers • Merge the nearest k neighbors from the reducers CS 6501: Text Mining

  16. Efficient instance look-up • Approximate solution • Locality sensitive hashing • Similar documents -> (likely) same hash values h(x) CS 6501: Text Mining

  17. Efficient instance look-up • Approximate solution • Locality sensitive hashing • Similar documents -> (likely) same hash values • Construct the hash function such that similar items map to the same “buckets” with a high probability • Learning-based: learn the hash function with annotated examples, e.g., must-link, cannot-link • Random projection CS 6501: Text Mining

  18. Recap: probabilistic interpretation of kNN • Approximate Bayes decision rule in a subset of data around the testing point • Let be the volume of the dimensional ball around containing the nearest neighbors for , we have Nearest neighbors from class 1 => Total number of instances With Bayes rule: Total number of instances in class 1 Counting the nearest neighbors from class 1 CS 6501: Text Mining

  19. Recap: effect of k • Large k -> smooth shape for decision boundary • Small k -> complicated decision boundary Error Error on testing set Error on training set Larger k Smaller k CS 6501: Text Mining Model complexity

  20. Recap: efficient instance look-up • Approximate solution • Locality sensitive hashing • Similar documents -> (likely) same hash values h(x) CS 6501: Text Mining

  21. Random projection • Approximate the cosine similarity between vectors • , is a random unit vector • Each defines one hash function, i.e., one bit in the hash value CS 6501: Text Mining

  22. Random projection • Approximate the cosine similarity between vectors • , is a random unit vector • Each defines one hash function, i.e., one bit in the hash value CS 6501: Text Mining

  23. Random projection • Approximate the cosine similarity between vectors • , is a random unit vector • Each defines one hash function, i.e., one bit in the hash value • Provable approximation error CS 6501: Text Mining

  24. Efficient instance look-up • Effectiveness of random projection • 1.2M images + 1000 dimensions 1000x speed-up CS 6501: Text Mining

  25. Weight the nearby instances • When the data distribution is highly skewed, frequent classes might dominate majority vote • They occur more often in the k nearest neighbors just because they have large volume Sports Politics Finance ? CS 6501: Text Mining

  26. Weight the nearby instances • When the data distribution is highly skewed, frequent classes might dominate majority vote • They occur more often in the k nearest neighbors just because they have large volume • Solution • Weight the neighbors in voting • or CS 6501: Text Mining

  27. Summary of kNN • Instance-based learning • No training phase • Assign label to a testing case by its nearest neighbors • Non-parametric • Approximate Bayes decision boundary in a local region • Efficient computation • Locality sensitive hashing • Random projection CS 6501: Text Mining

  28. Recall optimal Bayes decision boundary *Optimal Bayes decision boundary False positive False negative CS 6501: Text Mining

  29. Estimating the optimal classifier Requirement: Class conditional density Class prior #parameters: V binary features CS 6501: Text Mining

  30. We need to simplify this • Features are conditionally independent given class labels • E.g., This does not mean ‘white house’ is independent of ‘obama’! CS 6501: Text Mining

  31. Conditional v.s. marginal independence • Features are not necessarily marginally independent from each other • However, once we know the class label, features become independent from each other • Knowing it is already political news, observing ‘obama’ contributes little about occurrence of ‘while house’ CS 6501: Text Mining

  32. Naïve Bayes classifier Class conditional density Class prior #parameters: v.s. Computationally feasible CS 6501: Text Mining

  33. Naïve Bayes classifier By Bayes rule By conditional independence assumption y x3 xv x1 x2 … CS 6501: Text Mining

  34. Estimating parameters • Maximial likelihood estimator CS 6501: Text Mining

  35. Enhancing Naïve Bayes for text classification I • The frequency of words in a document matters • In log space Essentially, estimating different language models! Model parameter Feature vector Class bias CS 6501: Text Mining

  36. Enhancing Naïve Bayes for text classification I • For binary case where a linear model with vector space representation? We will come back to this topic later. CS 6501: Text Mining

  37. Enhancing Naïve Bayes for text classification II • Usually, features are not conditionally independent • Enhance the conditional independence assumptions by N-gram language models CS 6501: Text Mining

  38. Enhancing Naïve Bayes for text classification III • Sparse observation • Then, no matter what values the other features take, • Smoothing class conditional density • All smoothing techniques we have discussed in language models are applicable here CS 6501: Text Mining

  39. Maximum a Posterior estimator • Adding pseudo instances • Priors: and • MAP estimator for Naïve Bayes Can be estimated from a related corpus or manually tuned #pseudo instances CS 6501: Text Mining

  40. Summary of Naïve Bayes • Optimal Bayes classifier • Naïve Bayes with independence assumptions • Parameter estimation in Naïve Bayes • Maximum likelihood estimator • Smoothing is necessary CS 6501: Text Mining

  41. Today’s reading • Introduction to Information Retrieval • Chapter 13: Text classification and Naive Bayes • 13.2 – Naive Bayes text classification • 13.4 – Properties of Naive Bayes • Chapter 14: Vector space classification • 14.3 k nearest neighbor • 14.4 Linear versus nonlinear classifiers CS 6501: Text Mining

More Related