SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

5. Bayesian Learning

keon avatar
keon
31 diapositives · 558 Views · Oct 11, 2014

5. Bayesian Learning. 5.1 Introduction Bayesian learning algorithms calculate explicit probabilities for hypotheses Practical approach to certain learning problems Provide useful perspective for understanding learning algorithms. 5. Bayesian Learning. Drawbacks:

5. Bayesian Learning - slide 1
01 / 315. Bayesian Learning
1 / 17

Télécharger la présentation

5. Bayesian Learning
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. 5. Bayesian Learning 5.1 Introduction • Bayesian learning algorithms calculate explicit probabilities for hypotheses • Practical approach to certain learning problems • Provide useful perspective for understanding learning algorithms 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  2. 5. Bayesian Learning Drawbacks: • Typically requires initial knowledge of many probabilities • In some cases, significant computational cost required to determine the Bayes optimal hypothesis (linear in the number of candidate hypotheses) 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  3. 5. Bayesian Learning 5.2 Bayes Theorem Best hypothesis  most probable hypothesis Notation P(h): prior probability of hypothesis h P(D): prior probability that dataset D be observed P(D|h): prior probability of D given h P(h|D): posterior probability of h 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  4. 5. Bayesian Learning • Bayes Theorem P(h|D) = P(D|h) P(h) / P(D) • Maximum a posteriori hypothesis hMAP  argmaxhHP(h|D) = argmaxhHP(D|h) P(h) • Maximum likelihood hypothesis hML = argmaxhHP(D|h) = hMAP if we assume P(h)=constant 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  5. 5. Bayesian Learning • Example P(cancer) = 0.008 P(cancer) = 0.992 P(+|cancer) = 0.98 P(- |cancer) = 0.02 P(+|cancer) = 0.03 P(- |cancer) = 0.97 For a new patient the lab test returns a positive result. Should be diagnose cancer or not? P(+|cancer)P(cancer)=0.0078 P(-|cancer)P(cancer)=0.0298  hMAP = cancer 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  6. 5. Bayesian Learning 5.3 Bayes Theorem and Concept Learning What is the relationship between Bayes theorem and concept learning? • Brute Force Bayes Concept Learning 1. For each hypothesis hH calculate P(h|D) 2. Output hMAP  argmaxhHP(h|D) 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  7. 5. Bayesian Learning • We must choose P(h) and P(D|h) from prior knowledge Let’s assume: 1. The training data D is noise free 2. The target concept c is contained in H 3. We consider a priori all the hypotheses equally probable  P(h) = 1/|H|  hH 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  8. 5. Bayesian Learning Since the data is assumed noise free: P(D|h)=1 if di=h(xi) di  D P(D|h)=0 otherwise Brute-force MAP learning • If h is inconsistent with D: P(h|D) = P(D|h).P(h)/P(D) = 0.P(h)/P(D) = 0 • If h is consistent with D: P(h|D) = 1. (1/|H|) /(|VSH,D| /|H|) = 1/|VSH,D| 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  9. 5. Bayesian Learning  P(D|h)=1/|VSH,D| if h is consistent with D P(D|h)=0 otherwise Every consistent hypothesis is a MAP hypothesis Consistent Learners • Learning algorithms whose outputs are hypotheses that commit zero errors over the training examples (consistent hypotheses) 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  10. 5. Bayesian Learning Under the assumed conditions, Find-S is a consistent learner The Bayesian framework allows to characterize the behavior of learning algorithms, identifying P(h) and P(D|h) under which they output optimal (MAP) hypotheses 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  11. 5. Bayesian Learning 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  12. 5. Bayesian Learning 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  13. 5. Bayesian Learning 6.4Maximum Likelihood and LSE Hypotheses Learning a continuous-valued target function (regression or curve fitting) H = Class of real-valued functions defined over X h : X   L learns f : X   (xi,di)  D di = f(xi) + i i=1,m f : noise-free target function : white noise N(0,) 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  14. 5. Bayesian Learning 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  15. 5. Bayesian Learning Under these assumptions, any learning algorithm that minimizes the squared error between the output hypothesis predictions and the training data will output a ML hypothesis: hML = argmaxhHp(D|h) = argmaxhH i=1,mp(di|h) = argmaxhH i=1,m exp{-[di-h(xi)]2/22} = argminhH i=1,m [di-h(xi)]2 = hLSE 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  16. 5. Bayesian Learning 5.5 ML Hypotheses for Predicting Probabilities • We wish to learn a nondetermnistic function f : X  {0,1} that is, the probabilities that f(x)=0 and f(x)=1 • Training data D = (xi,di) • We assume that any particular instance xi is independent of hypothesis h 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  17. 5. Bayesian Learning Then P(D|h) = i=1,mP(xi,di|h) = i=1,mP(di|h, xi) P(xi) P(di|h,xi) = h(xi) if di=1 P(di|h,xi) =1-h(xi) if di=0  P(di|h,xi) = h(xi)di [1-h(xi)]1-di 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  18. 5. Bayesian Learning hML = argmaxhH i=1,mh(xi)di [1-h(xi)]1-di = argmaxhH i=1,mdi log[h(xi)] + [1-di] log[1-h(xi)] = argminhH [Cross Entropy] Cross Entropy  - i=1,mdi log[h(xi)] + [1-di] log[1-h(xi)] 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  19. 5. Bayesian Learning 5.6 Minimum Description Length Principle hMAP = argmaxhHP(D|h) P(h) = argminhH {-log2P(D|h)-log2P(h)}  short hypotheses are preferred Description Length LC(h): Number of bits required to encode message h using code C 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  20. 5. Bayesian Learning • - log2P(h)  LCH(h): Description length of h under the optimal (most compact) encoding of H • - log2P(D|h)  LCD |h(D|h): Description length of training data D given hypothesis h  hMAP = argminhH {LCH(h) + LCD |h(D|h)} MDL Principle: Choose hMDL = argminhH {LC1(h) + LC2(D|h)} 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  21. 5. Bayesian Learning 5.7 Bayes Optimal Classifier What is the most probable classification of a new instance given the training data? Answer: argmaxvjV hHP(vj|h) P(h|D) where vj V are the possible classes  Bayes Optimal Classifier 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  22. 5. Bayesian Learning 5.9 Naïve Bayes Classifier Given the instance x=(a1,a2,...,an) vMAP = argmaxvjVP(x|vj) P(vj) The Naïve Bayes Classifier assumes conditional independence of attribute values : vNB = argmaxvjVP(vj) i=1,nP(ai|vj) 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  23. 5. Bayesian Learning 5.10 An Example: Learning to Classify Text Task: “Filter WWW pages that discuss ML topics” • Instance space X contains all possible text documents • Training examples are classified as “like” or “dislike” How to represent an arbitrary document? • Define an attribute for each word position • Define the value of the attribute to be the English word found in that position 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  24. 5. Bayesian Learning vNB = argmaxvjVP(vj) i=1,NwordsP(ai|vj) V {like,dislike} ai 50.000 distinct words in English  We must estimate ~ 2 x 50.000 x Nwords conditional probabilities P(ai|vj) This can be reduced to 2 x 50.000terms by considering P(ai=wk|vj) = P(am=wk|vj)  i,j,k,m 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  25. 5. Bayesian Learning • How to choose the conditional probabilities? m-estimate: P(wk|vj) = (nk + 1) / (Nwords+ |Vocabulary|) nk: number of times word wk is found |Vocabulary| : total number of distinct words Concrete example: Assigning articles to 20 usenet newsgroups  Accuracy: 89% 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  26. 5. Bayesian Learning 5.11 Bayesian Belief Networks Bayesian belief networks assume conditional independence only between subsets of the attributes • Conditional independence • Discrete-valued random variables X,Y,Z • X is conditionally independent of Y given Z if P(X |Y,Z)= P(X |Z) 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  27. 5. Bayesian Learning 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  28. 5. Bayesian Learning Representation • A Bayesian network represents the joint probability distribution of a set of variables • Each variable is represented by a node • Conditional independence assumptions are indicated by a directed acyclic graph • Variables are conditionally independent of its nondescendents in the network given its inmediate predecessors 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  29. 5. Bayesian Learning The joint probabilities are calculated as P(Y1,Y2,...,Yn) = i=1,nP [Yi|Parents(Yi)] The values P [Yi|Parents(Yi)] are stored in tables associated to nodes Yi Example: P(Campfire=True|Storm=True,BusTourGroup=True)=0.4 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  30. 5. Bayesian Learning Inference • We wish to infer the probability distribution for some variable given observed values for (a subset of) the other variables • Exact (and sometimes approximate) inference of probabilities for an arbitrary BN is NP-hard • There are numerous methods for probabilistic inference in BN (for instance, Monte Carlo), which have been shown to be useful in many cases 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006

  31. 5. Bayesian Learning Learning Bayesian Belief Networks Task: Devising effective algorithms for learning BBN from training data • Focus of much current research interest • For given network structure, gradient ascent can be used to learn the entries of conditional probability tables • Learning the structure of BBN is much more difficult, although there are successful approaches for some particular problems 1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006