1 / 35

Foundations of Machine Learning: Formulating Learning Theories and Inductive Bias

In this lecture, Pushpak Bhattacharyya presents a deep dive into the foundational concepts of machine learning, based on the landmark paper by L.G. Valiant from 1984, which formalizes learnability through the Probably Approximately Correct (PAC) learning model. Key insights touch upon the significance of inductive bias, the conditions under which hypotheses are produced, and challenges related to various probability distributions. The discussion also includes the concept of VC-dimension as a measure of learnability. This critical understanding is essential for effective application and research in artificial intelligence.

hei
Télécharger la présentation

Foundations of Machine Learning: Formulating Learning Theories and Inductive Bias

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS344: Introduction to Artificial Intelligence Pushpak BhattacharyyaCSE Dept., IIT Bombay Lecture 36-37: Foundation of Machine Learning

  2. Attempt at formalizing Machine Learning (Landmark paper by L.G.Valiant, 1984, A Theory of Learnable, CACM Journal) IIT Bombay

  3. Learning Training (Loading) Testing (Generalization) IIT Bombay

  4. Training Internalization Hypothesis Production IIT Bombay

  5. Hypothesis Production Inductive Bias In what form is the hypothesis produced? IIT Bombay

  6. U Universe C h C h = Error region + P(C h ) <= Є + accuracy parameter Prob. distribution

  7. P(X) = Prob that x is generated by the teacher – the “oracle” and is labeled <x, +> : Positive example. <x, -> : Negative example.

  8. + Learning Means the following Should happen: Pr(P(c h) <= Є) >= 1- δ PAC model of learning correct. Probably Approximately Correct

  9. An Example Universe: 2- Dimensional Plane - A B - - - - + - + - + - C - D IIT Bombay

  10. Key insights from 40 years of machine Learning Research: 1) What is it that is being learnt , and how the hypothesis should be produced ? This is a “MUST”. This is called Inductive Bias . • “Learning in the Vacuum” is not possible. A learner already has crucial given pieces of knowledge at its disposal.

  11. y A B - + - - - - - + - - + - - - C - D x IIT Bombay

  12. Algo: 1. Ignore –ve example. 2. Find the closest fitting axis parallel rectangle for the data.

  13. Pr(P(c h) <= Є ) >= 1- δ y + c C h + A B - - - - - - + + - - + h - - - C - D Case 1: If P([]ABCD) < Є than the Algo is PAC. x

  14. Case 2 p([]ABCD) > Є y A B Top - - - - - - - - Right Left - - - C - D Case 2: x Bottom P(Top) = P(Bottom) = P(Right) = P(Left) = Є/4

  15. Let # of examples = m. • Probability that a point comes from top = Є/4 • Probability that none of the m example come from top = (1- Є/4)m IIT Bombay

  16. Probability that none of m examples come from one of top/bottom/left/right = 4(1 - Є/4)m Probability that at least one example will come from the 4 regions = 1- 4(1 - Є/4)m

  17. This fact must have probability greater than or equal to 1- δ 1-4 (1 - Є/4 )m >1- δ or 4(1 - Є/4 )m < δ

  18. y A B + + + + C D x

  19. (1 - Є/4)m < e(-Єm/4) We must have 4 e(-Єm/4) < δ Or m > (4/Є) ln(4/δ)

  20. Lets say we want 10% error with 90% confidence M > ((4/0.1) ln (4/0.1)) Which is nearly equal to 200

  21. Criticism against PAC learning The model produces too many –ve results. The Constrain of arbitrary probability distribution is too restrictive.

  22. In spite of –ve results, so much learning takes place around us.

  23. VC-dimension Gives a necessary and sufficient condition for PAC learnability.

  24. C C1 C3 Def:- Let C be a concept class, i.e., it has members c1,c2,c3,…… as concepts in it. C2

  25. Let S be a subset of U (universe). Now if all the subsets of S can be produced by intersecting with Cis, then we say C shatters S.

  26. The highest cardinality set S that can be shattered gives the VC-dimension of C. VC-dim(C)= |S| VC-dim: Vapnik-Cherronenkis dimension.

  27. y 2 – Dim surface C = { half planes} x IIT Bombay

  28. y S1= { a } {a}, Ø a x |s| = 1 can be shattered IIT Bombay

  29. y S2= { a,b } {a,b}, {a}, {b}, Ø b a x |s| = 2 can be shattered IIT Bombay

  30. y S3= { a,b,c } b a c x |s| = 3 can be shattered IIT Bombay

  31. IIT Bombay

  32. y S4= { a,b,c,d } A B C D x |s| = 4 cannot be shattered IIT Bombay

  33. Fundamental Theorem of PAC learning (Ehrenfeuct et. al, 1989) • A Concept Class C is learnable for all probability distributions and all concepts in C if and only if the VC dimension of C is finite • If the VC dimension of C is d, then…(next page) IIT Bombay

  34. Fundamental theorem (contd) (a) for 0<ε<1 and the sample size at least max[(4/ε)log(2/δ), (8d/ε)log(13/ε)] any consistent function A:ScC is a learning function for C (b) for 0<ε<1/2 and sample size less than max[((1-ε)/ ε)ln(1/ δ), d(1-2(ε(1- δ)+ δ))] No function A:ScH, for any hypothesis space is a learning function for C. IIT Bombay

  35. Paper’s • 1. A theory of the learnable, Valiant, LG (1984), Communications of the ACM 27(11):1134 -1142. • 2. Learnability and the VC-dimension, A Blumer, A Ehrenfeucht, D Haussler, M Warmuth - Journal of the ACM, 1989. Book Computational Learning Theory, M. H. G. Anthony, N. Biggs, Cambridge Tracts in Theoretical Computer Science, 1997.

More Related