10-701/15-781 Machine Learning (Recitation 1)

1. A significant portion of the slides are copies from Prof. Andrew Moore’s tutorials on Statistical Data Mining: http://www.autonlab.org/tutorials/ 10-701/15-781 Machine Learning(Recitation 1) By Fan Guo 9/14/06

2. We start with… • Probability —— Degree of Uncertainty • Random Variable • Probability Distribution • Discrete (probability mass function) • Continuous (cdf, pdf) • Remember, they should be normalized!

3. We are going to cover… • Compress the information from the pdf • Maximum Likelihood (ML) • Bayesian inference • Gaussian and Gaussian magic

4. Characterizing a distribution • Mean (or Expectation) • Variance • Standard Deviation • Mode • Min, Max • Entropy • the Plot!

5. Discussion • Prove that E[X] is the value u that minimize E[(X-u)2] • Answer: write E[(X-u)2] explicitly as a function of u, take derivatives with respect to u and set it to zero.

6. Discussion • What is the value u that minimizes E[|X-u|]? • Answer: The median. For continuous distribution, Let f be the cdf, it is f-1(0.5).

7. In 2 dimensions X – Mile per Gallon Y – Car weight

11. In 2 dimensions X – Mile per Gallon Y – Car weight • Expectation (Centroid) • Entropy • Marginal Dists. • Conditional Dists. • Covariance • Independence

12. Test your understanding When, if ever, Var[X+Y] = Var[X] + Var[Y]? • All the time?

13. Test your understanding When, if ever, Var[X+Y] = Var[X] + Var[Y]? • All the time? • Only when X and Y are independent?

14. Test your understanding When, if ever, Var[X+Y] = Var[X] + Var[Y]? • All the time? • Only when X and Y are independent? • It can fail even if X and Y are independent?

15. This slide is copied from Jimeng Sun’s recitation slides for the fall 2005 class.

16. What if? • X = {T, T, T, T, T} • L(p) = ? • pML = ?

17. Being Bayesian • We are uncertain about p… • We treat p as a random variable • We have a prior belief: p ~ Uniform(0,1)

18. Computing the Posterior

19. Comments on Bayesian Inference • We are uncertain about p • It is represented by a prior belief on p • We observe a data set X • We try to update our belief using the Bayes rule • The posterior distribution may be useful for future experiments/inference • Sometimes it is not easy to compute the posterior distribution because we have to take the integration to compute p(X). • If we use conjugate prior, the problem becomes easy. • The choice of the prior depends on the background knowledge, the model and the computational cost desired • Now let’s see how to estimate p after we compute the posterior distribution

20. The Posterior Belief • MAP • easier to compute • PosteriorMean • MAP may notbe desired for a skewed distribution

21. What we covered… • Collapse the pdf • Joint distribution could tell everything… • Likelihood, log-likelihood • ML estimation vs Bayesian inference • MAP, Posterior Mean • un-informative prior, conjugate prior

22. What we didn’t cover… • Many interesting and useful pdf • Conditional independence • Gaussian • http://www.autonlab.org/tutorials/ • MLE and Bayesian Inference for continuous distribution

23. Thank you!