Distributions and Concepts in Probability Theory

1. Distributions and Concepts in Probability Theory 10701 Recitation PengtaoXie

2. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

3. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

4. Distributions • Usage in machine learning • Expectation and variance • Bernoulli, Beta, multinomial, Dirichlet, Gaussian

5. Usage of distributions in ML • Gaussian: Least Square Regression, Mixture of Gaussians, Kalman Filtering, Gaussian Markov Random Field, Gaussian Process • Multinomial: Hidden Markov Model, Mixture of Gaussians, Latent Dirichlet Allocation, Naive Bayes classifier • Dirichlet: Latent Dirichlet Allocation, Dirichlet Process • Bernoulli: Logistic Regression, switching variables in Graphical Models • Beta: Beta Process

6. Expectation and variance • Expectation: the average value of a random variable under its probability distribution • Variance: a measure of how much variability there is in x around its mean value

7. Distributions Random Variable Discrete Random Variable Continuous Random Variable Two Outcomes Multiple Outcomes Bernoulli Distribution Multinomial Distribution Gaussian Distribution Conjugate Conjugate Beta Distribution Dirichlet Distribution Conjugate

8. Bernoulli • Model binary variable {0,1} • Probability mass function • Expectation

9. Multinomial distribution • Model variables taking K possible states • 1-of-K coding • Probability mass function • Expectation

10. Beta • Prior of the parameter in Bernoulli distribution • Probability density function • and are pesudo counts Refer to note1.pdf

11. Dirichlet distribution • Prior of parameters in multinomial distribution • Probability density function • are pesudo counts

12. Univariate Gaussian distribution • Model continuous variables • Probability density function • Expectation and variance

13. Multivariate Gaussian distribution • Defined on a continuous random vector • Probability density function • Expectation and covariance

14. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

15. Exponential Family Distribution • A class of distributions sharing a certain form • Natural parameters and sufficient statistics • Special cases: Bernoulli, Beta, multinomial, Dirichlet, Gaussian • Moment generating property Refer to note2.pdf

16. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

17. Conjugate Prior both Beta both Dirichlet both Gaussian From the same distribution Refer to note3.pdf

18. Outline • Important Distributions • Exponential Family • Conjugate Prior • Biased and Unbiased Estimators

19. Estimator Bias • Bias of an estimator • Unbiased estimator and biased estimator • Example: MLE for Gaussian mean and variance Refer to note4.pdf