1 / 53

Minimum Classification Error (MCE) Approach in Pattern Recognition

Minimum Classification Error (MCE) Approach in Pattern Recognition. Wu Chou, Avaya Labs Research, Avaya Inc., USA. Outline (1/2). Introduction Optimal Classifier from Bayes Desicion Theory Discriminant Function Approach to Classifier Design Speech Recogniation and Hidden Markov Modeling

kinipela
Télécharger la présentation

Minimum Classification Error (MCE) Approach in Pattern Recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Minimum Classification Error (MCE) Approach in Pattern Recognition Wu Chou, Avaya Labs Research, Avaya Inc., USA

  2. Outline (1/2) • Introduction • Optimal Classifier from Bayes Desicion Theory • Discriminant Function Approach to Classifier Design • Speech Recogniation and Hidden Markov Modeling • Hidden Markov Modeling of Speech • MCE Classifier Design Using Discriminant Functions • MCE Classifier Design Strategy • Optimization Methods • Other Optimization Methods • HMM as a Discriminant Function • Relation Between MCE and MMI • Discussions and Comments

  3. Outline (2/2) • MCE TRAINING BASED ON EMBEDDED STRING MODEL • String-Model-Based MCE Approach • Combined String-Model-Based MCE Approach • Discriminative Language Model Estimation • SUMMARY

  4. Introduction • The advent of powerful computing devices and success of statistical approaches • A renewed pursuit for more powerful method to reduce recognition error rate • Although MCE-based discriminative methods is rooted in the classical Bayes’ decision theory, instead of a classification task to distribution estimation problem, it takes a discriminant-function based statistical pattern classification approach • For a given family of discriminant function, optimal classifier/recognizer design involves finding a set of parameters which minimize the empirical pattern recognition error rate

  5. Introduction • Why we take this approach to design classifier? • We lack complete knowledge of the form of the distribution • Training data are inadequate • How to do? • Formulating the problem of self-learning into a classification problem which consists of optimal partitioning of the observation space into regions, Xk, for which the expected risk , R, is minimized • Then we apply generalized probabilistic decent algorithm to achieve the goal

  6. Optimal Classifier from Bayes Desicion Theory C1 C2 CM random 要分類 : x 不確定是 Ci，但被分到 Ci 的機率 但，我們並不知道標準答案

  7. 定義 loss function : 可以想成 Class i 與 Class j 的 distance, 將 Class i 的observation分到 Class j，分錯的 cost 假設 Class i 是正確答案， 則將 x 分錯而得到的cost之expectation (1) Optimal Classifier from Bayes Desicion Theory

  8. Optimal Classifier from Bayes Desicion Theory 當我們作決定時 雖然我們並不知道正確的答案，但可算出作此決定需付出的代價 (2) 如何作出較正確的決定？ 雖然不知道正確答案，但付出的代價愈小，則愈正確 【Decision Rule】 (3)

  9. Optimal Classifier from Bayes Desicion Theory 在SR及許多application中，我們常用的 loss function Posterior Probability (5) 所以【Decision Rule】可以改寫 Bayes’ risk MAP decision (6)

  10. Optimal Classifier from Bayes Desicion Theory OK!! 若 Posterior Probability知道，一切好辦  over 但一般來說，Posterior Probability 需有已知 class 的 labeled training data來估測 (這是不容易取得的) 本來是classifier design的問題  distribution estimation problem 由Bayes’ Theorem estimate the a posterior probabilities for any to implement the maximum a posterior decision for minimum Bayes risk (7) 可省略！

  11. Optimal Classifier from Bayes Desicion Theory • 三個 issue： • Classifier Designed 必需正確估算distribution的parameters，但是，real-world中，distribution常為了容易處理而妥協，使用較簡單或較容易作運算的distribution 如：Gaussian • Real-world中，distribution的parameter一定由『有限』的 training data set來估算，但這需要一個大前題：當training data set 的size改變時，訓練出來的parameter要能保持一致 • unachievable • 否則，則需要一定數量的 training data set 來使parameter較為可信賴，但由於data sparse • unachievable

  12. Optimal Classifier from Bayes Desicion Theory • Despite the conceptual optimality of the Bayes decision theory and its applications to pattern recognition, it can’t always be accomplished in practice • Most practical “MAP” decisions in speech and language processing are not true MAP decisions

  13. Discriminant Function Approach to Classifier Design 先只考慮 2-class 定義 discriminant function 分類用 One well-studied family of discriminant function is the Linear discriminant functionwhich has computational advantages (9)

  14. Discriminant Function Approach to Classifier Design More generally (10) (11)

  15. Discriminant Function Approach to Classifier Design 再來考慮 M-class (12) 也就是說，我們要一組『最佳discriminant functions』 (13) When the loss function is specified

  16. Discriminant Function Approach to Classifier Design This is quite different from the distribution estimation based approach in pattern classification

  17. Score from Acoustic Model Word Sequence Acoustic Feature Score from Language Model Best Word Sequence Speech Recogniation and Hidden Markov Modeling • A decoder performs a maximum a posterior decision

  18. Speech Recogniation and Hidden Markov Modeling • Basic components: • Acoustic Feature Extraction： • Used to extract the features from waveform. • We use to represent the acoustic observation feature vector sequence. • Acoustic Modeling： • Provides statistical modeling for the acoustic observation X. • Hidden Markov Model is the prevalent choice. • Language Modeling： • Provides linguistic constraints to the text sequence W. • Based on statistical N-gram language models

  19. Speech Recogniation and Hidden Markov Modeling • Decoding Engine： • Search for the best word sequence given the feature and model • This is achieved through Viterbi decoding Discrete observation Probability Word String State Sequence Continuous density HMMs

  20. Speech Recogniation and Hidden Markov Modeling • Hidden Markov modeling is a powerful statistical framework for time-varying quasi-stationary process and a popular choice for statistical modeling of speech signal

  21. SPEECH RECOGNITION AND HIDDEN MARKOV MODELING • Three basic problems have to be resolved： • The evaluation problem • estimate the probability • The decoding problem • find a best state sequence q • The estimation problem • estimate HMM parameters from a given set of training samples(ML based algorithms such as Baum-Welch al.)

  22. MCE Classifier Design Using Discriminant Functions (19) MCE classifier design based on 3 steps

  23. MCE Classifier Design Using Discriminant Functions • Misclassification measure (20) Generally we use

  24. MCE Classifier Design Using Discriminant Functions • <proof>

  25. MCE Classifier Design Using Discriminant Functions • Loss function (21) (22)

  26. MCE Classifier Design Using Discriminant Functions • Classifier Performance Measure (23) (24)

  27. MCE Classifier Design Using Discriminant Functions If posterior probability is used Then the Bayes’ minimum risk is (25) X 在 Class k 的機率不可最大，也就是說分錯的 loss

  28. MCE Classifier Design Using Discriminant Functions If posterior probability is used Then the Bayes’ minimum risk is (26) Empirical loss

  29. Optimization Methods • Expected Loss (27) We use GPD-based minimization algorithm to minimize it (28)

  30. Optimization Methods 若滿足下面三個properties，則 收斂

  31. Optimization Methods • Empirical Loss (31) (32)

  32. HMM as a Discriminant Function 使用HMM當作discriminant function (34) discriminant function利用 有三種方式來產生 (35) (36) (37)

  33. HMM as a Discriminant Function

  34. HMM as a Discriminant Function 假設 Maintain HMM 原有的constraints

  35. HMM as a Discriminant Function 所以我們使用parameter transformation來保留這些constraints

  36. HMM as a Discriminant Function , discriminant adjustment of the mean vector

  37. HMM as a Discriminant Function

  38. HMM as a Discriminant Function

  39. HMM as a Discriminant Function , discriminant adjustment of the variance

  40. HMM as a Discriminant Function

  41. HMM as a Discriminant Function

  42. HMM as a Discriminant Function

  43. HMM as a Discriminant Function • How to design the step size? • If the step size is too large, the classifier will be degraded at the start and sequential learning cannot be made successful • If the step size is too small, the convergence speed of the algorithm is too slow and it is practically not useful • It’s difficult to design it, the general solution is still lacking

  44. HMM as a Discriminant Function • Why we normalize mean vector? • The magnitude of variances can vary in the range between 100 and 10-5 . • If using a constant step size for all mean vectors, the algorithm will either not converge or will be too slow to become practically useless • This takes away the dependencies on the variance variations

  45. Relation between MCE and MMI

  46. Relation between MCE and MMI

  47. Relation between MCE and MMI

  48. Relation between MCE and MMI

  49. Relation between MCE and MMI

  50. Relation between MCE and MMI

More Related