1. Minimum Rank Error Language Modeling Jen-TzungChien, Meng-Sung Wu

2. Outline • Introduction • Language model for information retrieval • Minimum rank error model • Experiments • Conclusion

3. Introduction • the language model is useful for investigating the linguistic regularities in queries and documents for information retrieval • But the accuracy of classifying queries into the relevant documents is not concerned with the ranks of the retrieved documents • MCE training is also used in IR. In the MCE procedure, the expected loss function is minimized with probabilistic descent algorithm for optimal Bayes risk

4. Introduction • With MCE, the rate of misclassification is reduced. But rank result is still not consist with the performance measure, i.e. AP • The minimum rank error (MRE) language model is established by a gradient descent algorithm to obtain discriminative retrieval for training queries with minimum expected rank error loss

5. Language model for information retrieval • the ranking is calculated by the likelihood function using the -gram language model • Given a text document , the set of ML n-gram parameters

6. Language model for information retrieval • the document terms are often too few to train reliable ML model. Many words are unseen in the document, leading to zero probabilities in many n-gram events • the smoothed language model is obtained by linear interpolation of the document and background models

7. Minimum Classification Error Model • MCE is a training method based on Bayes decision theory. This method can reduce misclassification by minimize the expect loss with three step procedure • First, a misclassification measure is defined • Second, the misclassification measure is normalized as the classification error loss function ranging between 0 and 1 by the sigmoid function given as follows

8. Minimum Classification Error Model • The third step is to measure the classification performance by calculating the expected loss due to the observed queries and document models • through the descent algorithm, the parameter set can be update with iterative procedure and learning rate

9. Information Retrieval Measures • Receiver Operating Characteristic (ROC) is one kind of measure which consider the true positive rate and false positive rate; Area Under ROC Curve(AUC) gives a value for the ROC curve

10. Minimum rank error model • Average Precision (AP) Versus Rank Error • Minimum Rank Error (MRE) Model • Implementation and Interpretation

11. Average Precision (AP) Versus Rank Error • The information retrieval model can be estimated by optimizing the AP, but the minimization of the expected AP loss function is mathematically intractable • So we develop the rank error loss function instead of the classification error loss

12. Minimum Rank Error (MRE) Model • The MRE training procedure assures the model discriminability in sense of minimizing the ambiguity in the ranking problem • Like in MCE training, three step procedure is performed to estimate the MRE language model • First we define the misranking measure for a relevant document

13. Minimum Rank Error (MRE) Model • The rank error loss function is calculated by substituting the misranking measure into sigmoid function. And the expect rank error is calculated over the entire training set including all query and their relevant documents

14. Minimum Rank Error (MRE) Model • The document model is iteratively updated by the descent algorithm • Considering a logarithm bigram in document model, the differentials are calculated by

15. Implementation and Interpretation • The figure below shows the procedure of MRE language model training for information retrieval

16. Implementation and Interpretation • MRE and MCE are derived as the discriminative learning algorithms from the same Bayes decision theory, but they are different by two aspects • In performance metrics • MRE minimizes the Bayes rank risk based on the rank error loss function • MCE minimizes the Bayes risk due to classification errors • In use of training data • The MRE model uses queries and their corresponding document lists as training samples • MCE considers all irrelevant documents in a rank list

17. Experiments

18. Experiments

19. Conclusion • Most classification systems are based on minimization of classification errors, and thus do not reflect the ranking performance of retrieval systems • This paper focuses on the ranking problem, and presents a new discriminative retrieval model. The experiment results also shows MRE retrieves more relevant documents with high ranks than MCE