Learning to Rank
E N D
Presentation Transcript
Learning to Rank From Pairwise Approach to Listwise Approach
Agenda • Introduction • Ranking Problem • Ranking • Pairwise Ranking (Brief) • Listwise Ranking • Probability Models • Permutation Probability • Top one Probability • Results
Introduction • Construct model or method that learns to rank • Area of use: • Anti Spam • Product Rating • Expert Finding • ...
d_1^i d_2^i d_3^i . . d_n^i Introduction • Ranking Problem – Document retrieval Documents: {d_1, d_2, ... d_n} Ranking of documents Ranking System Query: q
Query Q D = {d , d , ...., d } 1 2 n Instance : (d , d ) 1 2 Pairwise Ranking • Classification of objects Relevance label Classification Model
Pairwise Ranking • Support Vector Machine • Ranking SVM • Boosting • RankBoost • Neural Network • RankNet
m (i) (i) L (y , z ) i=1 Listwise Ranking • Training (1) (m) Q = {q , ...., q } Queries d = {d , ...., d } Documents y = {y , ...., y } Judgements x = {x , ...., x } Features f (x ) Score Func. z = {f(x ) , ...., f(x )} Scores (i) (i) (i) (i) n 1 (i) (i) (i) (i) m (i) (i) 1 n T = {(x , y )} (i) i=1 (i) (i) (i) 1 n (i) j Listwise loss function (i) (i) (i) (i) n 1
Listwise Ranking • Ranking • New Query : q • Associated Docs. : d • Feature vectors : x • Trained rank. Func. : f (x ) • Rank documents in descending order ( i’ ) ( i’ ) ( i’ ) ( i’ ) j’
Probability of pi given s o(S ) o(S ) n Pi(j) Pi(k) P (pi) = s n j=1 k=j Permutation Probability • f : s probability distribution pi = <pi(1), pi(2), ...., pi(n)> s = (s , s , .... s ) 1 2 n
Top one prob. of j P (j) = P (p). s s P(1)=j,p n Top one Probability • Probability of being ranked on top of list
f = f x = f (x ) z (f ) = {f (x ), ..., f (x )} w (i) (i) w j j (i) (i) (i) w w 1 1 ListNet • Optimizing loss function • Neural Network as model • Gradient Descent as optimization alg. w = neural network
Results • TREC • Web pages from .gov domain • OSHUMED • Documents and queries on medicine • CSearch • Data from commerciel search engine
Results • NDCG – Normalized Discounted Cumulative Gain • Relevance judgements > 2 • Korrekt – Delvist korrekt - Ukorrekt • MAP – Mean Average Precision • Relevance judgements = 2 • Korrekt - Ukorrekt
Results • NDCG on TREC
Results • NDCG on OSHUMED
