1 / 13

COMP 4332 Tutorial 5 March 11 CHEN Zhao zchenah@ust.hk

Classification and evaluation. COMP 4332 Tutorial 5 March 11 CHEN Zhao zchenah@ust.hk. Outline. Classification process Evaluate models Automating experiments . Classification process . Classifier parameters. Training Phase. Training data. Classifier Training. Model.

zola
Télécharger la présentation

COMP 4332 Tutorial 5 March 11 CHEN Zhao zchenah@ust.hk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Classification and evaluation COMP 4332 Tutorial 5 March 11 CHEN Zhao zchenah@ust.hk

  2. Outline • Classification process • Evaluate models • Automating experiments

  3. Classification process Classifier parameters Training Phase Training data Classifier Training Model Testing Phase Predicted scores/labels Testing data Model

  4. What determines a model? • 1. Training Data • Different preprocessing methods lead to different training data • GIGO: Garbage in, garbage out • 2. Classifier type • SVM/Decision Tree/KNN/Naïve Bayes/etc. • 3. Parameter of the classifier • Kernel in SVM • Regularization coefficient in Linear Family • K in KNN

  5. Evaluation: Is my model good? • Evaluation metric: accuracy, AUC, precision-recall • Evaluation options: • Train-Test Split • Use train • Use test • Random split • Cross Validation • 3/5/10 folds • Leave-one-out

  6. Evaluation for KDDCUP 2009 • Training data released first, test data quite late. • Submit at most 5 times per day to get AUC score on 10% of them. • Train on training data, and test on training data? • Split training data into two datasets: one for training and one for testing. • Cross validation.

  7. How AUC is calculated? All are classified as positive, so tp/pos = 1 • Sort the predicted scores: • -- score of i-th sample, and • -- true label (-1 or 1) of the i-th sample • For each , use it as a threshold: • For samples 1 to i, classify them as negative (-1) • For sample i+1 to n, classify them as positive (+1) • Calculate Sensitivity = tp/pos, Specificity = tn/neg and a point in the curve is obtained. • Calculate the area under the curve. https://cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf

  8. The Result Table >Thousands of combinations of * Preprocessing * Classifier type * Parameter

  9. Automate the experiments! • Case: • Change in LibLinear with 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 100.0, 1000.0 (13 values) • Do it 13 times in your command line? • Or write a script?

  10. DEMO: evaluation.pycross validation for evaluation • Cross validation • Average AUC score for different C in LibLinear

  11. A few tips on programming • Use command line tools of the classifiers and execute them in a script. • Weka can also be used as command line tools. • LibSvm/LibLinear/FEST/sgd/etc. all command line style • Store your result gradually in case of crash in the middle. • Python’s logging module/just plain text files • Make your experiments repeatable. • Set the seed of random number generators

  12. Assignment 2 • Task: Do cross validation on training set. • Reference: tutorial 5 and its example code. • Report – As concise as possible: • what you do, • what you get (result), • and findings/analysis • Deadline: 26 March 2014 • Submission:emailtozchenah@ust.hk

  13. Project 1 Schedule • Project 1 Results Deadline: 25 March 2014 • Assignment 2 Deadline: 26 March 2014 • Do cross validation on training set • Project 1 Report Deadline: 31 March 2014 • Report and what you have tried and more importantly what you have found • Preprocessing steps • Classifiers • Ensemble methods

More Related