Semi-supervised Learning on Partially Labeled Imbalanced Data

1. Semi-supervised Learning on Partially Labeled Imbalanced Data Jianjun Xie and Tao Xiong May 16, 2010

2. What Problem We Are Facing Six data sets extracted from six different domains Domains were removed in the contest They are all binary classification problems They are all imbalanced data sets Percentage of positive labels varies from 7.2% to 25.2% This information was removed in the competition They were significantly different from the development sets They all have one known label to start with

3. Datasets Summary Final contest datasets

4. Stochastic Semi-supervised Learning Condition: Label distribution is highly imbalanced, positive labels are rare Known labels are few Unlabeled data are abundant Approach to A, C, and D: Randomly pick one record from unlabeled data pool as “negative” Use the given positive seed and picked “negative” seed as initial cluster center for k-means clustering Label the cluster as positive where the positive seed resides Repeat above process n times Take the normalized cluster membership count of each data point as the first set of prediction score Our approach when number of labels <200

5. Stochastic Semi-supervised Learning -- continued Approach to A, C, and D: When more labels are known after query, use both known labels and randomly picked “negative” seeds as initial cluster center Label cluster using known positive seeds Discard cluster whose membership is not clear Store the cluster membership of each data points Use normalized positive cluster membership counts as prediction score Our approach when number of labels <200

6. Stochastic Semi-supervised Learning -- continued Approach to B, E, and F: Randomly pick 20 unlabeled data as “negative” labels for each known positive label. Build over-fit logistic regression model using above dataset Repeat above random picking and model building process n times Final score is the average of n models. Our approach when number of labels <200

7. Supervised Learning Using Gradient Boosting Decision Tree (TreeNet)

8. Querying Strategy One critical part of active learning is the query strategy Popular approaches: Uncertainty sampling Expected model change Query by committee What we tried: Uncertainty sampling + density based selective sampling Random sampling (for large label purchase) Certainty sampling (try to get more positive labels)

9. Dataset A: Handwriting Recognition Global score = 0.623, rank 2nd. Pie Chart Title Column Chart Title

10. Dataset B: Marketing Global score = 0.375, rank 2nd. Pie Chart Title

11. Dataset C: Chemo-informatics Global score = 0.334, rank 4th. Passive learning. Pie Chart Title

12. Dataset D: Text Processing Global score = 0.331, rank 18th. Pie Chart Title

13. Dataset E: Embryology Global score = 0.533, rank 3rd. Pie Chart Title Column Chart Title

14. Dataset E: Embryology Performance gets worse with more labels Newly queried labels did too much correction to the existing model This phenomenon was common in this contest Global score = 0.533, rank 3rd. Pie Chart Title

15. Dataset F: Ecology Global score = 0.77, rank 4th. Pie Chart Title Column Chart Title

16. Dataset F: Ecology Performance gets worse with 2 more labels at beginning Most of the time, too many small queries do more harm than good to global score Pie Chart Title

17. Summary on Results Overall rank 3rd. Pie Chart Title

18. Discussions How to consistently get better performance with only a few labels across different datasets How to consistently improve model performance with the increase of labels in a given dataset Does the log2 scaling give too much weight on first few queries? What about every dataset starts with a little bit more labels?