1 / 16

Efficient Feature Subset Selection with Local Search

The process of optimizing feature space by selecting a minimal subset from a given set of features, enhancing prediction performance and interpretability. Various approaches such as filters, wrappers, and embedded methods are discussed, with a focus on local search following a wrapper approach using genetic algorithms. Experimental design using the "Arcene" dataset is detailed, highlighting the use of Genetic Algorithm Components (GAlib) and Weka's J48 algorithm. Future study suggestions include more re-runs of GA search and better initialization techniques.

sophie
Télécharger la présentation

Efficient Feature Subset Selection with Local Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Feature Subset Selection with Local Search GokcenCilingir

  2. Feature Subset Selection • Definition: • Eliminating redundant and irrelevant features from a given a set of features • Process of choosing a minimum subset of M features from the original set of N features (M ≤ N), so that the feature space is optimally reduced according to a certain criterion (Ex: accuracy of the induced classifier is maximal)

  3. Feature Subset Selection • Motivation: • Improving prediction and generalization performance of the learning model by defying the curse of dimensionality • Increasing data interpretability • Gaining space and time efficiency in learning

  4. Feature Subset Selection Approaches • Filters • Independence assumption among features • According to a scoring function, features are ranked, and the k highest ranked features are selected. • Independent from the classifier; pre-processing step • Ex: Chi-squared test

  5. Feature Subset Selection Approaches • Wrappers • Search the space of feature subsets using the prediction performance of a given learning machine as the scoring function • Search strategies: Greedy approach, simulated annealing, genetic algorithms Image source: George H. John Ron Kohavi, Wrappers for feature subset selection, Artificial Intelligence, 97:273-374, 1997

  6. Feature Subset Selection Approaches • Embedded methods • Incorporate feature selection with learning, performing selection in the process of training • Implicit feature selection, not a preprocessing step • Ex: Decision trees, boosting

  7. Focus: Local search for feature subset selection following wrapper approach • State definition • A feature subset can be represented as an n-dimensional vector with 0/1 values for absent/present features • Transition model or neighborhood definition • Addition or deletion of a feature (or a number of features) from a subset define its neighbors • Objective function • Goal test/stopping criteria

  8. Genetic algorithms

  9. Experimental design

  10. Data set • A data set called “Arcene”, published in the Feature Selection Challenge carried in NIPS 2003 Workshop on Feature Extraction • Original data set: mass-spectrometry analysis results coming from two classes: patients with cancer (ovarian or prostate cancer), and healthy patients • 100 training and 100 validation instances with 10,000 features.

  11. Experiment: results

  12. Details on GA use • Random initialization • Overlapping populations • Parallel populations with allowed migration • Uniform crossover and point mutation operations are defined • Terminated when a number of generations passed without finding a fitter individual • Population diversities are monitored and preventive action is taken automatically towards local optima traps by gradually increasing mutation rate up to a limit and by changing crossover selection criteria

  13. Implementation • A C++ library for Genetic Algorithm Components (GAlib) by Matthew Wall • C++, Visual Studio 2008 • Weka’s J48 algorithm and chi-squared test

  14. Future study • More re-runs of GA search • Extensions for the current GA use • Better initialization using prior knowledge or other feature selection techniques • More genetic operator choices • Classifier choice can be done more systematically

  15. Thanks for listening! • Any questions?

More Related