Feature Selection and Extraction

1. Feature Selection and Extraction ☺ Given a set of features S={v1, v2, …, vD}, find a subset of S’ with |S’|=d < D such that J(S’)≥J(T) for any subset T of S, |T|=d. ☺ Given a set of features S={v1, v2, …, vD}, find a set S’ with |S’|=d < D derived from S such that J(S’)≥J(T) for any set T, with |T|=d, derived from S.

2. Feature Selection To exhaustively select d optimal features out of D needs to evaluate D!/[(D-d)!d!] feature sets which is not practical even for small D and d, e.g., when D=20, d=10, 184756 feature sets would have to be considered. Two ways to overcome this problem. • Whitney’s method (1971) 1101-1103. (2)Branch and Bound (1977) 917- 922.

3. Whitney’s Nonparametric Method 1. Use 1-nn decision and leave-out-out error 2. First feature selected should have the smallest error 3. Next feature selected is the one joined with the previously selected features has the smallest error 4. Continue step 3 until d features are selected.

4. Results of Whitney’s on iris, imox, 8OX Data Sets (in # of errors) Feature ordering 3 4 1 2 Iris18 7 7 6 /150 Imox 6 8 7 1 5 2 4 3 81 37 13 8 7 5 4 10 /192 8OX 7 6 3 5 1 2 4 8 17 7 2 1 1 0 0 3 /45

5. Features of Characters 8,O,X,3

6. Features of Characters 8,O,X,3

7. Branch and Bound (1/3) Let the number of features in the original set be n. We have to select a subset of features so that the value of a criterion is optimized over all subsets of size m < n. Let (Z1,Z2,….,Zk) be the k=n-m features to be discarded to obtain an m feature subset. Each variable Zi can take on values in {1,2,….,n} but the order of Zi’s is immaterial, hence we consider only sequences of Zi’s, such that Z1 < Z2 < …. <Zk

8. Branch and Bound (2/3) The feature selection criterion, Jm(Z1,Z2,...,Zk), is a function of the m (=n-k) features obtained by discarding Z1,Z2,...,Zk from the n feature set. The feature subset selection problem is to find the optimum subset {Z1*, Z2*,…, Zk* } such that Jm(Z1*, Z2*,…, Zk* ) = max {Jm(Z1,Z2,...,Zk)} where the criterion J must satisfy the monotonicity, which is defined by J(Z1)≧J(Z1,Z2) ≧…≧ Jm(Z1,Z2,...,Zk)

9. Branch and Bound (3/3) Let B be a lower bound on the optimum (maximum) value of the criterion Jm(Z1*, Z2*,…, Zk* ), i.e., B≦Jm(Z1*,Z2*,…,Zk* ) If J(Z1,Z2,...,Zh) (h<k=n-m) were less than B, then Jm(Z1,Z2,...,Zh,Zh+1,…,Zk)<B for all possible {Zh+1,…,Zk}