The 2005 UK Workshop on Computational Intelligence 5-7 September 2005, London

1. The 2005 UK Workshop on Computational Intelligence 5-7 September 2005, London L2-SVM Based Fuzzy Classifier with Automatic Model Selection and Fuzzy Rule Ranking Shang-Ming Zhou and John Q. Gan Department of Computer Science, University of Essex, UK

2. Background and Objectives(1/4) • Advantage of SVM: • Parsimonious solutions based on quadratic programming • The challenges : • To apply SVM techniques to parsimonious fuzzy system modelling for regression and classification. • Difficult to link the kernel functions in SVM to basis functions in fuzzy system.

3. Background and Objectives(2/4) • Chen and Wang’s work [Chen and Wang 2003]: • Established this sort of relation for fuzzy classification based on L1-SVM techniques. • Parameters: kernel parameters and regularization parameter not updated optimally from data for fuzzy rule induction. • One objective : • To apply L2-SVM techniques to fuzzy system modelling to optimally learn the parameters from data in terms of radius-margin bound J; • Radius-margin bound: not hold in L1-SVM.

4. Background and Objectives(3/4) • Rule ranking, rule selection: • Rule base structure [Setnes and Babuska 2001] • SVD-QR with column pivoting algorithm and pivoted QR decomposition method [Yen and Wang 1998,1999, Setnes and Babuska 2001]; • Contribution of fuzzy rule consequents: • More effective [Setnes and Babuska 2001] • OLS [Chen et al 1991] • Both rule base structure and contribution of fuzzy rule consequents: • Highly desired [Setnes and Babuska 2001] • Not reported yet in literature.

5. Background and Objectives(4/4) • Another objective: • -values of fuzzy rules: • Contribution of rule consequents; • -values of fuzzy rules: • Rule base structure and contribution of rule consequents.

6. L2-SVM based Fuzzy Classifier Construction (1/10) • Fuzzy Classifier:

7. L2-SVM based Fuzzy Classifier Construction (2/10) • Conditions of Applying SVM to Fuzzy Classifier Construction: • are Mercer kernel; • If are generated from a reference function through location shift, then are Mercer kernel [Chen and Wang 2003]; • leading to Gaussian MFs; • Kernel parameters manually selected in [Cheng and Wang 2003].

8. L2-SVM based Fuzzy Classifier Construction (3/10) • L2-SVM based Fuzzy Classifier: • Parameters optimally updated in terms of radius-margin bound: • The number of rules L, prototypes , weights , bias , and scaling parameters .

9. L2-SVM based Fuzzy Classifier Construction (4/10) • Two quadratic programming problems: 1) st where are Lagrangian multipliers,

10. L2-SVM based Fuzzy Classifier Construction (5/10) 2) st • Radius-margin bound:

11. L2-SVM based Fuzzy Classifier Construction (6/10) • Automatic Model Selection Algorithm

12. L2-SVM based Fuzzy Classifier Construction (7/10) • Extraction Fuzzy Rules from L2-SVM Learning Results • The number of fuzzy rules L is the number of support vectors; • The premise parts of fuzzy rules: where is the jth element of the ith support vector . • The consequent parts of fuzzy rules: where are the non-zero Lagrangian multipliers.

13. L2-SVM based Fuzzy Classifier Construction (8/10) • Fuzzy rule ranking based on L2-SVM learning • R-values of fuzzy rules: [Setnes and Babuska 2001] • Absolute values of the diagonal elements of matrix R in the QR decomposition of firing strength matrix; • -values of fuzzy rules: • Determining the depth of the effect of the rule consequent. • -values of fuzzy rules: • Considering both rule base structure and effect of the rule consequent.

14. L2-SVM based Fuzzy Classifier Construction (9/10) • Fuzzy rule selection procedure • Evaluate the misclassification rates (MRs) of on the validation data set V and the test data set T separately: and ; • Select the most influential fuzzy rules where is the threshold. • Construct a fuzzy classifier by using the influential fuzzy rules selected.

15. L2-SVM based Fuzzy Classifier Construction (10/10) • Fuzzy rule selection procedure (cont.) • Apply to the validation data set V and the test data set T to obtain new MRs and ; • If > , stop selection; otherwise, assign a higher threshold value and go to step 2.

16. Experimental Results(1/6) • Benchmark problem-ringnorm • 2 classes; • 7400 samples; • 20 attributes; • Theoretically expected MR: 1.3% [Breiman 1998] • 400 training samples; 5000 testing samples; 2000 validation samples. • Initial conditions: • C=1; • ; • Learning rates for updating C and : 0.0001 and 0.01 separately • Threshold for updating the radius-margin bound:

17. Experimental Results(2/6) • L2-SVM Induced Fuzzy Classifier: • 249 fuzzy rules generated; • MR: 1.32% on test data set; • Comparison with the well-known methods on generalization performance:

18. Experimental Results(3/6) • Fuzzy rule ranking results:

19. Experimental Results(4/6)

20. Experimental Results(5/6)

21. Experimental Results(6/6) • Fuzzy rule selection results:

22. Conclusions and Discussions(1/1) • To have applied L2-SVM to fuzzy rule induction for classification: • Fuzzy rules optimally generated in term of radius-margin bound. • Efficient way of avoiding the “curse of dimensionality” in high dimensional space. • Two novel indices for fuzzy rule ranking: • Experimentally proved to be very effective in producing parsimonious fuzzy classifiers.