Evolving Decision Rules (EDR)

1. Evolving Decision Rules (EDR) Alma Lilia Garcia & Edward Tsang

2. Layout • Motivation • Repository Method procedure • Receiver Operating Characteristic (ROC) • Experimental design • Experimental results • Conclusions Alma Lilia Garcia & Edward Tsang

3. Motivation Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions In recent years computers have shown to be a powerful tool in financial applications, for that reason many machine learning techniques has been applied to financial problems. Genetic Programming (GP) has been used to predict financial opportunities. However, when the number of profitable opportunities is extremely small it is very difficult to detect those cases. Alma Lilia Garcia & Edward Tsang

4. Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions Motivation When the frequency of an event is very low the training data contains very few positive cases in comparison with the negative cases. In such situations, machine learning has limitations to deal with imbalanced environments because this favors negative classifications, which has a high chance of being correct. Alma Lilia Garcia & Edward Tsang

5. Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions The problem with few opportunities Motivation Predictions Predictions Reality Moves from  to + Accuracy = 98.2% Precision = Recall = 10% (Accuracy dropped from 99%) Easy score on accuracy Accuracy = 99%, Precision = ? Recall = 0% Random move from  to + Accuracy = 98.02% Precision = Recall = 1% Ideal prediction Accuracy = Precision = Recall = 100% Alma Lilia Garcia & Edward Tsang

6. Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions Evolving Decision Rules The new population is processed until the maximum number of iterations is reached Initial Population generated at random New population generate a new population of decision tree by mutating and hill-climbing on the rules in the repository and generating trees at random Every Rule Rk whose precision achieve a predefined precision threshold is simplified to remove redundant and vacuous conditions R’k Is compared to the rules in a rule collection R1 R2 … R3 Ra Rb … Rn Every decision tree is divided in rules (patterns) . . . . . . if R’k is a new rule it is added to the rule set R’k Alma Lilia Garcia & Edward Tsang

7. Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions Evolving Decision Rules In order to mine the knowledge acquired by the evolutionary process Repository Method performs the following steps: 1- Rule extraction 2- Rule simplification R1 R2 … Rn The rule Rk is selected by precision; Rk is simplified to R’k Evolve a GP to create a population of decision trees Rα … Rµ 3- New rule detection R’k is compared to the rules in the repository by similarity (genotype) R’k 4- Add rule to the repository If R’k is a new rule R’k is added to the rule repository Alma Lilia Garcia & Edward Tsang

8. Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions Evolving Decision Rules Once the repository of rules have been created, a range of classifications is created by means of dividing the rule collection. The rules will be grouped by a precision threshold called τ τ =0.05 Every sub-collection of rules produces a different classification τ =0.10 R1 R2 … Rn τ =0.90 τ =0.95 . . . Alma Lilia Garcia & Edward Tsang

9. Motivation Evolving Decision Rules Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions ROC space The Receiver Operating Characteristics (ROC) has been used extensively in Machine Learning to measure the performance of classifiers. A single classification produces a point in the ROC space. However, some classifiers are able to produce a range of classifications, in that cases a curve is produced, this moves from the liberal to the conservative area. Alma Lilia Garcia & Edward Tsang

10. Motivation Repository Method Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions ROC Curve The main advantages of using ROC are: • It is able to deal with imbalance classifications • It is able to deal with classifiers that produce a range of classification • Lets the user to calculate the best trade-off between misclassifications and false alarms The Area Under the ROC curve (AUC) has been used widely to measure compare the performance of different classifiers. Slope = μ (1ρ)/(β ρ) where ρ = the % of + cases Alma Lilia Garcia & Edward Tsang

11. Motivation Repository Method Experimental design Receiver Operating Characteristic (ROC) Experimental results Conclusions Experimental design • The aims of this work are: • to show that RM is able to produce a range of solutions capable to suit the investor requirements • To show that EDR is able to classify the minority class in imbalanced environments • The objective of this experiment is to compare EDR performance with EDDIE-ARB. The Arbitrage data set is imbalanced because the minority class is composed by only the 24% of the total cases in the training data and 25% in the testing data set. Alma Lilia Garcia & Edward Tsang

12. Motivation Repository Method Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions Experimental results Evolving Decision Rules for Arbitrage EDDIE-Arb EDDIE-Arb results: Recall= 42%, Precision=100%and Accuracy= 85%. This result is plotted in (0.0, 0.42) Alma Lilia Garcia & Edward Tsang

13. Motivation Repository Method Receiver Operating Characteristic (ROC) Experimental design Experimental results Conclusions Conclusions It has been shown that EDR offers a range of solutions to suit the risk guidelines of the investors. Thus the user can choose the best balance between miss-classification and false alarms according to his/her requirements. This makes EDR a valuable tool for investors in balancing between not making mistakes and not missing opportunities. Alma Lilia Garcia & Edward Tsang

14. Questions? Alma Lilia Garcia & Edward Tsang

15. Confusion Matrix True Positive Rate (recall) = TP/(TP+FN) = 350/(350+200) = 63.6% False Positive Rate = FP/(FP+TN) = 50/(50+400) = 11.1% Precision = TP/(TP+FP) = 350/(350+50) = 87.5% Predictions TN FP Reality FN TP

16. Experimental Details • Machine: Windows XP, 2GB RAM • Program written in Java • Running EDR on Arbitrage data: about 1.5 to 2 hours • Data set: FTSE 100 spot index • Training: 1,006 instances selected from 1991-98 • Testing: 635 instances from 1998-99 • 7 variables