Neural Networks: Reduction to Practice

1. Neural Networks:Reduction to Practice • Robert J. Marks II Baylor University CIA Lab School of Engineering

2. Few are perishing ... • In 1995, DAILY rates were • 6 Evolutionary Computation Papers • Per Day, • 11 Fuzzy Papers Per Day, • 20 A.I. Papers Per Day, • 25 Neural Networks Papers Per Day & • 34 Computational Intelligence Papers • Per Day

3. Continued Controversy of fuzzy logic, neural networks, ... approximate reasoning, and self-organization in the face of dismal failure of traditional methods. This is pure unsupported claptrap which is pretentious and idolatrous in the extreme, and has no place in scientific literature.” “The image which is portrayed is of the ability to perform magically well by the incorporation of `new age’ technologies Professor Bob Bitmead, IEEE Control Systems Magazine, June 1993, p.7. <bob@syseng.anu.edu.au> < http://keating.anu.edu.au/~bob/>

4. The Wisdom of Experience ... ??? • “(Fuzzy theory’s) delayed exploitation outside Japan teaches several lessons. ...(One is) the traditional intellectualism in engineering research in general and the cult of analyticity within control system engineering research in particular.” E.H. Mamdami, 1975 father of fuzzy control (1993). • "All progress means war with society." George Bernard Shaw

5. "If we knew what it was we were doing, it would not be called research, would it?" A.Einstein

6. “In theory, theory and reality are the same. In reality, they are not.”

7. Better Than Average (?) U.S. News & World Report says that a poll of university professors found that 94% of the respondents thought that they were better at their jobs than their average colleague. U.S. News & World Report 16 Dec 96 p26

8. Steel Industry: Cold Rolling Mill Process Cold rolling mill flattens a steel strip to a desired thickness Cho, Cho & Yoon, “Reliable Roll Force Prediction in Cold Mill Using Multiple Neural Metworks”, IEEE TNN, July 1997.

9. Cho, Cho & Yoon, “Reliable Roll Force Prediction in Cold Mill Using Multiple Neural Metworks”, IEEE TNN, July 1997. Cold Rolling Mill... • Neural Network predicts the roll force 30% to 50% better. (Pohang Iron & Steel Co., Korea)

10. Intelligent Control for a Steel Plant • Coating Process Control • Controlling Alloying Thermal Cycle • Induction Furnace Control • Sollac & CRAN (Research Centre for Automatic Control of Nancy) • Bloch, Sirou & Fatrez, “Neural Intelligent Control for a Steel Plant”, IEEE TNN, July, 1997. Estimation of induction temperature

11. A Political Application • Robert Novak syndicated column • Washington, February 18, 1996 • UNDECIDED BOWLERS • “President Clinton’s pollsters have identified the voters who will determine whether he will be elected to a second term: two-parent families whose members bowl for recreation.” • “Using a technique they call the ‘neural network,’ Clinton advisors contend that these family bowlers are the quintessential undecided voters. Therefore, these are the people who must be targeted by the president.”

12. Robert Novak syndicated column Washington, February 18, 1996 (continued) “A footnote: Two decades ago, Illinois Democratic Gov. Dan Walker campaigned heavily in bowling alleys in the belief he would find swing voters there. Walker had national political ambitions but ended up in federal prison.”

13. Plasma Etch Process Control • Controls dielectric etcher drift • > 50 time variant variables DEC & NeuralWare Card, Sniderman & Klimasaukas, “Dynamic Neural Control for a Plasma Etch Process”, IEEE TNN, July 1997.

14. Modelling the Environment Neural networks are used to monitor the interactions between ozone pollution, climate & crop ozone sensitivity United Nations Economic Commission for Europe Roadnight, Balls, Mills & Palmer-Brown, “Modelling Complex Environmental Data”, IEEE TNN, July 1997.

15. Train Ticket Machine Maintenance in Hong Kong Why? • 2.45 million people daily. • Increase 1% each year • Machine breakdowns cause delays in service Liu & Sin, “Fuzzy Neural Networks for Machine Maintenance in Mass Transit Railway System, IEEE TNN, July 1997.

16. Hong Kong Mass Transit Railway Corporation System Liu & Sin, “Fuzzy Neural Networks for Machine Maintenance in Mass Transit Railway System, IEEE TNN, July 1997.

17. Neural Networks in Telecommunication Software • Assesses reliability telecommunications software. > 13 million code lines. • Failure prone software modules identified. • Khoshgoftaar, Allen, Hudepohl & Aud, “Application of Neural Networks to Software Quality Modeling of a Very Large Telecommunications System”, IEEE TNN, July, 1997 Nortel & Bell Canada Florida Atlantic U

18. Avionics • ADALINE is used to optimize the engine control of Concord. • Who: Alan Gerber, University of London (Queen • Mary College) late 60’s. • Source: Professor Igor Aleksander • Head of Neural Systems Engineering (EEE Department) • Pro-Rector (External) • Imperial College, London

19. Pattern Recognition in Aerospace • The Boeing Airplane Company uses an ART-1 neural network system (NIRS) for the identification and retrieval of 2-D and 3-D representations of engineering designs. • Avoids redesign of existing parts and tools • Production solid model data base > 55,000 entries • 2-D data base > 95,000 entries S.D.G. Smith, R. Escobedo, Michael Anderson, T.P. Caudell, “A Deployed Engineering Design Retrieval System Using Neural Networks”, IEEE TNN, July 1997.

20. Neural Networks for Police Classification • The Democrats held their 1996 presidential convention in Chicago. • Twenty six years before, in 1968, the Chicago presidential convention was marred by violent clashes between demonstrators and police. A presidential commission called the conflict a “police riot”. • In 1968, Richard J. Daley was mayor of Chicago. In 1996, his son, Richard M. Daley, was mayor. • In order to belay a repeat incident at the 1996 convention, the Internal Affairs Department of the Chicago Police Department used a neural network to classify `bad cops’ who might provoke conflict.

21. Scientific American , December 1994, (p.44). “THE (NEURAL NETWORK) PROGRAM FORECASTS WHETHER EACH OF THE 12,500 OFFICERS ON THE FORCE IS LIKELY TO BEHAVE IN A MANNER SIMILAR TO NEARLY 200 COLLEAGUES WHO WERE DISMISSED OR RESIGNED UNDER INVESTIGATION DURING THE LAST FIVE YEARS FOR ACTIONS RANGING FROM INSUBORDINATION CRIMINAL MISCONDUCT.”

22. A total of 91 officers were identified. They were to enroll in a counseling program. The neural network results, though, were chal- lenged by the police union. The neural network, as a “black box”, contained no causal mechanism to specify the reason or reasons the classification of potential ‘bad cop’ was made. The neural network lacked an explanation facility.

23. Power System Dynamic Security Assessment Problem: In a power system, what contingencies may cause power system violations and stability? System operator aid. i.e.brownouts & blackouts! The describing coupled set of nonlinear differential equations are computationally intensive.

24. Neural Net Solution ... • Train a neural network to emulate the complex equations. • Development & Use: • B.C. Hydro & • 1293 buses • Hydro Quebec • 963 buses Reference: Mansour, Vaahedi & El-Sharkawi, IEEE TNN, July 1997

25. VAR Flow A neural network was used to control the VAR flow to Vancouver Island, Canada Done by: Geoff Neily

26. Short Term Load Forecasting • Problem: Forecast the power demand of a given • geographical area. • If under - expensive power must • be purchased elsewhere • If over - must sell power in an uncertain market • Features: current temperature, forecaster temperature • day of week (holiday?), current load, humidity

27. ANNSTLF:Artificial Neural Network Short-Term Load Forecaster Actual (solid) vs. forecast (dashed line) for a seven day interval. (X 104 MW vs. hours) Khotanzad, Afkhami-Rohani, Abaye & Matatukulam, “ANNSTLF - A Neural Network Based Electric Load Forecasting System” IEEE TNN, July 1997.

28. 1. Alabama Electric Cooperative 2. Allegheny Power System 3. BC Hydro (Canada) 4. Bonneville Power Administration (WA) 5. Buckeye Power Inc. 6. Central & Southwest Corp. 7. City Public Service of San Antonio 8. Detroit Edison 9. Entergy (Lousiana) 10. Houston Lighting and Power Company 11. Idaho Power 12. Illinois Power Company Khotanzad, Afkhami-Rohani, Abaye & Matatukulam, “ANNSTLF - A Neural Network Based Electric Load Forecasting System” IEEE TNN, July 1997. Who is forecasting loads using ANNSTLF?

29. 13. Kansas City Power & Light 14. Kentucky Utilities Company 15. Madison Gas & Electric 16. Metropolitan Edison 17. Nevada Power Company 18. New England Power Exchange 19. North East Utilities 20. Northern Indiana Public Service Company 21. Ottertail Power Company 22. PECO Energy (PA) 23. Pennsylvania Power & Light Company 24. Potomac Electric Power Company (WA DC)

30. 25. PJM Interconnection (PA) 26. Public Service Electric & Gas (New Jersey) 27. Rochester Gas & Electric 28. Salt River Project (Arizona) 29. San Diego Gas & Electric 30. Southern California Edison 31. Southern Company Services (Alabama) 32. Tennessee Valley Authority 33. Texas Utilities Electric 34. Western Area Power Administration (CA) 35. Wisconsin Power & Light

31. Chang, Han, Valverde, Griswald, Duque-Carrillo & Sanchez-Sinencio, “Cork Quality Classification System Using a Unified Image Processing Fuzzy-Neural Methodology”, IEEE TNN, July, 1997. Wine Cork Classification First Class 3rd Class 5th Class 8th Class: holes cracks bugs

32. Cork Classification Steps... • Morphological filtering • Contour extraction & following • Fuzzy-neural network classifier • Result: 6.7% rejection vs. 40% traditional Cork image and corresponding contours Chang, Han, Valverde, Griswald, Duque-Carrillo & Sanchez-Sinencio, “Cork Quality Classification System Using a Unified Image Processing Fuzzy-Neural Methodology”, IEEE TNN, July, 1997.

33. Finance, Neural Nets & Dart Throwing A Random Walk Down Wall Street W.W. Norton & Co, NY 1973 Sixth Edition, 1996 Burton G. Malkiel, Princeton

34. Advice ... "Stocks have reached what looks like a permanently high plateau." Irving Fisher, Professor of Economics, Yale University, 1929. “October. This is one of the peculiarly dangerous months to speculate in stocks. The others are July, January, September, April, November, May, March, June, December, August, and February,” Mark Twain

35. Factor Models for • Tactical Asset Allocation • Factor models are widely used in portfolio management. Performance differentials between the main asset classes (bonds vs equities) can be explained in terms of changes in fundamental economic • and financial variables. • This project uses neural • networks instead of regression • analysis to model relative • performance between the main • asset classes on the basis of • their exposure to a set of (17) • economic and financial factors. Dr. Apostolos-Paul Refenes, London Business School

36. Factor Models for Tactical Asset Allocation... • The neural models significantly outperform multiple linear regression in terms of forecasting accuracy. • Initially for the UK markets and at a later stage at the international and global level, portfolios are reset on a monthly basis. • Non linear methods for variable selection, and for analysing the sensitivity of differential returns to changes in the independent variables have been developed for this. • In use, since 1995, by Postel (Hermes) • Investment Management FOR THE • CHIEF ECONOMISTS OFFICE AT HERMES Dr. Apostolos-Paul Refenes, London Business School

37. Arbitrage Models for Tactical Asset Allocation • Arbitrage models are finding increasing use in tactical asset allocation as an alternative to factor models. • The basic idea is to exploit short-term pricing anomalies between the different asset classes. Dr. Apostolos-Paul Refenes, London Business School

38. Arbitrage Models for Tactical Asset Allocation... • This is analogous to a hedging strategy designed to exploit short-term over reactions to external events. A model for the UK, exploiting daily pricing anomalies between equities and gilts was completed on January 1995. • A simple long/short trading strategy is used to achieve excess returns of 30% better than buy-and-hold and up to 100% better when adjusted for risk. • Societe Generale. IN USE SINCE 1995 - INITIALLY AT SOCGEN NOW AT T. K. HOARE BROKERS. T. K. HOARE BROKERS Dr. Apostolos-Paul Refenes, London Business School

39. Nonlinear Cointegration in European Equity Futures • A nonlinear co-integration model of the FTSE with a basket of European indices was developed on daily data. • The residuals of the cointegration are modelled as a nonlinear function of exogenous variables (e.g. interest rate volatility, oil price changes, etc) selected via ANOVA and neural network analysis. Dr. Apostolos-Paul Refenes, London Business School

40. Nonlinear Cointegration in European Equity Futures... • Trading the mispricing on a weekly basis with simple long/short strategies yields annualised returns of over 20% net of transaction costs. • Robust neural network models are used to obtain smooth equity curves. The methodology is being extended to other markets and asset classes. • With: CitiBank., IN USE SINE 1996 - FOR PROPRIETARY TRADING AT EUROPEAN EQUITY DERIVATIVES. LATER MODELS ARE ALSO IN USE AT DEUCHE-MORGAN GRENFELL SINCE JAN. 1997.

41. Forecasting Intra-day Volatility for Option Pricing • Multivariate neural models produce estimates of implied volatility for option pricing for futures contracts. • High frequency tick-data is used. • The neural networks give significant performance improvements in terms of forecasting accuracy over time-series models & regression. • Sensitivity analysis is used to verify the plausibility of the neural models, and to provide closed form representations of the pricing formulae. Dr. Apostolos-Paul Refenes, London Business School

42. Modelling and Trading UK Gilts vs. Equities. • This project models differential returns between UK equities and Gilts as a bivariate time series problem. • The approach is purely technical. The input variables being technical indicators on the differential (normalized) levels. • The resultant network architecture uses a trend following indicator and an oscillator as inputs and attempts to switch between the two, thus exploring changes in dynamics. Dr. Apostolos-Paul Refenes, London Business School

43. Modelling and Trading UK Gilts vs. Equities... • The "hybrid" neural network system outperforms both indicators showing a profit of approximately 60 percentage points (net of transaction costs) over 3 years in out-of-sample data against significantly lower figures for the trend following systems with a higher Sharpe ratio and a smaller drawdown. Dr. Apostolos-Paul Refenes, London Business School

44. “Money is better than poverty, if only for financial reasons.” Woody Allen Tactical Intra-day Currency Trading • Neural networks are used to generate buy/sell signals by switching between trend following and reversal-based indicators which model the two primary market states. • Tick-data is first transformed into "variable time" (e.g. periods of 100 ticks) before additional indicators are computed both within and between each period. Dr. Apostolos-Paul Refenes, London Business School

45. Tactical Intra-day Currency Trading... • For an institution such as a bank with relatively low transaction costs the results show significant profitability. • Software and algorithms have been installed including • software for checking high-frequency data integrity, • transformations to variable time, • construction of technical indicators, • bayesian neural networks, and • network/trading rule evaluation (Customers: Proprietary) Dr. Apostolos-Paul Refenes, London Business School

46. Currency Portfolio Models • A portfolio of non-linear and linear models are used to predict 6 and 12 hour deviations from a 12 hour running mean. • With the use of 12 hour changes from the current (12 hour) trend positive returns are obtained with linear models and simple trading strategies.

47. Currency Portfolio Models... • A family of trend-following and mean-reverting indicators are modelled as "states" in a Hidden Markov process which switches between the two on the basis of performance deterioration differentials. • A portfolio approach is taken with up to 20 technicals which is "tilted" towards good performing models. • The results produce much smoother equity curves than either family alone. • (Customers: Proprietary) • Dr. Apostolos-Paul Refenes, London Business School

48. Dr. Apostolos-Paul Refenes, London Business School Term-structure Models of Eurodollar Futures Neural networks are used to model the "volatility factor" in the term-structure of Eurodollar futures. The "volatility factor" is the third principle component. It represents a flexing of the yield curve on a portfolio of short, medium and long maturity contracts.

49. Dr. Apostolos-Paul Refenes, London Business School Term-structure Models of Eurodollar Futures... • This component is shown to be mean-reverting and it is linked to volatility among other factors. • The neural network model estimates variations in this component which are then used as signals to reset the portfolio. • IN USE SINCE 1996. NOW BEEING REDEVELOPED FOR DRESDNER BANK AND BANQUE NATIONALE DE PARIS.

50. Portfolio Replication for Risk Management “We took risks. We knew we took them. Things have come out against us. We have no cause for complaint.” Robert F. Scott - found in his diary after the party froze in Antarctica • With modern derivative portfolios, Monte Carlo simulation and bootstrapping is often required to measure risk exposure. • For large portfolios of over 1,000 derivative and synthetic instruments the computational requirements for the simulations are often unrealistic. Dr. Apostolos-Paul Refenes, Senior Research Fellow, London Business School