Validation of Credit Risk Models

1. Validation of Credit Risk Models March 2010 Valere Baerts Business Architect CPF 1

2. AGENDA • From regulation to standard risk management • A generic Model Management Framework • Validation of a Credit Risk Model: the building blocks • Summary and conclusions

3. AGENDA • From regulation to standard risk management • A generic Model Management Framework • Validation of a Credit Risk Model: the building blocks • Summary and conclusions

4. From regulation to standard risk management Basel one, two, three …. Так что? The Basel Accord(s) refers to the banking supervision agreements (recommendations and regulations), Basel I (1988) and Basel II (2004) issued by the Basel Committee on Banking Supervision (BCBS). The Basel Accords are implemented in ~ 100 countries worldwide. The Central Bank of Russia intends to implement also these accords but (amongst others) due to the financial crisis no exact planning is known (to me).

5. From regulation to standard risk management • What do these “accords” try to do? • Quantitative element (called Pillar 1 in Basel II) • Make sure that there is enough capital available in Bank for stress circumstances. • The failure of a bank impacts the global financial system: links to other banks (Interbank market), credit crunch (macro economic effects) and “poor” depositors loosing their (part) of their savings. • As a consequence, governments try to manage the risks in banks by requiring a minimum level of capital relative to its risk assets; in other words to place a limit on the ability of a bank to gear its balance sheet. • Qualitative element (not in Basel I, called Pillar 2 and 3 in Basel II) • Risk Governance (internal processes) • and transparency (disclosure of data)

6. From regulation to standard risk management • How is (regulatory) capital calculated? • Depending on which Basel approach the Bank chooses or has to use, there are 3 options to calculate capital: • Standard approach: for smaller not complex banks • - values of parameters to calculate capital are predefined by regulators • Foundation and Advanced approach: for larger, more complex banks • - values of (some) parameters to calculate capital are model based

7. From regulation to standard risk management • Key Drivers of credits (*) used as input into the capital calculation: • the borrowers’ probabilities of default (PD)( ~ rating of borrower ) • (IRBF and IRBA) • loss given default (LGD) and exposure at default (EaD) on an exposure-by-exposure basis (IRBA and for retail portfolio also IRBF) • Maturity (IRBF and IRBA) • (K) = [LGD * N [(1 - R)^-0.5 * G (PD) + (R / (1 - R))^0.5 * G (0.999)] - PD * LGD] * (1 - 1.5 x b(PD))^ -1 . (1 + (M - 2.5) * b (PD)) • Risk weighted assets = 12.5 * K * EAD • (*) We call this key drivers as these parameters mainly determine the credit risk related to lending products (loans, credit cards …). As such using these parameters in your credit process is “good practice” independent if the bank applies Basel II or not !!

8. From regulation to standard risk management Credit Risk Borrower characteristics Facility characteristics Who are we lending to? Probability of Default How much exposure will we have should the borrower default? Time to contractual repayment? What is the % we expect to loose taken into account recovery on collateral? Maturity Exposure at Default Loss Given Default

9. From regulation to standard risk management Use of Key Drivers in Credit Process

10. From regulation to standard risk management • How do we calculate the “Key Drivers” of Credit Risk? • The drivers in essence classify the customer on a scale of creditworthiness and loss probability. • how likely is it that the borrower will pay back his loan? PD • what will the exposure be when borrower defaults? EaD • what will the loss be given the borrower defaults? LGD • Key word in these questions is “will”, in other words we need to look into the future to find answers. The calculation of these drivers is equal to making a prediction. • In general predictions can be “expert” based or “model” based … in both cases however the output is mainly based on historical data ( the credit experts opinion is most likely influenced with experiences from the past) • …… so finally the models arrived in our story

11. From regulation to standard risk management The only thing we know about the future is that it will be different. (P. Drucker) • A (any) model is a simplified version of reality. It is in general a theoretical construction that represents processes by a set of variables and their relationships (in case of credit risk models the process can be described as behavior of a customer related to repayment). • What can go wrong with a model ( = model risk *) • Errors in the model development process (wrong data, assumptions,..) • Errors in implementation of models (bugs) • Wrong usage of models 
 • Due to these shortcomings the bank will refuse good customers, accept bad customers, price products to cheap, build up a bad performing portfolio, calculate to high or to low provisions etc… • * Model risk can be considered as a subset of operational risk

12. From regulation to standard risk management How do we manage “model risk”? Source of inspiration = Basel II regulation = good risk management practice Regulatory Requirements on “Validation of internal estimates” (B II §§500-505) “Banks must have a robust system in place to validate the accuracy and consistency of rating systems, processes, and the estimation of all relevant risk components…”[Basel II, §500] “Banks must regularly compare realized default rates with estimated PD’s for each grade and be able to demonstrate that the realized default rates are within the expected range for that grade.” [Basel II, §501] “ …… So “validation” is the prescribed solution but how can we bring this concept into practice?

13. AGENDA • From regulation to standard risk management • A generic Model Management Framework • Validation of a Credit Risk Model: the building blocks • Summary and conclusions

14. A generic Model Management Framework • The purpose of a Model Management Framework is to: • Describe a generic model life cycle (see next slide) • - Allocate responsibilities to entities involved in the model life cycle • Which entity develops transactional models? • Which entity develops portfolio models? • Who validates the model? (internal – external) • Define key functions • Model owner, Modeller, Implementation Manager, Validator • The function of validator is always separated from all other model management functions (also requested by regulators)

15. A generic Model Management Framework

16. A generic Model Management Framework • Model design includes: • - description of model requirements • - data collection and methodological choices • - design of the mathematical model (including testing) • definition of policies on model usage, monitoring and reporting • Implementation includes both technical (ICT) and process (organizational) aspects. • Once a model and its policies are implemented, they need to be monitored on an ongoing basis (minimum frequency of monitoring are part of the aforementioned policies of monitoring). The monitoring can result in minor changes (maintenance) or in a full review of the model.

17. AGENDA • From regulation to standard risk management • A generic Model Management Framework • Validation of a Credit Risk Model: the building blocks • Summary and conclusions

18. Validation of Credit Risk Model: the building blocks • Components of a validation exercise: • Validation of the quantitative aspects • - Data collection • - Predictive model • Ratings and calibration • Validation of the qualitative aspects • - Policies • - Implementation • Process of a validation exercise

19. Validation of Credit Risk Model: the building blocks • Validation of the quantitative aspects – Data • Data are often the “Achilles' heel” in model development. For this a validation exercise should always focus in depth on potential data issues. • Examples of questions to be answered: • - Is a randomly selected hold-out sample used? • Is the sample representative for the population targeted by the model? • Is the targeted population unchanged with respect to the design? • - How are good clients, bad clients defined? • - How are cure events treated (observations which turn bad, then cure and become good again, and subsequently turn bad a second time .. count as only one bad observation?) • - How are outcome and observation period defined? • - Is the number of missing values and deleted observations acceptable? • - Do all variables have an intuitive relationship with good/bad assessment? • Do all variables have an acceptable predictive power? • Is reject inference used? • - Are the variables not highly correlated?

20. Validation of Credit Risk Model: the building blocks • Validation of the quantitative aspects – Predictive model • In the next step of the validation one should independently verify the discriminatory (ranking) power of financial and non-financial sub-models and the total power on default and non-default basis. • For this step the same methods are used as in scorecard/model development (ref. presentation “Credit Risk Scorecard Development”) • Examples of questions to be answered: • Are cut-off points of the scorecard well described and argumented? • Is the power of the scorecard on the whole population acceptable? • Is the test of the power based on the hold-out sample? • Is the distribution of model scores acceptable? • Are there significant differences between the distribution of model scores for the hold-out sample and the development sample?

21. Validation of Credit Risk Model: the building blocks • Validation of the quantitative aspects – Ratings and Calibration • Calibration of a model is the mapping from the scores, generated by the multifactor analysis or any other ranking technique, into PD’s. • Some questions to be answered: • How are default percentages calculated? • Does the calibration function decrease in a monotone manner? • Is the average predicted model PD for the portfolio close to the central tendency (for TTC models) or to the expected default percentage for the next year (for PIT models)? • Are cut-off’s used (worst or best ratings skipped)? • How are the rating classes defined?

22. Validation of Credit Risk Model: the building blocks Validation of the qualitative aspects – Policies & Implementation (1) In this step of the validation exercise we mainly focus on how the developed model is brought into production. This contains as well IT aspects as business aspects (manuals, procedures etc ..). Examples of questions to be answered: Are measures taken and controls implemented to organize the access to the model in such a way that only authorized personnel have access to read, use and change the model? Are the types of transactions/counterparts/cases for which the model is intended clearly described? Are the entities that will use the model listed? Does the policy clearly mentions if and when usage of the model is mandatory? Are measures taken and controls implemented to guarantee that model inputs are correct and consistent with the data used to develop the model?

23. Validation of Credit Risk Model: the building blocks Validation of the qualitative aspects – Policies & Implementation (2) …. some more questions: Does the policy describes if and under what circumstances the output of the model can be overruled? Is the model output used in an appropriate way, is specification given on how and when the output of the model should be used in the process? Is the reporting regarding model related information adequate and timely? Are the reports that will be created and distributed to monitor the model and the underlying portfolio (content, author, frequency and distribution) described?

24. Validation of Credit Risk Model: the building blocks • Process of a validation exercise • - Read available design/review documentation • - Check recommendations of previous validations, audit and regulators • Interview the model stakeholders • Independently recode (part of) the model • - Benchmark the modeller’s analysis and calculations • Perform ad hoc testing (assess the implementation and its operational risks) • - Document the validation exercise including conclusions and recommendations

25. AGENDA • From regulation to standard risk management • A generic Model Management Framework • Validation of a Credit Risk Model: the building blocks • Summary and conclusions

26. Summary and conclusions • Model management, translated as limiting model risk, starts with defining a clear framework containing the model life cycle, roles and responsibilities of stakeholders. • While doing a validation exercise following checks are deemed mandatory: • Data validation (representative, complete and appropriate) • Statistical validation (discrimination, calibration and out of sample-performance) • Validation of model implementation and usage (Timely, correct and correct usage?)

27. Summary and conclusions • Depending on the importance of the model, the depth of the validation can be extended IF data and time are available. • Issues not discussed: • - Timing and frequency of validation exercises. • Quality of underlying processes ( how well are defaults captured? How accurate are data registered?). • Reject inference. • Stability of model through time (aging of portfolio and changing environment). • Benchmarking against external ratings. • Blind testing (experts). • Stress testing. • Validation techniques are and will always improve but will never take away model uncertainty. In the light of the famous Pareto principle (80/20 rule) I believe that the basics discussed today, can give bank management and regulators an acceptable level of comfort related to model risk.

28. конец! Вопросы?