Composite Indicators: The Controversy and the Way Forward

1. Composite Indicators - The Controversy and the way forward Andrea Saltelli, Michela Nardo, Michaela Saisana and Stefano Tarantola European Commission Joint Research Centre of Ispra andrea.saltelli@jrc.it Statistics, Knowledge and Policy: OECD World Forum on Key Indicators Palermo November 2004

2. Prepared with Michela Nardo, Michaela Saisana & Stefano Tarantola Based on: [3] Saisana M., Saltelli A., Tarantola S., 2005, Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, J. R. Stat. Soc. A, 168(2), 1-17. [11] Joint OECD JRC handbook on good practices in composite indictors building.

3. Outline • CI controversy • Composite Indicators as models • Wackernagel’s critique of ESI … • Putting the critique into practice: the TAI example • Conclusions

4. CI controversy • EU structural indicators – scoreboards versus indices

5. Report from the Commission to the Spring European Council 2004, Annex 1 Relative Performance Relative Improvement in Performance (av. since 1999)

6. Relative Performance

7. Assessing policies: Green – Country policy on a good path; Yellow – Country policy on a bad path (expert judgment)

8. Enter the FT analysts … Source: Spring Report, European Commission 2004 Source: Financial Times Thursday January 22 2004

9. Categorisation (star rating[*]) in three groups • LEADERSUK, NL SE, DK, AT,LU • MIDDLE OF THE ROADDE, FI, IE, BE, FR • LAGGARDSIT, GR, ES, PT • done by FT and based likely on same synoptic performance and improvement tables in the Spring Report, 2004, Annex 1 (yellow-green boxes) • [*]Like in the UK NHS hospital rating

10. Can “league tables” be avoided? Or are they an ingredient of an overall analysis and presentational strategy: Long list of 107 Short List of 14 Synoptic tables League tables

11. <<The aggregators believe there are two major reasons that there is value in combining indicators in some manner to produce a bottom line. They believe that such a summary statistic can indeed capture reality and is meaningful, and that stressing the bottom line is extremely useful in garnering media interest and hence the attention of policy makers. The second school, the non-aggregators, believe one should stop once an appropriate set of indicators has been created and not go the further step of producing a composite index. Their key objection to aggregation is what they see as the arbitrary nature of the weighting process by which the variables are combined.>> • “Literature Review of Frameworks for Macro-indicators”, • Andrew Sharpe, 2004, Centre for the Study of Living Standards, Ottawa, CAN.

12. Reviews on methodologies and practices on composite indicators : State-of-the-art Report on Current Methodologies and Practices for Composite Indicator Development (2002) Michaela Saisana & Stefano Tarantola, European Commission, Joint Research Centre Composite indicators of country performance: a critical assessment (2003) Michael Freudenberg, OECD. Literature Review of Frameworks for Macro-indicators (2004), Andrew Sharpe, Centre for the Study of Living Standards, Ottawa, CAN. Measuring performance: An examination of composite performance indicators (2004) Rowena Jacobs, Peter Smith, Maria Goddard, Centre for Health Economics, University of York, UK. Methodological Issues Encountered in the Construction of Indices of Economic and Social Well-being (2003) Andrew Sharpe Julia Salzman Methodological Choices Encountered in the Construction of Composite Indices of Economic and Social Well-Being, Julia Salzman , (2004) Center for the Study of Living Standards , Ottawa, CAN. http://farmweb.jrc.cec.eu.int/ci/

13. Pros & Cons (Saisana and Tarantola, 2002) • Pros • Composite indicators can be used to summarise complex or multi-dimensional issues, in view of supporting decision-makers. • Composite indicators provide the big picture […]. They facilitate the task of ranking countries on complex issues. • Composite indicators can help attracting public interest […] • Composite indicators could help to reduce the size of a list of indicators […].

14. Cons • Composite indicators may send misleading, non-robust policy messages if they are poorly constructed or misinterpreted [… or ] may invite politicians to draw simplistic policy conclusions […] • The construction of composite indicators involves stages where judgement has to be made: the selection of sub-indicators, choice of model, weighting indicators and treatment of missing values etc. […] • There could be more scope for disagreement among Member States about composite indicators than on individual indicators […].

15. Pros & Cons (JRSS paper) “[…] it is hard to imagine that debate on the use of composite indicators will ever be settled[…] official statisticians may tend to resent composite indicators, whereby a lot of work in data collection and editing is “wasted” or “hidden” behind a single number of dubious significance. On the other hand, the temptation of stakeholders and practitioners to summarise complex and sometime elusive processes (e.g. sustainability, single market policy, etc.) into a single figure to benchmark country performance for policy consumption seems likewise irresistible.”

16. Composite indicators as models … and the critique of models

17. Indicators as models … and the critique of models The nature of models, after Rosen

18. Decoding F Formal system N Natural system Entailment Entailment Encoding The critique of models After Rosen, 1991, ”World” (the natural system) and “Model” (the formal system) are internally entailed - driven by a causal structure. [Efficient, material, final for ‘world’ – formal for ‘model’]Nothing entails with one another “World” and “Model”; the association is hence the result of a craftsmanship.

19. Wackernagel’s critique of ESI …

20. The critique of indicators Environmental sustainability Index, figure from The Economist, Green and growing, The Economist, Jan 25th 2001, Produced on behalf of the World Economic Forum (WEF), and presented to the annual Davos summit this year.

21. The critique of indicators: Robustness … Mathis Wackernagel, mental father of the “Ecological Footprint” and thus an authoritative source in the Sustainable Development expert community, concludes an argumented critique of the study done presented at Davos by noting:

22. "Overall, the report would gain from a more extensive peer review and a sensitivity analysis. The lacking sensitivity analysis undermines the confidence in the results since small changes in the index architecture or the weighting could dramatically alter the ranking of the nations.” The critique of indicators: Robustness …

23. The quality of a composite indicator is in its fitness or function to purpose. The economist A. K. Sen, Nobel prize winner in 1998, was initially opposed to composite indicators but was eventually seduced by their ability to put into practice his concept of ‘Capabilities’, ‘the range of things that a person could do and be in her life’.Sen A., 1989, Development as Capabilities Expansion, Journal of Development Planning19, 41-58. The critique of indicators: Fitness

24. The example of the ‘capabilities’ is relevant to the issue: CI are supposedly good at capture complex (someone would say poorly defined) concepts such as sustainability, welfare, achievement of an EU internal market, competitiveness, etc. Said otherwise, complex processes call for scoreboards, and scoreboards cry for an index. The critique of indicators: Fitness

25. In discussing pedigrees matrices for statistical information Funtowicz and Ravetz note (in Uncertainty and Quality in Science for Policy, 1990 [6])“[…] any competent statistician knows that "just collecting numbers" leads to nonsense […] so in "Definition and Standards" we put "negotiation" as superior to "science", since those on the job will know of special features and problems of which an expert with only a general training might miss”. We would add that, however good the scientific basis for a given composite indicator, its acceptance relies on negotiation and peer acceptance. The critique of indicators: Fitness

26. (1) A composite constructed on the basis of underlying indicators with high internal correlation will give a very robust CI, whose values and ranking are moderately affected by changes in the selection of weights, the normalisation method and other steps involved in the analysis (see paper, this conference). Open issues in CI Building 1 – Variables correlation

27. (2) When building composite indicators using automated tools such as factor analysis, one seeks to obtain a set of totally uncorrelated new variables. While this can be a powerful tool to benchmark countries performance, or to produce e.g. leading or lagging synthetic indicators, the interpretation in terms of original variables becomes more difficult. Variables correlation

28. (2) At the same time, it would be very difficult to imagine a composite indicator made of truly orthogonal variables.(3) In a multicriteria context, one would consider the existence of correlation among the attributes of an issue as a feature of the issue, not to be compensated for. A car’s speed and beauty are likely correlated with one another, but this does not imply that we are willing to trade speed for design. Variables correlation

29. (1) Munda, and Nardo, 2003 [12], noticed how weights, customarily conceived as ‘importance’ measures, act in practice as substitution rates, e.g. wi/wj is the ratio of substitution (or compensation) of indicator ‘i’ with indicator ‘j’. Open issues in CI Building 2– Compensability

30. (2) This may be perceived as an important limitation of a CI (e.g. literacy should not be traded with GDP per capita). When one is not willing to accept this kind of trade offs, e.g. when the variable cannot be compensated with another, a multi criteria approach can be applied. Compensability

31. (2) This may be perceived as an important limitation of a CI (e.g. literacy should not be traded with GDP per capita). When one is not willing to accept this kind of trade offs, e.g. when the variable cannot be compensated with another, a multi criteria approach can be applied. See paper for a simplified description of a Condorcet-type of ranking procedure based on Munda, 1995, [13]. This approach produces rankings (ordered sequence of countries) instead of an index. Compensability

32. (3) The ordering thus obtained is only based on the weights, and on the sign of the difference between countries values for a given indicator, the magnitude of the difference being ignored. (4) With this approach no compensation occurs. To exemplify, a country that does marginally better on many indicators comes out better than a country that does a lot better on a few ones because it cannot compensate deficiencies in some dimensions with outstanding performances in others. Compensability

33. Points touched upon in this brief discussion of open issues in CI building are tackled in a forthcoming joint paper from OECD and JRC on composite indicators building. It aims to be a guide to the construction and use of CI. Ongoing work: the OECD JRC handbook

34. Theoretical framework - What is badly defined is likely to be badly measured. Data selection – The quality of composite indicators depends largely on the quality of the underlying indicators. Multivariate analysis – Multivariate statistic is a powerful tool for investigating the inherent structure in the indicators’ set. Imputation of missing data– The idea of imputation is both seductive and dangerous. Ongoing work: the OECD JRC handbook

35. Normalisation – Avoid adding up apples and pears.Weighting and aggregation – Relative importance of the indicators and compensability issues. Robustness and sensitivity – The iterative use of uncertainty and sensitivity analysis during the development of a composite indicator can contribute to its well-structuring. Ongoing work: the OECD JRC handbook

36. Link to other variables – Correlation with other simple indicators or composite indicators. Visualisation – If arguments are not put into figures, the voice of science will never be heard by practical men.Back to the real data – Deconstructing composite indicators for analytical purposes. Ongoing work: the OECD JRC handbook

37. http://farmweb.jrc.cec.eu.int/ci/