Comparative Politics The Comparative Manifesto Project

1. Comparative PoliticsThe Comparative Manifesto Project Giorgio Sirtori Prof. Marco Giuliani

2. CMP’s objective To measure policy positions of all relevant parties competing in any democratic election in post WWII period for OECD and EU members, Central and Eastern Europe and (in the future) Latin America and South-East Asia

3. Manifesto (election program) → the text published by a political party in order to compete for votes in national election Why manifesto? • Parties’ only authoritative policy statements • Indicators of the parties’ policy preferences at a given point in time

4. Two-step process 1. Unitising − cutting text in quasi-sentence The coding unit is a quasi-sentence, that contains exactly one statement e.g.: ″We need to address our close ties with our neighbours (107) / as well as the unique challenges facing small business owner in this of economic hardship (402)″

5. . 2. Coding − find the right code for a quasi- sentence Attribute to each coding-unit one, and only one, category CMP developed a category system composed of 56 categories, grouped in 7 policy areas, designed to be comparable between parties, countries, elections and across time

7. . Manifesto data can be used, and have been used, to provide valid and reliable measurements of party policy position e.g.: Left-right scale estimates from Mapping policy preferences (Budge et al. 2001)

8. CMP left-right scale

9. CMP left-right measure Left-right sore = proportion(right - left) % (R – L)/(R + L) % Scale ranges from -100 to +100

10. CMP left-right scale positions for British parties (1945-1997)

11. Comparative PoliticsShortcomings of locating Political Parties using Party Manifesto Data Irina Dámátár Prof. Marco Giuliani

12. Estimating left-right positions of political parties • The aim of the Comparative Manifestos Project is to place parties on the left-right political spectrum, parting from the information provided within their electoral programmes • Each party’s position is measured as the difference (in percentages) between the right-associated text mentions and left-associated ones; the final scale ranges from -100 to 100 • The identity of left and right manifesto coding categories was determined using a series of within-country exploratory factor analysis of a wide range of coding categories

13. The components of the CMP left-right scale

14. Shortcomings • No indication of the uncertainty associated with any CMP estimate - when placing a policy in a certain point on the left-right dimension, no associated error is indicated • CMP left-right measure consists of pre-defined and fixed scale components - the constituent elements of the left-right scale are defined in the same way for all countries, at all time periods but the meaning of left and right can vary in time and space(e.g.: market regulation is a left issue in the CMP dataset, while in Germany is a valance issue) - CMP measure may not include relevant variables, that can explain much of the parties’ variation, while considering unrelated ones

15. . • The CMP scale combines party positions with party specific measures of the relative salience of the left-right scale, producing a measure that is affected by non-right-left issues mentioned in the party manifesto - positions of the parties are in part determined by coding categories that are not in the CMP scale • The scores which CMP interpret as positional data may indicate in fact the willingness of a party to move in accordance to changing political circumstances rather than their actual position - hence, the measurement made by CMP and the score attribute to a certain party lacks precision

16. . • CMP approach identifies position on the basis of saliency scores; but this can be seen more as a movement of position towards the electorate rather than a party’s true position -to eliminate this shortcoming, Franzmann & Kaiser stress upon the distinction between position issues valence issues matters upon which the electorate’s matters evaluated either negatively opinions are clearly divided or positively by all voters - only position issues can structure the policy space -hence, they classify items based on their position or valance character

17. . • Using factor analysis to compute party positions leaded to highly difficult interpretations when the number of variables out weighted the number of cases (low measure of sampling adequacy) • Addressing this shortcoming by focusing on the economic “super-issue” as equivalent to the left-right scale disregards the fact that left and right are not perceived only in economic terms • Nevertheless, in its second stage CMP followed a two step approach: issues were coded as left or right, and the resulting composition was examined through factor analysis -> to see whether these issues did indeed fit in the assumed dimension - factor analysis is also problematic because aims at a reduction of dimensions

18. Explaining differences between CMP scale positions and Expert Surveys judgements • Computing the position of a party using an expert survey estimate may lead to different results than the ones of the CMP • Why? Although there is no pattern to explain these differences, there are a number of hypothesis that can be considered: • I-st possibility: party policies change between the time of the election for which CMP scored a party manifesto, and the time when the expert survey was conducted

19. . • II-nd possibility: one of the two measurements contains significant error • III-rd possibility: measuring different quantities - party manifestos may have policy preferences not expressed in their election programmes and which hence cannot be captured in the scores provides by CMP - no single pre-defined scale will accurately characterize the left right dimension of politics in all countries - “new” dimensions of politics, such as immigration and environment, whose omission from the measurement can cause inaccuracies (e.g.: until 1980s, in Germany, environmental protection was a valance issue or at least was not ideologically structuring the policy space)

20. Please note that no sheep were harmed in the making of this presentation

21. Comparative politicsExpert SurveysParty Policy in ModernDemocracies Kenneth Benoit Michael Laver Silvia Merisio Prof. Marco Giuliani

22. Analysing Party Competition • Why voters choose to supportspecific political parties? • Why parties do what they do once the election is over? Policy adopted by political parties Public needs and wants

23. Need to estimate party policy positions • Problem of gathering reliable information Election Manifestos How parties behave (e. g. Roll call votes) Surveying party politicians or party supporters Not systematic analysis Extremely expensive and difficult across countries Expert surveys

24. Expert surveys • “Expert” most comprehensive population of countryspecialists • These experts are asked to locate party policy positions, in the party systems of which they have expert knowledge • On a set of predefined policy dimensions

25. Policy dimensions • Economic Policy • Social Policy • European integration • General policy (environment, immigration, etc.) • Specific dimensions for Post-communist countries (e.g. privatization, religion, etc.) • Subjective sympathies/closeness to party • General Left-Right dimension All dimensions are evaluated on a scale ranging from 1 to 20

26. Expert surveys They provide information on party positions Common format Wide range of countries Expert surveys are a systematic way to summarize the judgments of the consensus of experts on the matters at issue

27. Risks The same concept/ dimension can be interpreted in different ways How to assess the content of validity Importanceof broadly equivalent dimensions e.g. economic policy Reasonableness of numbers and comparison of the result with other method

28. The survey Reference to a previous work by Laver and Hunt (1992) • Survey conducted in 1988-89 • 24 countries • Core dimensions on economic and social policies Benoit and Laver (2006) • Survey conducted in 2003 • 47 countries (all European countries from the former Soviet Bloc) • 38 policy dimensions (including specific dimensions for each country) Real continuity between the two works

29. The survey (2) • Inclusion of countries classified as non-fully democracies by Freedom House; e.g. Belarus, Russia, Albania, Moldova, Ukraine • Translation in 24 languages; • Web or paper format.

30. Survey details from Eastern Europe countries

31. Policy dimensionsfrom the expert surveys (numbersprovide the total expert responses on eachdimension)

32. Surveydetailsfromitaly

34. Surveydetailsfromlithuania

36. Conclusions • High response rate in all countries • Wide sample of countries which had never been investigated before • Flexible model • Possibility to use expert surveys for different aims and topics

37. Comparative Politics SHORTCOMINGS OF EXPERT SURVEYS Adelaida Foitik Prof. Marco Giuliani

38. EXPERT SURVEYS – ADVANTAGES -> POPULARITY • economical way of measuring party positions • provide information on party policy positions in a common and standardised format   • can be administered at any time, unlike manifestoes-tied to electoral calendars • as long as experts are willing to respond to surveys, the expert survey methodology may probe topics that do not surface in manifestos or other data sources e.g. internal dissent within a party • quick and easy compared to other methods(content analysis of party electoral programs or legislative behavioural studies) • they reflect the judgements of experts=> weight and legitimacy BUThow valid is the information?? ANALYZE: what,how & when

39. THE VALIDITY OF EXPERT JUDGMENTS Framing survey questions: • What is the basis of the judgments that experts offer? • Do experts answer questions in the way they were intended? -> Budge (2000): 4types of problems • what ‘party’ is being judged by the expert? - Is it the party in the electorate, the party in government or the party organization? • what criteria do experts bring to bear when they judge party positions? - what do abstract labels like ‘left’ and ‘right’ mean to the expert? • do experts judge the intentions of parties or their behavior? • what is the time frame for the judgments that we ask experts to make? ->Fundamental question: how do experts interpret the questions in expert surveys and how do they link substantive knowledge about parties to those questions?

40. REMARK Wildly varying considerations when judging party positions • misleading expert judgments • Solution: expert survey design- good questionnaire will attempt to: • avoid ambiguous terms such as ‘party’ and ‘left-right’ • give them a more circumscribed meaning indicating precisely what judgment should concern • specify time frames explicitly to limit variation on this dimension

41. Cont’d • e.g.: experts may be asked to judge the position of the party leadership on issue X during the past year • e.g. in evaluating the left-right position of parties, experts may be asked to describe what ‘left-right’ means in a particular country or what criterion they used to define this dimension HOWEVER, better expert survey design might not alleviate all of the validity concerns raised by Budge: problems like telescoping (recalling facts for the wrong time period before/after they happened or afterwards) are bound to plague such judgments, even when we provide an explicit time frame Even the most carefully crafted question may still leave an interpretative space for experts that could distort their judgments.

42. I. THE 1999 EXPERT SURVEY OF NATIONAL PARTY POSITIONS ON EU INTEGRATION ->evaluated by Budge, developed by Ray (1999), objective: to assess the positions of national political parties vis-à-vis European integration, run 1996 - 1999 , 15 EU Member States NOTE: in order to minimize response variation due to differential scale interpretations by experts: the question specified • the object that was to be evaluated (the party leadership) • the time frame for the evaluation (1999) • response options were explicitly labeled => the question was designed to put the experts in a common frame of mind so that they would be judging the same object, on the same dimension, at the same point in time

43. CONCLUSION of this survey => encouraging findings- the correlations among experts are on average very high, both within particular countries and across the whole set -> substantial convergence in the judgment criteria that experts use: remarkable consistency in responses, agreement in expert placements of political parties with other measures. => considerable confidence in the expert data

44. II. BENOIT AND LAVER (2006) • instead of using experts’ answers to locate parties within a given policy space, use them to directly map the ideal points of the respondents Focus: the Italian case-rely on two expert surveys conducted with the same methodology in a short time span (2003 ,2006)=> possible to check for the robustness of our results • locate each party on the L-R dimension-all aspects of party policy • policy preferences of parties- 4 substantive policy dimension (economic policy, social policy, the decentralization of decision making and environmental policy ) • “sympathy scale” asked experts to place all parties on a scale indicating their own closeness to each party’s-> to test for any possible respondent bias by checking whether expert placements of parties on substantive dimensions were correlated with their personal sympathy for a party’s policies

45. EXPERT PREFERENCES AND BIAS: AN ANALYSIS - party positions estimated on the basis of survey data are not always reliable -respondents tend to place the parties they like closer to where they locate themselves( = rationalization problem) =>This is a point well recognized in BL: “the classic problem of a sample bias is not a concern [in expert surveys], even if the experts we consult hold strong political preferences, as long as these preferences do not interfere with their expert knowledge” RESULTS: => on a pool of 158 observations (i.e., the number of parties surveyed), 20 biased parties (12.7%) were found. NOTE: almost two-third (65%) of the biased parties are extreme-right ones. HOWEVER: not every extreme-right party is biased (just the 59%) AND not every biased-party is indeed an extreme-right one (one-third of the total)

46. REMARKS • the consequent level of bias (1.2 out of 10parties), may have nothing to do with the “true” characteristics of the different communities of political experts because information drawn from individuals is quite vulnerable to selection bias • the selection bias is often due to low response rates. Instead of trying to increase the rate of response in the hope of decreasing the level of bias (if a selection bias problem is indeed present), we should attempt to give a more circumscribed policy-meaning to the questions administered, avoiding as far as possible ambiguous terms such as, “Left-Right”

47. CONCLUSION of this survey • arationalization problem is indeed present: this bias is relevant because it can jeopardize the usefulness of expert scores for empirical analysis • the ambiguity of a label as Left-Right locations => expert judgments become unreliable • evidence of bias in the expert placements of parties especially against right-wing parties (not necessarily extreme-right ones) • the Left-Right position of a party is less important than the policy preferences of experts (and their variance) to explain the probability of a party being biased HOWEVER, the bias is less pronounced when we pass from a label as “Left-Right” to less abstract policy dimensions

48. CONCLUDING REMARKS • Reliance on expert judgments is an attractive option for measuring complex phenomena such as party positions about policies: they are comparatively straightforward to conduct, but we need to make sure about their validity. • As suggested by Budge’s concerns, those who use expert surveys should be cautious. • We have investigated the shortcomings of the expert survey with particular attention to 2 specific cases: while in the first one the results appear to be encouraging revealing the remarkable agreement among experts about the placement of parties, in the second one there is evidence of bias in the expert placements of parties along the Left-Right dimension • Validity assessment is an exercise that should be a central part of expert survey

49. THANK YOU FOR YOUR KIND ATTENTION