Outline

2. Outline • Field Research Project, 2007 – 2010 • Research outcomes • Project outcomes • Research gaps and questions

3. OTTAWA, ON CORNWALLIS, NS Pearson Peacekeeping Centre (PPC) • Founded in1994; established to support Canadian contribution to peace and security • Celebrating 15 years in 2009 • Independent not-for-profit • International reputation as centre of excellence in inter-agency approach to research-led education, training and capacity building in peace operations

4. Field Research Project • Why did the PPC undertake this project? • A multi-phased research project aimed to identify the philosophies, assumptions, theories, values, and concepts that drive stakeholder methods for measuring intervention activities (2007-2009); • Interview-based approach; research resulted in a book, a training module, and a handbook for practitioners; • The book produces a discourse of measuring and includes civilians (governmental, non-state actors, humanitarians, developers), civilian police, military, and recipient populations;

5. Interview QuestionsBosnia-Herzegovina • In your opinion, how effective was the peace operation in Bosnia? • What were your needs at that time? Were your needs met by the peace operation in the short-term? How? • Were your needs met by the peace operation in the medium to long-term? How? • At the time, did you understand the efforts of the international community here in Bosnia? • Remembering back to this time, what were the short-term and longer-term effects of the international community becoming involved in Bosnia? • How did these effects impact your life? (for example, your places, your community, your culture, your groups/families)

6. Interview QuestionsBrussels, London, Paris, Ottawa, Washington D.C., Canberra • What principles, concepts, values or philosophies guide the development of MOEs? What guidance is there (i.e. doctrine, policy or strategy papers)? What guidance or policies are needed? What would be helpful? • In your experience, what effects are measured? Why should these effects be chosen? How are these effects measured? When should they be measured? • What terminology does your organization use to describe measurement activities? • Are there synergies between international stakeholders and their measuring systems (US inter-agency, UK, NATO, UN, EU, OECD-DAC, OSCE, etc.)? • What timeframe does your organization work to, when measuring its activities? How is that timeframe harmonized with other stakeholders?

7. Interview Questions con’tBrussels, London, Paris, Ottawa, Washington D.C., Canberra • What measurement methods are the most appropriate in crisis environments? • Why should the various stakeholders involved in ‘integrated’ peace operations and/or crisis management measure the effectiveness of their activities? • How would you council your successors/counterparts regarding measures of effectiveness (MOEs) in their deployments? • Can you identify any future trends in the area of measurement? • Are you aware of other work in this area? What gaps exist in this field? What would be helpful to you or your organization?

8. Research Outcomes • The problem is not having appropriate mechanisms and tools for measuring, rather that there are political obstacles that impede appropriate measurement of success, progress, and effectiveness. • There is an R&D funding disparity between stakeholders to develop mechanisms and tools for measuring both outcomes and impacts. • Collectively, the international community is attempting to use two dimensional measuring systems to capture the complicated and interconnected nature of the complex environment of operations. • Measuring ‘schools’ compete with one another (quantitative vs. qualitative), and there is a need to prove that one is more appropriate than the other. • Quantitative measuring is seen as easy, verifiable and irrefutable, whilst qualitative measuring is seen as difficult, costly and refutable (tangible vs. intangible). • Worldviews and identities of stakeholders matter just as much as what is being measured (everyone believes they are “right”).

9. Research Outcomes • The analysis identified the following issues: • Most stakeholders are not aware of their own lenses through which they measure intervention activities; • There are two dominant measuring modalities: internal (performance, auditing) and external (impact, effects upon recipient populations/host nations); • There is value in considering the comprehensive intervention space as a common “tradespace”; • Forward progress in this common tradespace is built upon utility of understanding (not unity of effort, but unity of understanding between stakeholders); • There is a growing need for shared knowledge (not just information) between stakeholders to support the cooperative elements of the common tradespace.

10. Research Outcomes • The doctrine of the international community offers guidance for comprehensive norms, standards, and codes that will transform interests, values, services, and products into a common tradespace. • The notion of a tradespace is a collocation of the terms trade-off and play space and is a framework for analyzing the complex resources, costs, and provisioning involved in very large projects with multiple stakeholders, objectives, and interdependent variables. • Trade-offs are an exchange of one thing in return for another, especially the relinquishment of one benefit or advantage for another considered to be equally or more desirable; play space is the area bounded by the functionality and purpose of those stakeholders involved in that space. • Tradespaces are multi-variant interdependent spaces recognized for a collection of processes that span multiple sectors in which stakeholders each have their own core objectives.

11. Research Gaps • What do stakeholders have to give up and to take on in order to function within the common tradespace? • How can stakeholders shift from information management towards knowledge management to support more meaningful measurement of intervention activities? • How can recipient populations become meaningfully engaged in the measurement process? (especially salient for security and defence stakeholders). • Further examination of the political obstacles impeding appropriate and inclusive measurement.

12. Project Outcomes • Book: Measuring what Matters in Peace Operations and Crisis Management (McGill-Queen’s University Press, 2009) • Training module • Handbook • Supporting website: www.peaceoperations.org/measuring

13. Steps towards strengthening research and methodological approaches • Continue building consortium of stakeholders involved in measuring what matters; • Be aware of our own organizational lenses through which we assess, plan, implement, and evaluate our intervention activities; there are other lenses besides our own; • Share our lessons with one another: become learning stakeholders (build learning into stakeholder efforts); • Strengthen the common tradespace by making available open-source, free, shared, web-based mechanisms and tools, glossary of terms, and data.

14. Compendium of tools and mechanisms: peaceoperations.org/measuring

15. Order the book: Measuring What Matters in Peace Operations and Crisis Management http://mqup.mcgill.ca/book.php?bookid=2358