Different Levels of Performance Measures for Different Uses in Science and Technology

1. Different Levels of Performance Measures for Different Uses in Science and Technology Evaluation 2005: Crossing Borders, Crossing Boundaries William G. Kennedy, PhD National Research Council Research Associate at the Naval Research Laboratory Washington, DC This work was performed while the author held a National Research Council Research Associateship Award at the Naval Research Laboratory. The views and conclusions contained in this document should not be interpreted as necessarily representing the official polities, either expressed or implied, of the U.S. Navy.

2. Background • GPRA’s purpose: “holding Federal agencies accountable for achieving program results” • OMB “evaluates the effectiveness of agency programs, policies, and procedures” • PART (Program Assessment Rating Tool) • PART measures in budgets and strategic plans

3. Focus on “Programs” • GPRA focused on programs • PART focused on programs • Congress focuses on programs+

4. More than “Programs” • Administration’s policies broader than agencies • OSTP: justify new programs with “national priorities” • PMA latest report begins with establishing clear definition of success for every program. • … All at a level outside and above programs • “Projects” defined level within and below programs

5. Problems with Program Focus • OMB and agencies struggle with PART. • Congress has ignored PART assessments. • Little focus on project management. • Political appointees evaluated based on program execution, but also on their policies & initiatives. • Program Managers do not create their own marching orders and, therefore, should not be evaluated on them. • Degree of accountability should match the scope of authority.

6. “PPP” Proposal • Government use three levels of measurement and evaluation:

7. “PPP” Proposal • Government use three levels of measurement and evaluation: • Program reasonably defined and covered by PART process (with some changes).

8. “PPP” Proposal • Government use three levels of measurement and evaluation: • Program reasonably defined and covered by PART process (with some changes). • Project below programs & reports to programs.

9. “PPP” Proposal • Government use three levels of measurement and evaluation: • Policy above programs, establishes marching orders for programs, has different measures and evaluation methods. • Program reasonably defined and covered by PART process (with some changes). • Project below programs & reports to programs.

10. Examples of Levels & Use

11. Examples of Levels & Use

12. Examples of Levels & Use

13. Examples of Levels & Use

14. Next Steps • Shared with colleagues at DOE

15. Next Steps • Shared with colleagues at DOE • Forwarded to senior officials at DOE

16. Next Steps • Shared with colleagues at DOE • Forwarded to senior officials at DOE • Forwarded to senior officials at OMB & OSTP

17. Next Steps • Shared with colleagues at DOE • Forwarded to senior officials at DOE • Forwarded to senior officials at OMB & OSTP • Requesting your support in interactions with “policy-making” officials. • Paper and references available at http://www.mllab.com/pbm

18. Questions or Comments?