Toward a standard benefit-cost methodology for publicly-funded S&T programs

1. Toward a standard benefit-cost methodology for publicly-funded S&T programs Crossing Borders, Crossing Boundaries 2005 Joint CES/AEA Conference Toronto, Canada October 26-29, 2005 Jeanne Powell Economic Assessment Office Advanced Technology Program National Institute of Standards and Technology

2. Outline • Background • AEA 2005: Analysis of Differences and Considering an Approach to Comparability • Moving Forward

3. Background • ATP Mission and Project Selection Criteria • Evaluation Framework • Role and Types of Benefit-Cost Analysis • Performance Metrics • Benefit-Cost Studies To Date • Study Similarities

4. ATP Mission …To accelerate the development of innovative technologies for broad national benefit through partnerships with the private sector. Chemistry Biotechnology Electronics Manufacturing Advanced Materials Photonics Information Technology

5. ATP Selection Criteria • Scientific and Technological Merit (50%) • Technical innovation • High technical risk with evidence of feasibility • Detailed technical plan • Potential for Broad-Based Economic Benefits (50%) • National economic benefits • Need for ATP funding • Pathway to economic benefits

6. Short-Term Mid-Term Longer-Term ECONOMIC IMPACTS Total Economic Benefits Benefits to Awardees -1 0 1 2 3 4 5 6 7 8 9 10 or More Years Announce Compe- tition Announce Award Complete Project Post-Project Period What We Measure When (Inputs/Outputs) • Award/participant characteristics • Project funding • R&D partnering • Acceleration of R&D • Innovative technology development • Patents • Publications • Competitive advantage • Prototype products & processes (Outputs/Outcomes) • Commercial activity • New products • New processes • Licensing • Attraction of capital • Strategic alliances • Company growth (Impacts) • Broad national economic benefits • Return on investment • Public • Private • Social • Inter-industry diffusion • Increased GDP/ tax base • Societal impacts

7. Types of B-C Case Studies in ATP • Traditional cash-flow based, return on investment measures • Benefit-to-cost ratios • Internal rates of return • Net present value • Hedonic price index models • Macroeconomic models

8. Benefit-cost Analysis:Return on Investment Measures • Private return • return to ATP project participants on their own investment • Public return • return to nation on ATP investment • Social return • return to ATP project participants and nation on total ATP and private investment

9. Performance Metrics Return on Investment Metrics • Benefit-to-Cost ratio is computed by dividing the present value of benefits by the present value of investments • Net present value (NPV) is the present value of benefits minus the present value of investments • Internal rate of return is calculated by iterative solution for a rate at which the discounted value of investments equals the discounted value of benefits

10. ATP’s B-C Studies To Date • 12 studies published, including quantitative case studies of 28 projects • 2 studies underway, including quantitative case studies of 4 projects • All studies performed by independent contractors

11. ATP’s B-C Studies To Date

12. Study Similarities • Address ATP Mission, emphasizing public benefits to industry customers and end users and need for ATP funding to achieve these benefits • Cash-flow based approaches use: • Good practices consistent with public finance literature; NPV, B:C, IRR • OMB-mandated 7% real discount rate (all but 1 study) and constant $ estimates • Year of ATP project start for base year • Relatively short study periods • Results are not presented as “representative”, but rather as portfolio minimums

13. AEA 2005 • Analyzing Differences and Considering an Approach to Study Comparability

14. AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability • Analyzing four major differences : 1) Study timing relative to ATP funding and uncertainties about future outcomes 2) Identifying the counterfactual and attribution of benefits 3) Which metrics? Public versus social return on investment 4) Adjusting for timing differences across studies and projects

15. AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability • Considering a standard approach based on three criteria: 1) Meeting evaluation objective 2) Quality and accuracy 3) Comparability

16. 1. Study Timing and Uncertainties About Future Outcomes Issues: • Uncertainties concerning technology benefits are reduced over course of innovation process • At time of ATP funding, risks of technical failure (or of meeting only limited technical objectives) are very high. • Market/financial/business risks remain high during and after ATP funding phases of technology development and into commercialization phases • Innovation life-cycle/timing varies by technology area • Earliest studies conducted early in innovation process • No projects had matured • Direct experience with range of possible outcomes (and empirical data about innovation process) was limited

17. 1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies

18. 1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Notes: • Tissue Engineering: 3 technologies made considerable technical progress and commercialization continues to evolve but much more slowly than anticipated; 3 technologies failed to develop and companies dissolved; 1 is in transition to new company, future as yet is uncertain • Component-Based Software: focus on existing products and short-term product life reduced the amount of research required per project and enabled study of more projects

19. 1. Study Timing and Uncertainties About Future Outcomes: Conclusions Under conditions of considerable technical/business uncertainty about future outcomes: • Rigorous, complex economic models may or may not generate useful performance metrics. • Probability-based outcome measures that consider the broad technical, market, and business risks, as well as technology benefits, are likely needed • Metrics are more reliable/capable of precision as projects overcome technical, market, and financial barriers

20. 2. Identifying the Counterfactual and Attribution of Benefits Issue: • Where does project being analyzed start and end? And what is the counterfactual, or baseline? • In theory, investment costs and benefits included in analysis need to measure increment over the counterfactual (what would have happened without the project being measured). • In practice, benefits (e.g., profit from sale of a product embodying new technology) derive from multiple public and private investments and from one or two of many product generations.

21. 2. Identifying the Counterfactual and Attribution of Benefits—Contrasting Studies

22. 2. Identifying the Counterfactual and Attribution of Benefits—Conclusions • Researchers use a variety of mechanisms to model benefits relative to a counterfactual (e.g., increased probability of achieving benefits, acceleration, allocation in accordance with amount or importance of funding sources) • ATP-funded technologies are often embodied in multiple products, each of which resulted from a number of different public/private funding sources and “projects”. Researchers measure benefits of products with strongest ATP linkages and share attribution where needed • Social rate of return has weaker assumption about “causation” than public return on ATP $ does.

23. 3. Which metrics? • Social return metrics • Economics literature compares social rate of return to private rate of return (to see economic spillover) and then further compares private rate of return to private hurdle rate to identify market failure that justifies public funding • Cost-shared projects imply comparison of benefits to both public and private recipients with investments from all sources AND/OR • Return on ATP investment • Directly addresses: What is the ATP program impact? • Involves substantially less historical/company proprietary information than social return metrics

24. 3. Which metrics? Contrasting Studies Issue: • Some studies/contractors emphasize social return metrics; others emphasize public return

25. 3. Which metrics? Contrasting Studies

26. 3. Which metrics? Contrasting Studies

27. 3. Which metrics? Social versus Public Returns—Conclusions • Both have merit for different purposes, and both have advantages and disadvantages • Social return metrics reflect cost-sharing aspect of ATP and other public-private partnership projects; however, they require company-proprietary data and historical data about multiple funding sources (generally only rough estimate is feasible) . • Public return on ATP investment provides measure of direct program impact; however, it requires judgments about of “causation”

28. 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Issue: • Different case studies based cash flow estimates on constant dollars of different years • Although nearly all studies used the same discount rate (7%-real), they discounted cash flow estimates to different points in time (start of each project). (This practice evolved in financial world’s prospective analysis of potential investments and is used in Excel.) • Thus, reported NPV metrics are not comparable across projects.

29. 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example 1: Ranking does not change

30. 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example 2 : Ranking Changes

31. Adjusting to Constant 2005$ NPV in Constant 2005$ = Where: IPDGDP is Implicit Price Deflator for GDP and IPDGDPstudy is IPDGDP for Constant $ year used in study • Q2 2005 IPDGDP – IPDGDPstudy • IDPGDPstudy 1 + x Reported NPV

32. Adjusting Base Year to 2005 Adjusted NPV for Base Year 2005 = NPV in Constant 2005$ x 1.07n Where: n = 2005 minus base year used in study

33. 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example Note: • Rankings “before and after” adjustment are for illustrative purposes only. • NPVs indicate net benefit magnitudes • NPVs do NOT indicate investment efficiency

34. 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Conclusions: • Both computed value of NPV and rank order can change by adjustment to constant $ and common base year • NPV adjustments to constant $ and common base year are easy to make—directly from reported NPV • Comparability and aggregation of NPV results indicate need for such adjustments although practitioners may differ on best approach • IRR and B:C do not require adjustment; however, they cannot be aggregated

35. Moving Forward • Work with contractors/researchers toward greater understanding of need for comparability of B-C results while seeking improved methodologies • Recognize and build on strengths of individual contractors/researchers ; help them overcome inconsistencies with comparability goals • Improve understanding of B-C methodologies among S&T evaluation community • Work with NIST experts in both B-C analysis and standards development

36. For further information: • Call Jeanne Powell: 301-975-4196 • Send e-mail to: jeanne.powell@nist.gov • Visit our website:www.atp.nist.gov • View our publications: www.atp.nist.gov/eao/eao_pubs.htm