Breakout Session # 309 Joanie F. Newhart, CPCM Chief, Procurement and Contracting

2. The Art of Source Selection: Providing the Best Solution Breakout Session #309 Joanie F. Newhart, CPCM Chief, Procurement and Contracting U.S. Securities and Exchange Commission April 12, 2006 1:20 to 2:20 PM NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

3. Disclaimer The Securities and Exchange Commission, as a matter of policy, disclaims responsibility for any private publication or statement by any of its employees. The views expressed herein are those of the presenter and do not necessarily reflect the views of the Commission or of the presenter’s colleagues upon the staff of the Commission NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

4. Agenda • What Is Source Selection? • Why Is It So Important? • The Five Phases of Source Selection • Acquisition Planning • Solicitation Preparation • Proposal Evaluation • Source Selection • Protests?? • Best Practices • What NOT To Do! NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

5. What Is Source Selection? Source Selection is the decision process used in competitive, negotiated contracting to select the proposal that offers best value to the organization Four Objectives of Source Selection: • Select offer that is the most advantageous to the Government • Maximize competition • Minimize complexity • Ensure a fair and comprehensive evaluation NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

6. Why Is Source Selection So Important? A successful source selection: • Provides a solution that directly contributes to the organization’s success • Minimizes the risk of the acquisition • Saves precious resources • Maintains procurement integrity NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

7. Is Source Selection Art or Science? It Has Aspects of Both! • Science (Objective): • Use of tools • Following a process • Art (Subjective): • Creativity • Judgment Art and science just so happen to share a common destination – truth* *Nora Murphy, http://www.trincoll.edu/zines/tj/tj3.7.96/articles/cover.html NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

8. First Phase: Acquisition Planning Three Questions to Answer: • What is the requirement? • What are the applicable conditions and risks? • What is the plan of action, including competition enhancement and source selection procedures? NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

9. First Phase: Acquisition Planning Who’s Responsible for Preparing the Acquisition Plan? The Program Office, but………. Best practice is to do it with a cross-functional team: contracting, legal, engineers, logistics….. Whoever can add value! NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

10. Evaluation Criteria How the organization measures the various elements of the offers to determine the discriminators of value • Factors must be consistent with the SOW • Factors should be limited to the minimum needed to determine the best value offer (too many factors tends to group the offers) • Factors must be independent, valid, and relevant NOTE: Every evaluation factor is another chance for a protest NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

11. Types of Evaluation Factors • Quality* • Technical (evaluates offer) • Approach • Understanding the requirement • Business (evaluates offeror) • Management • Past Performance* • Resources (staff and facilities) • Price* * required by FAR 15.3 NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

12. Best Practices in Selecting Evaluation Factors • Start with the SOW (but you don’t have to evaluate all areas of the SOW!) • Identify the most important tasks • Identify the tasks which have caused problems in the past • Select areas likely to differentiate between offerors---if all offerors submit the same info, what is the point in evaluating it? (example: audit task) NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

13. Best Practices in Selecting Evaluation Factors • Be sure to include stakeholders in the discussion • Set up a brainstorming session with other contracting colleagues • Want some new ideas? Check out the solicitations on fedbizopps---see what federal agencies are doing NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

14. Second Phase: Solicitation Preparation • Section M – How will the offer be evaluated? • Description of how each factor will be evaluated • Relative weighting of evaluation factors • Section L – What information is needed to evaluate the proposals? (require only the minimum needed) The two sections must be consistent – factors flow in the same order (most important to least important) NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

15. Section M Evaluation Process There are lots of options: • Multi-step process • Go/No-Go Checklist • Oral Presentations • Demonstrations • Site Visits (also known as “road trips”) NOTE: The challenge is: is there enough time? NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

16. Section L Proposal Submission Tell the Offerors how you want them to submit the proposal in as much detail as you can Why? For ease of evaluation--otherwise you will get a huge variety of formats Examples: For Volume I, Technical Proposal, Tab A is xxxx, Tab B is yyyy, Tab C is zzzz For Volume II, Price Proposal, submit price information on the attached excel spreadsheets NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

17. Best Practices for the Solicitation Phase • Have extensive exchanges with industry • Share the draft Solicitation with potential offerors • Have a preproposal conference • Allow for Q&As throughout the process • Allow sufficient time for Offerors to prepare proposals NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

18. Third Phase: Proposal Evaluation Training of the Evaluation Team is Critical! • Acquisition strategy and background • Overview of solicitation (they should have read the solicitation and amendments ahead of time!) • Discussion of source selection process and best value methodology NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

19. Proposal Evaluation Requirements • Proposals evaluated consistently – a comment about something on one proposal should elicit the same comment on another proposal • Offers are not compared by the evaluation team • Proposals evaluated strictly in accordance with the RFP NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

20. Proposal Evaluation Best Practices • Send the evaluation team the Solicitation and amendments in sufficient time for review prior to the evaluation • Prepare a checklist of what should be submitted in the proposals to assist in initial proposal reviews • Develop a POC list of your offerors to assist in contacting them during the evaluation process NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

21. Proposal Evaluation Best Practices • Sequester evaluators until completion as much as you can! • Use “advisors” as necessary • Subject matter experts • Do not score or rank proposals (they just provide strengths/weaknesses) • Inform offerors in Solicitation NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

22. Proposal Evaluation Best Practices • Use color ratings based on aggregation of strengths/weaknesses (be careful how you describe these in the evaluation plan; i.e., for exceptional, “there are no weaknesses” or for acceptable, “strengths outbalance any weaknesses” – you have to follow what you say!) • Evaluators justify every rating with references back to the proposal, including page numbers • Stay away from summary scorecards---somehow they always have errors! NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

23. Proposal Evaluations: What Evaluators should NOT Do! • Provide generic comments: “Although product meets stated requirements in some areas, some major shortcomings were identified.” “Section C.3.2.3 requirements not clearly addressed.” • Rate other than what is in the Solicitation: “It was not made clear in the proposal if there is any knowledge of auditing software.” NOTE: SOW did not require auditing software NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

24. Proposal Evaluations: What Evaluators should NOT Do! • Rate outside the evaluation factor: For a subfactor entitled “Ease of Customization,” evaluator cited a weakness “Vendor requires fee and living expenses for initial training.” • Provide a conclusion with no supporting evidence: “Clearly demonstrated their understanding of the requirements. Excellent, thorough, very specific and extensive explanations. Superior understanding of technical requirements.” “Product has been developed over many years therefore the usual problems associated with customization appear to have been resolved.” NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

25. Proposal Evaluations: What Evaluators should NOT Do! • Provide a comment with no explanation of why it is a strength or weakness: “Product provides standard Windows interface and intuitive layout. Product can run in several configurations.” • Provide a strength or weakness that is really just a fact: “The past performance of the offeror is clearly stated by the eight referenced organizations. They are applicable to our requirements and are relevant in size and scope.” “Vendor only provided 3 interface options.” NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

26. Phase Four: Source Selection The Source Selection Authority’s decision is based upon comparing the proposals against the evaluation criteria in the solicitation • Uses analysis of others • Decision based not upon the ratings, but upon the comments justifying the ratings NOTE: The Source Selection Authority is NOT bound by the findings of the evaluation panel or contract specialist NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

27. Documentation of the Decision • Provides the rationale behind the decision • Compares the successful offer to the other offers based upon the criteria in the Solicitation • Discusses any technical/price tradeoff • Addresses differences between evaluation panel or contract specialist recommendations and decision NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

28. Best Value Best Value – Negative perception in 90s “Typically it means a procurement in which price is negligible or even zero-weight source selection factor and the evaluation criteria are so flexible that award is left up to the total discretion of the agency.” • House Committee on Government Operations, 1991 Report NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

29. Best Value: A TradeOff Process TECHNICAL High Low High Is it worth the extra $$$?? – tradeoff analysis PRICE Is it worth the extra savings?? – tradeoff analysis Low NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

30. The Source Selection Decision • It is an Art, as well as a Science • Ratings, analysis, documentation are tools to support the decision • It is NOT just rehashing previous information, instead it provides conclusions based on that information • It is a stand-alone document NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

31. Fifth Phase: Protests Primary bases for challenge • Technical/price evaluations were not done in accordance with the Solicitation • The technical/price tradeoff decision was unreasonable • The probable price was not well founded and documented to support the tradeoff NOTE: GAO will not second guess the Source Selection Authority’s judgment NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

32. In Conclusion…… The search for best value is the search for truth, but….. as Oscar Wilde said: “The pure and simple truth is rarely pure and never simple” NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

33. Questions? • Call me at 202-551-7303 • E-mail me at newhartj@sec.gov NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management