Appendix

1. Appendix • Summary Products Delivered On September 30, 2003 • Summary Findings and Recommendations • Business Processes • Tools and Technologies • Human Capital

2. Summary of work products delivered by September 30, 2003 • Business Analysis Executive Summary • Task 1: Project Plan • Project Plan (MS PowerPoint) • Project Plan (MS Project) • Task 2: Business Framework Assessment • Executive Summary • Business Framework Assessment Appendices • Task 3: Business Process Current Environment (Baseline) • Executive Summary • Merit Review (MR) Baseline • Award Management & Oversight (AM&O) Baseline • Performance Assessment & Accountability (PA&A) Baseline • Resource Allocation (RA) Baseline • (Initial) Knowledge Management (KM) Baseline • Applicant Survey Analysis • Task 3: Human Capital (HC) Current Environment (Baseline) • Executive Summary • HC Supply Analysis • HC Workload Analysis update • HC Competency Modeling • HC Management Review • Task 3: Tools and Technology (TT) Current Environment (Baseline) • Executive Summary • TT Baseline Analysis and Appendices • Business Function Baseline • Application Survey • TT Application Survey

3. Appendix • Summary Products Delivered On September 30, 2003 • Summary Findings and Recommendations • Business Processes • Tools and Technologies • Human Capital

4. In merit review, the increasing volume and complexity of proposals requires NSF to re-think how it conducts this core process • Finding: Increases in the volume and complexity of proposals impacts the effectiveness of Program Directors in performing their merit review and award management responsibilities.Volume of proposals has increased by 28 percent in the past 5 years with few adjustments in current staffing levels. Currently, a PD will manage an average of 90 competitive proposal actions, 82 active awards and 67 post-award actions per year. PDs spend 12 percent of their available time on award management and oversight activities. Results of applicant survey indicate that lack of quality feedback on a proposal is major driver of dissatisfaction with the process. Current perception among Program Officers that they have little time to perform other activities critical to their position. Recommendation: In Task 4, NSF should explore alternative methods for conducting reviews that maintain the integrity of the process while seeking to reduce the workload of NSF staff and to minimize the impact on the research community (i.e., virtual panels, etc.). Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

5. In merit review, the increasing volume and complexity of proposals requires NSF to re-think how it conducts this core process (cont) • Finding: Additionally, the increase in volume of proposals is making recruiting and attracting reviewers more difficult. Currently, the research community “donates” $17-20M in time and effort to prepare for panel reviews and/or submit “ad hoc” reviews. Total number of reviews has increased by 15 percent during the past 4 years. Percentage of mail review declines has climbed 29 percent in the past 4 years and mail review return rates have dropped by 6 percent during this same period. Recommendation: In Task 4, NSF should explore ways to increase the available pool of reviewers to accommodate potential increases in proposal volume. • Finding: Compounding the workload issue is the lack of formalized processes for programs that require cross-directorate coordination. Programs that require cross-directorate coordination comprise more than 19 percent of the NSF portfolio. A sample of cross-directorate programs demonstrates that the volume of proposals has increased by more than 50 percent in the past 3 years. Cross-directorate activities are often assigned in addition to regular responsibilities and require additional coordination. Recommendation: In Task 4, NSF should develop scenario models that can be applied across the foundation for collaborative/cross-directorate programs and/or initiatives. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

6. In the future, NSF must formalize the criteria for how awards are managed and decide how best to leverage the project reporting system • Finding: At the Program level, NSF requires more formal procedures for how to manage the technical risk of an award and how to assess the contribution that individual projects are making to the scientific community. NSF manages a combined portfolio of $18.7B that spreads risk among the scientific Directorates and Offices.Award management has limited operating practices for performing programmatic oversight activities such as reviewing annual and final reports, performing site visits, and other award management activities. PD’s currently dedicate approximately 12 percent of their time managing an average portfolio of 82 active awards. Recommendation: In Task 4, NSF should develop more formal procedures at the program level for how to manage and assess the contribution that individual projects are making to the advancement of science. Greater emphasis on award management will require additional resources. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

7. In the future, NSF must formalize the criteria for how awards are managed and decide how best to leverage the project reporting system (cont) • Fundamental to how awards are managed is determining how to better use NSF’s project reporting system and provide guidelines for the types of awards that require greater technical management.Currently, all project reports are required to be reviewed and approved; however, project “results” are not assessed as part of the approval process.Project reporting system was initially designed as a collection system for “results of science” to serve GPRA purposes. The system is currently being employed as a condition for future funding decisions (e.g. CGIs, renewals, pending proposals, etc.). NSF should evaluate the purpose and use of annual and final project reports in order to provide greater value for the Foundation. Recommendation: NSF needs to determine how to better use its project reporting system (PRS) and provide guidelines for the types of awards that require greater technical management. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

8. In order to effectively grow, NSF must more pro-actively manage its administration and management • Finding: Due to NSF’s decentralized management structure, operational planning and execution is often delegated to the Directorates without a mechanism or body to address systemic (i.e., day-to-day) management issues of the Foundation. Currently, ‘annual performance planning’ is performed through the use of various committees (I.e., GIIC-WG, GIIC, SMIG) that include participation from the Directorates. Senior Management Integration Group (SMIG) currently provides management guidance; however, NSF could benefit from an executive management body that is focused on the daily operations of the Foundation. Several internal committees exist to provide recommendations on different aspects of management; however, few address NSF-wide process improvement issues. Recommendation: NSF should consider chartering a team of senior-level managers representing each of the Directorates and appropriate functional areas (e.g., OIA, DIS, DAS, etc.) for the purpose of addressing systemic issues of the Foundation (e.g., eJackets implementation, operational performance management, etc.). Team should be empowered to decide and implement NSF-wide operational changes. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

9. In order to effectively grow, NSF must more pro-actively manage its administration and management (cont) • Finding: NSF has sound business practices in place for measuring performance; however, NSF requires additional metrics in order to effectively manage growth. Specifically, NSF needs to include metrics that focus on its people, the impact on the community, and the “bottom line”. NSF has several key operational metrics for measuring customer service and select financial metrics; however, growth will require that NSF to spend additional time ensuring manageable workloads, timely hiring of key staff, assessing the impacts on the research community and making the necessary investments in administration. Recommendation: NSF requires additional metrics in order to effectively manage growth. Specifically, NSF needs to include metrics that focus on its people, the impact on the community, and the “bottom line”. • Finding: Currently, few Directorates develop Directorate or Division-level goals and plans to support the achievement of NSF’s strategic and operational goals.Most Directorates consider GPRA Annual Performance plans in their management approach; however, few Directorates establish Directorate-level goals that then contribute to the achievement of NSF goals. Perception that operational goals follow a “one-size-fits-all” that do not consider the unique characteristics of each Directorate. Additionally, the current set of goals does not easily translate into meaningful or useful tools for managing performance. Subsequently, few individual performance assessments are tied to agency-wide goals. Recommendation: In Task 4, NSF should consider establishing Directorate- and Division-level operational goals that are subordinated to the NSF-level goals and that consider the unique operational characteristics of each Directorate. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

10. The resource allocation process can be improved by formalizing the priority setting process, exploring alternative technologies for budget formulation, and better managing the timing of its commitments • Finding: Each Directorate or Office follows a different process for ‘setting priorities’ creating the perception that priorities are ‘top-down’ driven.Key to NSF’s ability to respond to and facilitate scientific advancement is the “bottom-up” input received from Program Director’s that feed into the Foundation-wide priority setting process. While some Directorate/Offices have formal input required from Program Directors, others are informal and may have little to no input. Recommendation: In Task 4, NSF should consider developing priority setting guidance based on both internal and external best practices that can be promulgated among the Directorates. • Finding: While the budget execution and reconciliation activities are highly automated, the front-end budget formulation activities are manually intensive.Presently, budget formulation activities are supported by MS Excel (“Thematic” budget spreadsheet) and Word. While these tools have been highly effective because of their ease of use and familiarity to NSF staff, the spreadsheets and documents introduce opportunities for error and rework due to the hand-offs between BFA and the Directorate/Offices. Additionally, the “Thematic” budget spreadsheet has limited utility for budget management. Recommendation: In Task 4, NSF should explore other budget formulation and execution tools. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

11. The resource allocation process can be improved by formalizing the priority setting process, exploring alternative technologies for budget formulation, and better managing the timing of its commitments (cont) • Finding: Currently NSF commits a significant portion of its budget in the last quarter of the fiscal year resulting in additional demand on limited staff resources. While the trend of receiving appropriations is mid-fiscal year or later, 85 to 100 percent of previous year’s funds are available for commitment at the beginning of the FY. Data analysis indicates that proposal receipt and program deadlines are major drivers for commitment patterns with a significant percentage of proposals received in January and July thus driving increased award activity the last quarter of the FY. Commensurately, NSF’s culture is to “conserve” funds throughout the year in order to take advantage of unplanned opportunities and to provide flexibility in funding decisions. Contributing to the “conservatism” is the perception among many Program Directors that they are often unaware of their final budget allocations even after Congress has appropriated funds. Recommendation: NSF should explore setting program deadlines based on award size, community needs and impacts on Foundations operations. Recommendation: NSF should explore establishing Directorate/Office goals for committing funds throughout the year. Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

12. KM can help NSF achieve its organizational excellence goals, however there is little awareness of the benefits associated with a KM strategy • Finding: Different interpretations of KM exist among senior leadership. KM is viewed as a method for achieving improved business results via mechanisms such as best practice transfer, collaboration, and continuous learning. It is also viewed as an enabler to improving external customer relationships by providing stakeholders with access to information/knowledge. Overall, there is limited understanding of the formal concepts, practices, and benefits of Knowledge Management within the organization. Currently, NSF does not have a authoritative KM definition, vision, or strategy • Finding: KM can help NSF achieve its organizational excellence goals.KM can serve as a key enabler to improving organizational performance and managing customer relationships by providing increased access to information/knowledge for both NSF staff and customers. Currently, there is minimal guidance provided to staff regarding the importance of creating, capturing, and using/reusing institutional knowledge, such as best practices and lessons learned. There are minimal resources devoted to ensure the creation, capture, and management of the knowledge generated by NSF Interim Recommendation: As facilitated by the KM Advisory Team, NSF should reach consensus regarding the following -- the purpose of KM, NSF vision and goals for KM, a KM strategy Merit Review Award Management & Oversight Performance Assessment & Accountability Resource Allocation Knowledge Management

13. Appendix • Summary Products Delivered On September 30, 2003 • Summary Findings and Recommendations • Business Processes • Tools and Technologies • Human Capital

14. NSF budgetary controls and IT management could be improved in order to reduce redundancy and cost, ease technology adoption and increase quality of service to staff and customers • Finding: The computing environment within NSF is highly fragmented. 199 non-major and 20 major applications have been identified of which only 23 are well known and monitored. As these applications expand to effect the enterprise, more must be known about them so as not to preclude the use of any particular enterprise technology. Little or no enterprise-wide visibility into IT investments made outside of DIS. • Recommendation: NSF budgetary controls, enterprise management, change control and policies could be improved and consistently applied across the organization in order to reduce redundancy, cost, speed time to adoption and increase quality of service • Finding: NSF Strategic Information Assets are not currently managed at the enterprise level. Instead, data is managed locally by each application resulting in data that is difficult to find, understand, use and trust • Recommendation: Instituting a Strategic Information Management Plan (and people and resources to support it) as well as transforming the existing data architecture into a more flexible configuration (e.g., data warehouse) will produce an architecture that can support future decision management tools, security, business, and application requirements. • Finding: IT Operations support a wide range of IT services and functionalities. Trends indicate that the range of services will continue to grow along with the Foundation. IT budget has remained relatively flat compared to the overall budget; IT headcount has remained relatively flat in keeping with staffing trends within the NSF. • Recommendation: IT budget and headcount should be increased and existing staff skills updated in order to meet current and future IT challenges and responsibilities Computing Environment Strategic Information Assets IT Operations and Staff IT Infrastructure Applications & Technology Source: Booz Allen Hamilton Analysis and 2003 Non-major Application Survey

15. Standard application development methods and supporting technologies should be uniformly applied and used across NSF • Finding: NSF data on PIs, reviewers, NSF employees and partner institutions is currently kept in eight different systems with no one system serving as the authoritative record. NSF has made recent investments in enterprise application, web and portal servers that will provide the fundamental services necessary for flexibility and scalability of future systems. • Recommendation: NSF should continue to emphasize information and application consolidation as well as directory services as a centerpiece for all current and future systems’ access, data and security. • Finding: Configuration and change management are inconsistently applied across the organization. While, change control and configuration management structures are being established in DIS, these functions have not been adopted to any significant extent outside of DIS. • Recommendation: NSF should establish a change control group with binding decision-making authority for IT projects across the organization. • Finding: NSF core processes are supported by 20 major applications and a number other non-major applications. Both major and non-major applications are not supported by a strong integration model. Technologies supporting applications tend toward being disjointed in architecture, function and are often redundant in capabilities and information manipulation. Programming language standards and re-usable application architectures are in the process of being adopted by DIS. • Recommendation: NSF should continue to expand standards to involve application development across the entire Foundation, thereby reducing costs and increasing probability of integration across Directorates. Computing Environment NSF Progress Strategic Information Assets NSF Progress IT Operations and Staff IT Infrastructure NSF Progress Applications & Technology Source: Booz Allen Hamilton Analysis and 2003 Non-major Application Survey

16. NSF has many applications per supported service and inconsistent distribution of sharable services resulting in redundant applications, increased cost and decreased quality of service • Finding: NSF accomplishes business goals along 13 lines of business (LOB) and 58 sub lines of business. There are an average of 11 services and 12 applications per LOB (more than a 1:1 ratio of applications to services). The implication is that a different application is developed/used for each business function that is needed to be performed. • Recommendation: By adopting a service-oriented approach to managing IT, NSF could provide a more robust services offering, reduce cost, the number of supporting applications, time to implementation as well as provide a mechanism for using good technology across the enterprise.One exception to reducing target applications is in Technology Management where analysis has revealed that tools are insufficient to maintain current and future levels of development and operation. • Finding: 12 of 28 available Data Services are not implemented to any significant extent. Most of these services relate to the management of strategic information assets of which data management comprises a significant portion • Recommendation: NSF should work toward increasing data-related service offerings and staff across the enterprise including middleware components and meta-data management • Finding: While Network Services are relatively stable and reliable, 19 of the available 34 network services will require some measure of reconstruction. Most of these services relate to systems monitoring and performance, fault tolerance/high availability, critical events monitoring, asset detection, management and software distribution, patch and updates management. • Recommendation: NSF should continue to work toward the management of network operations and services around a unified enterprise management system Functional Services Data Services Network Services Security Services Source: Booz Allen Hamilton Analysis and 2003 Non-major Application Survey

17. NSF should continue to secure the its computing environment by increasing the use of security services and requiring them across the Foundation NSF Progress • Finding: Security services have been greatly improved over the past year. While all 20 of the available security services currently require some level of reconfiguration, this is not unusual given prevalent security concerns across the government/industry. “Availability” and “critical events” monitoring are unevenly distributed across the enterprise for major and non-major systems. Security Services are not required for application development outside of DIS. 73 of 199 non-major applications do not have any password protection. • Recommendation: Given the fragmented nature of NSF’s computing environment, more attention should be given to the security risks posed by non-major systems. Additionally, Security services in all software development should be required within the organization. Functional Services Data Services Network Services Security Services Source: Booz Allen Hamilton Analysis and 2003 Non-major Application Survey

18. Appendix • Summary Products Delivered On September 30, 2003 • Summary Findings and Recommendations • Business Processes • Tools and Technologies • Human Capital

19. NSF should use the workforce supply analysis to inform the development of future target workforce objectives, and to identify gaps between current supply and future workforce requirements • Finding: NSF draws upon a diverse mix of employee, rotator, contractor, and “voluntary” staff from external research communities to accomplish the agency’s work. NSF’s FY2002 1,490 personnel workforce comprises 27 percent non-permanent staff (e.g., temporary, intermittent, VSEE, and IPA), and reflects a mix of program type (45 percent Science and Engineering, 32 percent Business Operations, 20 percent Program Support, and 3 percent Other.) NSF also uses approximately 200 on-site contractors who perform commercial administrative duties. In addition, NSF draws extensively upon members from external science and engineering communities who “donate” some portion of their time to assist NSF in conducting merit review of research proposals and in evaluating its programs through Committees of Visitors and Advisory Committees Recommendation: NSF should use baseline workforce supply findings to help inform the development of future target workforce objectives including size, composition, and mix Recommendation: As future workforce requirements are determined, NSF should compare future workforce requirements against current workforce supply to identify key gaps, and to inform the development of strategies (e.g., recruiting, selection, training) to close or mitigate identified gaps Workforce Supply Competency Modeling HR Practices: Foundational Issues HR Practices: Operational Issues

20. NSF is improving the strategic and operational management of its human capital by developing a competency-based HR system that links business strategy to individual performance • Finding:NSF has recognized that the lack of a competency-based HR system inhibits the organization from aligning its business strategy with individual performance. The way that NSF’s employees are selected, trained, developed, assessed, and managed varies across NSF and is dependent upon the individual supervisor(s) to whom NSF employees report. Recommendation: NSF should take a proactive role in driving employee performance and achieving organizational outcomes by developing a competency-based HR system encompassing planning, recruiting, educating, developing, motivating, and transitioning activities Recommendation: In order to support the development of a competency-based HR system, NSF should, through Task 5, identify and define “to-be” competencies that support expected changes to business strategy and processes, assess current proficiency levels, and develop recruiting, training, and development strategies to close or mitigate identified gaps Workforce Supply Competency Modeling HR Practices: Foundational Issues HR Practices: Operational Issues

21. NSF can use “As Is” competencies to improve its ongoing human capital management activities • Finding: 41 job families exist that encompass NSF’s 288 unique position titles among its 1,500 employees and rotator personnel. Incumbent employees identify a total of 33 general competencies and 70 technical competencies that are currently required for successful performance among the job families. Of the 33 general competencies, eight “core” competencies have emerged that apply to all employees. Of the “core” values, six are leadership in nature. While each job families’ competency model is unique, comparisons among models resulted, in some cases, in notable similarities and differences. Recommendation: In Task 5, NSF should evaluate the eight “core” competencies currently identified to assess whether they reflect the organization’s future needs and direction, and identify additional “core” competencies as required Recommendation: While “to-be” competencies are identified and modeled, NSF should use the current “core” competencies and existing competency models in ongoing recruiting, selecting, and training activities Workforce Supply Competency Modeling HR Practices: Foundational Issues HR Practices: Operational Issues

22. NSF should continue to align its business strategies with human capital by clarifying and communicating HC management strategy and responsibilities… • Finding: Historically, NSF has not aligned business strategies with human capital and therefore has struggled to prepare for the impacts on its workforce stemming from changes in business practices, increased automation and growth in workload. This challenge is due to lack of a human capital representative in NSF’s top leadership, absence of a HCMP that sets the strategic direction for human capital, and a perceived need for clarity regarding HRM’s strategic roles and responsibilities and NSF managers’ human capital roles and responsibilities. Recommendation: Ensure that a human capital management expert is identified and included in strategic business and operational decisions Recommendation: Develop a Human Capital Management Plan to provide an actionable “roadmap” and direction for improving human capital management within NSF and within the HRM Division Recommendation: Clarify and communicate HCM responsibilities and develop accountability mechanisms; develop approaches to better recruit and train individuals who will hold HC management positions Workforce Supply NSF Progress Competency Modeling NSF Progress HR Practices: Foundational Issues HCMP Action HR Practices: Operational Issues

23. …and by clarifying HRM’s role in driving human capital improvements • Finding: While HRM currently provides both strategic and operational support to NSF, HRM could provide a more consultative role to help guide management of human capital issues and may need to improve its competencies in order to do so Recommendation: Identify and define HRM’s strategic roles and responsibilities and associated competencies. Consider restructuring HRM to optimize the balance between new strategic roles and ongoing operational responsibilities. Assess needs and identify training opportunities to build critical competencies for HR professionals. Workforce Supply NSF Progress Competency Modeling HR Practices: Foundational Issues HR Practices: Operational Issues

24. NSF should create a workforce plan / blueprint that determines needed workforce size, composition and mix, and that supports future business needs • Finding: Workforce requirement data compiled in support of budget requests does not currently support the development of organization-wide workforce plans. Vacancy data is not aggregated and does not support the identification of immediate staffing needs NSF-wide. Aside from a specific diversity-in-hiring goal and an identified need for 150 additional FTE, NSF has few stated workforce goals. Recommendation: In Task 5, and considering the relative priority of HCMP action strategies, NSF should establish a blueprint or model of NSF’s future workforce that incorporates key workforce elements including workforce size, composition, structure, and mix and that supports expected future business needs Workforce Supply HCMP Action Competency Modeling HR Practices: Foundational Issues HR Practices: Operational Issues

25. NSF’s training and development opportunities can be improved by linking competencies to curricula Workforce Supply • Finding: Employee time spent in training and corresponding budgetary support have consistently grown since FY2000; however, on average, 38% of employees do not participate in training in a given year. NSF’s training and development opportunities, including NSF Academy curricula, are not yet tied to specific positions, job families, or competencies that would allow employees and supervisors to more strategically identify and select appropriate opportunities. While common career advancement routes exist, competency-based career paths that outline positions to which employees may advance are lacking. Recommendation: NSF should use current competency models to assess and identify immediate training and development needs and should revise training course content accordingly Competency Modeling HCMP Action HR Practices: Foundational Issues HR Practices: Operational Issues

26. NSF should identify the requirements for a competency-based or other performance management system • Finding: NSF’s employee performance management system does not clearly distinguish between high and low performers, affecting NSF’s ability to recognize and motivate high performance and to identify and address low performance. While 96% of employees are rated as “Outstanding” or “Very Good” in 2002, supervisors indicate that ratings may not reflect true performance due to unwillingness to become involved in potential employee grievances, among other factors. A majority of general workforce employees receive an annual performance award, compared to roughly half of SES employees Recommendation: In Task 5, and considering the relative priority of HCMP action strategies, NSF should devise the requirements for a competency-based or other performance management system that supports both employee and organizational performance Workforce Supply Competency Modeling HCMP Action HR Practices: Foundational Issues HR Practices: Operational Issues