2012 EXAMINER TRAINING Introduction to Evaluating a Baldrige Application

1. 2012EXAMINER TRAININGIntroduction to Evaluating a Baldrige Application Presented by The Granite State Quality Council and The Northern New England Alliance for Excellence

2. Agenda • Stages of a Baldrige Evaluation • The Evaluation Process • Before you begin • Key Factors • Key Themes • Evaluating each Item • Process Item Demonstration • Results Item Demonstration • Review of the Assignment

3. What is a Baldrige Application? • A structured response to the questions posed by the Criteria • May be for internal use or for state/national award • Contains: • Eligibility forms • Glossary of Terms • Organizational Profile (5 pages) • Categories 1-7 (50 pages)

4. The Role of a Baldrige Examiner • Provide objective feedback on Strengths and Opportunities for Improvement (OFIs) • First pass is based solely on the Application • A Site Visit may follow to verify and clarify feedback from Application

5. The Goal of an Application Evaluation • Unbiased feedback on Strengths and Opportunities For Improvement (OFIs) • Baldrige feedback is • Non-prescriptive • Non-judgmental • Grounded in the Criteria & Core Values • Weighted on the Organization’s Profile • Feedback may be used to determine award level

6. The Baldrige Application Evaluation Process - Stages Goal: Value Added Feedback To the Applicant Site Visit Comments Consensus Comments Stage 3 Stage 2 Independent Observations Stage 1

7. Application Evaluation Process

10. The Application Evaluation Process – Independent Observations Evaluate Each Criteria Item Check for Conflict of Interest Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook

11. Contents of a Baldrige Application Case Study: Tillingate Living • Eligibility Information • Pages i – x • Additional Information • Org Charts, Glossary/Abbreviations (pages xi – xv) • Responses to Baldrige Criteria Questions • Preface: Organizational Profile (pages xvi – xx) • Categories 1 – 7 (pages 1-50)

12. The Application Evaluation Process Evaluate Each Criteria Item Check for Conflict of Interest Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook

13. What you will use to Evaluate the Application Application Criteria On-line Worksheets No independent research or contact with Applicant

14. On-line Worksheets What you will use to Evaluate the Application

15. The Narrative Application Evaluation Process Evaluate Each Criteria Item Check for Conflict of Interest Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook

16. Key Factors • What are they? • Significant attributes of an organization • that influence the way the organization operates • Where do they come from? • The Organizational Profile • Criteria Items • Eligibility Forms • Guidelines • Group under the five Areas to Address in the Organizational Profile • Write in phrases rather than complete sentences • Numbered list (for use in rest of analysis) • Revise throughout the evaluation process • Limit to 1–3 pages in length

17. Org Profile Criteria Items Key Factors How to Identify Key Factors Answers to “What” questions from the Organizational Profile No Evaluation of Key Factors

18. Key Factors: Mission, vision, and values Employee/staff profile Customer and market segments and customer requirements Competitive position and critical success factors Strategic challenges Governance structure Not Key Factors: Strategic planning approach Complaint management process Rewards and Recognition System Methods to promote and ensure ethical behavior Key Factors: Example Answers “What” - facts or attributes that affect the way the organization operates Answers “How” - processes that the applicant uses in managing its work

19. Criteria Page 4 Case Study page xvi

21. The Key Factors Worksheet Key Factors: a bulleted list of Applicant responses to questions in the Organizational Profile. Key Factors Worksheet P.1a Organizational Environment • For-profit, privately held organization/Care Model - For-profit, privately held organization providing assisted living and skilled nursing care in 23 facilities located in Pennsylvania, Kentucky, Tennessee, and Virginia whose facilities are known for a sense of societal responsibility. Care model is centered on advancing seniors’ independence and quality of life utilizing the Aging Actively Consortium's (AAC’s) dimensions of wellness: emotional, intellectual/cognitive, physical, spiritual, social, professional/vocational, and environmental. • Mission, Vision, Values - Mission (Figure P.1-2): Provide ageless care and timeless living to individuals in a homelike environment that supports their lifestyles and need for care with dignity and respectVision (Figure P.1-2): Be among the top 10% of skilled nursing facilities (SNFs) and assisted living facilities (ALFs) and be a top choice for careValues(Figure P.1-2): Agility, Patience, Empathy, and Excellence (APEX) • Core Competencies (Figure P.1-2) - •Designing, innovating, and managing facilities to support various lifestyles and deliver excellent clinical outcomes•Developing clinical and service competencies for a caring and exceptional staff•Designing and delivering rehabilitation services to support residents’ activities of daily living•Creating an educational environment to support a sense of mastery for residents

22. The Narrative Application Evaluation Process Evaluate Each Criteria Item Check for Conflict of Interest Assemble Materials and Read Application Draft Initial Key Factors Finalize Key Themes Observations Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook

23. What are Key Themes?A Summary of Key Points found in the Evaluation Key Theme: A high-level strength or opportunity for improvement that • Is common (cross-cutting) to more than one Item/Category and/or • Is significant in terms of the applicant’s key factors and/or • Addresses a Core Value of the Criteria • Must be traceable to comments found in the Item Worksheets As a whole Key Themes: • Summarize how well an applicant has addressed the Criteria requirements, Key Factors, and Core Values • Serve as an Executive Summary Importance: • Key Themes are usually the first and sometimes the only part of the feedback report read by senior leaders • Used by examiners to focus site visit • Used by judges to get a “big picture” of the Applicant

24. Organization of the Key Themes Worksheet A. What are the most important strengths or outstanding practices found in Process Items? B. What are the most significant opportunities, concerns, or vulnerabilities identified in Process Items? C. what are the most significant strengths found in its response to Results Items? D. What are the most significant opportunities, vulnerabilities, and/or gaps found in the Results Items?

25. Steps in Developing Independent Key Themes 1. Read application & capture initial impressions. 2. Evaluate application at the Item-level. • Key Themes developed at Consensus • Refined after Site Visit

26. Agenda Review • Stages of a Baldrige Evaluation • The Evaluation Process • Before you begin • Key Factors • Key Themes • Evaluating each Item • Process Item Demonstration • Results Item Demonstration • Review of the Assignment

27. Baldrige National Quality Program Developing Independent Review Observations Process Item Demo

28. The Application Evaluation Process Evaluate Each Criteria Item Check for Conflict of Interest Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook

29. Evaluate Each Criteria Item Read Criteria Requirements Determine 4-6 Key Factors Read Applicant's Response Develop ~6 Strengths and OFI’s and Supporting Evidence Write 2 Feedback Ready Comments Determine Score

30. Step 1: Read Criteria Requirements

31. Item 5.1 Requirements • Support Info: • Notes • Category & Item Description • Page 42 Page 18 in Criteria Booklet

32. Step 2: Key Factors

33. Item 5.1 Independent Review Worksheet – Key Factors From Key Factors Worksheet – Handout What 4-6 Key Factors do we most expect to be addressed by this Item? (or – which Key Factors do you most want to keep in mind as you evaluate this Item) 4-6 Key Factors are requested at the top of every Item Worksheet All Key Factors Item Requirements Item Key Factors

34. Item 5.1 Independent Review Worksheet – Key Factors 4-6 Key Factors are requested at the top of every Item Worksheet

35. Step 3: Read Application Item Response Categories 1-6 Demonstrate Processes “HOW?”

36. Read through the Item: Example - Item 5.1a(1) What does the Criteria ask? How does the Applicant respond? Case Study: Page 22 & 23

37. Page 62 in Criteria Book Definition – PROCESS • Linked activities with the purpose of producing a product or service for a customer (user) within or outside the organization • Involves combinations of people, machines, tools, techniques, materials, and improvements in a defined series of steps • Service situations - used more generally • May spell out what must be done including a preferred or expected sequence • Knowledge work (e.g., strategic planning, research, development, and analysis) • does not always imply formal sequences of steps • implies understandings regarding competent performance, such as timing, options to be included, evaluation, and reporting.

38. Analyzing a Process Item The purpose of Process Items is to permit diagnosis of an organization’s most important processes — the ones that contribute most to organizational performance improvement and contribute to key outcomes or performance results. KEY Processes

39. Read through the Item: Example - Item 5.1a(1) What Processes can you identify in 5.1a(1)? What does the Criteria ask? How does the Applicant respond? Case Study: Page 22 & 23

40. Step 4: Develop Around 6 Total Strengths and OFI’s and Supporting Evidence

41. Developing Strengths and OFIs • Strengths • Practices/processes or results to be reinforced & build upon • Opportunity for Improvement • Potential gaps, blind spots & vulnerabilities • Categories 1-6 reflect Processes • Category 7 reflects Results

42. First Pass: Focus on identifying… Main Idea Supporting Evidence A-D-L-I Strengths Relationship with Key Factors Relationship with Criteria

43. Gaps and OFIs

44. From our Example - Item 5.1a(1)

45. Example OFIs from - Item 5.1

46. Evaluating a Process Item • Processes include • Approach (A) • Deployment (D) • Learning (L) • Integration (I) • A-D-L-I is a tool to help in application evaluation and scoring • A-D-L-I is the P-D-S-A of High Performing Organizations

47. Approach • The methods and processes used by an organization to accomplish its work • HOW an organization addresses the item requirements • Approach considers… • Appropriateness for organization • Effectiveness • Alignment with the organization’s needs • Appropriate review and improvement • Approaches are systematic if they • Are well ordered • Are repeatable • Use data and information so learning is possible • Build in the opportunity for evaluation, improvement, and sharing thereby permitting a gain in maturity

48. Deployment • Refers to the EXTENT to which an organization’s approach is applied • The breadth and depth of the approach to relevant work units throughout the organization • Implementation on steroids… Is the approach deployed from the President to the janitor?

49. Deployment Deployment is process specific. Appropriate deployment varies from item to item Examples: • Category 1 – Leadership Processes • Deployed to workforce segments, stakeholders, other key processes • Category 3 – Customer Processes • Deployed to customer segments, work force segments • Category 5 – Workforce Processes • Deployed to workforce segments, locations, departments

50. Learning:Organizational and Personal • New knowledge or skills acquired through evaluation, study, experience, and innovation. • Criteria looks for both personal and organizational learning • There are three dimensions of organizational learning: • Continuous improvement of existing processes • Innovation leading to breakthrough changes • Knowledge sharing of such improvements and innovations