Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC)

1. Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC) Planning for a Service Program Evaluation - Case Study Public Administration 522 Presented by Ervina Castillo-Newton, John Toadlena and Scott Atole October 16, 2012

2. Background on Service Program Case Study Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Leukemia & Lymphoma Society (LLS) Information Resource Center (IRC) “World’s largest voluntary health organization dedicated to funding blood cancer research, education, and patient services” • Private non-profit service organization • Funded by individual and corporate contributions • 68 chapters: U.S and Canada • Masters level Information specialists responding to 80,000 inquiries on leukemia, lymphoma, myeloma and other blood cancer patients, etc. • Provide information, guidance & support (phone based)

3. Background on Service Program Case Study Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation • Evaluation of Information Resource Center(IRC) • Email – initial contact • Program and academic interest & expertise match • Successful response to RFP • Initial discussions including questions identified in RFP response. Additional details expected in the planning process • RFP: Pilot a patient navigation model for IRC follow up services • Navigate health care systems • Overcome barriers • Evaluation of service enhancement-scheduled phone call follow up to select callers. Need identified by program staff • LLS-service oriented and committed to the evaluation process • Evaluate new programs and ongoing evaluations • Internal funding for this evaluation

4. Background on Service Program Case Study Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation • Initial Planning: • 3 months (email to launch) • Report provided 6 months later • LLS moved to fund a second/larger evaluation on Patient navigation • EPIC model (steps before the pilot study)

5. Assess the Context Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Understanding the Political & Organizational Environment • Pre-project-consulted various sources • Internal/external people • Internet/LLS website • LLS Promotional items/collaterals • RFP • LLS Mission Statement • LLS Organizational structure (background materials/site visit) • Commitment to organizational cause. • Political environment associated with advocacy and organizational commitment • Funding-individual/corporate • LLS operations (private sector-accountability) • How will the evaluation findings be used? Future funding from LLS Board of trustees. Internally competitive • Location of IRD within LLS structure. department

6. Assess the Context Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Understanding the Political & Organizational Environment • LLS values program evaluation • Objective data to support the IRC program enhancement • IRC – immediate program context • 7 full time information specialists • 80,000 phone calls annually • Navigation model – innovative among other national programs • Navigation to identified subgroup (2 specialists) • National organization • 68 chapters – IRC link to local chapters • Need for cancer information (consumer friendly) • Potential for collaboration “community of cancer information service professional”

7. Assess the Context Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Define the Relationship Between Evaluator & Sponsor • Internal/External Evaluator and Funding • LLS External Evaluator/Internal funding • Findings were reported to the LLS Board of Trustees • Primary Point of Contact – LLS Vice President or Patient Services & Disease Programs (Includes IRC) • Advantages/Disadvantages-resistance to int/ext pressures, produce positive findings, objectivity, credibility, timelines(learning/building relationship), potential biases • Clarify roles and the relationship early • Evaluator and Sponsor roles • Funding arrangements • LLS –Participatory Relationship in planning and implementation • LLS – 2 Managers • IRC – 1 Supervisor (Ongoing Liaison) and 1 staff member • Ongoing Education – Responsibilities • LLS Staff – Provide feedback • Evaluator-Drafts, Technical assistance (research process), Assess stakeholder knowledge • Evaluator skill • Relevant experience, experience and interest • Qualitative vs. Quantitative

8. Assess the context Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Determine the Level of the Evaluation • Leukemia & Lymphoma Society (LLS) is local • Information Resource Center (IRC) is national. Seek to create the connection with local chapters. • LLS Target population • Newly Diagnosed • Those seeking information on clinical trials • Patients residing in Wyoming • Reasoning for Target Population selection (Where will this enhancement be most helpful?): • Limited LLS resources to offer the program to all callers • Navigation approach –reduction in health disparities to vulnerable populations • Wyoming – no local chapters • Outcome vs. Process Evaluation • Primary focus on outcomes • IRC program enhancement – include and document the process

9. Gather Reconnaissance Scott Atole UNM-Main PADM 522 Planning for a Service Program Evaluation Specify Evaluation Use • LLS-findings will be used for program sustainability and continued and amount of future funding. • Evaluations used to make decisions. • Important to understand the implication of negative findings • Will this program be duplicated? Will the literature/results be made available nationally? Validate Perspectives • Communications with nearly every level of LLS/IRC • Welcome evaluation • Supportive of the project • Important to understand that it is a program evaluation

10. Engage Stakeholders John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation • Potential stakeholders for a service program include: • Two main reasons why the evaluator must engage stakeholders in evaluation planning. • Identify and Invite Stakeholders. • Establish rapport. • Participants involved in the evaluation planning. • “Sponsor” role. • Define Stakeholder Roles and Structure for Their Input. • Importance of engaging the few stakeholder groups. • Survey design.

11. Engage Stakeholders John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation • Establish Group Processes for Ongoing Stakeholder Involvement. • The communication process. • Methods used. • Evaluation plan timeline. • Preliminary reports.

12. Describe the Program John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation • Establishment of the stakeholder. • Become immersed in the nature of the program to be evaluated. • “Framework for Program Evaluation in Public Health”. • Understanding the IRC. • Conceptualize Program Theory or Rationale. • Usefulness of logic models and conceptual models. • Evaluation Planning Matrix (EPM). (Holden and Zimmerman, pg.78-79) • Lists the evaluation questions considered. • Short-term, Intermediate, or Long-term outcomes and identifies relevant data sources corresponding to each question. • Outcome surveys and questions.

13. Focus the Evaluation John Toadlena UNM-Gallup PADM 522 Planning for a Service Program Evaluation • Focusing the evaluation. • The planning process. • Study design team. • Preliminary evaluation questions. • Theoretical frameworks. • Ensure Feasibility. • The focus on feasibility. • Control or comparison group. • Assess Potential Data Collection Burden. • Data collection burden an issue on two levels. • How much surveying could the participants tolerate. • What kinds of data collection operation could IRC take on. • Sample size. • Draw conclusions from the results.

14. Lessons Learned Ervina Castillo-Newton UNM-WS PADM 522 Planning for a Service Program Evaluation • Service Programs are unique • Start with the Mission Statement • Size and scope of the activities • Evaluation • Attitudes • Stakeholders’ knowledge • Stakeholders • Expertise & knowledge • Minimizes resistance • Planning Process • Dissemination • Goals • This case – positive and knowledgeable

15. Conclusion Ervina Castillo-Newton UNM-WS PADM 522 Planning for a Service Program Evaluation Comparison Group • Based on comparison of callers • Standard Service through IRC • Enroll in intervention • Information Specialists • Qualified • Trained • Experienced • Outcomes