1 / 109

2004 Public Health Training and Information Network PHTIN Series

Site Sign-in Sheet . Please mail or fax your site's sign-in sheet to:Linda WhiteNC Office of Public Health Preparedness and ResponseCooper Building1902 Mail Service CenterRaleigh, NC 27699FAX: (919) 715 - 2246. Outbreak Investigation Methods. From Mystery to Mastery. . . 2004 PHTIN Training Development Team.

betty_james
Télécharger la présentation

2004 Public Health Training and Information Network PHTIN Series

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. 2004 Public Health Training and Information Network (PHTIN) Series

    2. Site Sign-in Sheet Please mail or fax your sites sign-in sheet to: Linda White NC Office of Public Health Preparedness and Response Cooper Building 1902 Mail Service Center Raleigh, NC 27699 FAX: (919) 715 - 2246

    3. Outbreak Investigation Methods From Mystery to Mastery

    4.

    5. 2004 PHTIN Training Development Team Pia MacDonald, PhD, MPH - Director, NCCPHP Jennifer Horney, MPH - Director, Training and Education, NCCPHP Anjum Hajat, MPH Epidemiologist, NCCPHP Penny Padgett, PhD, MPH Epidemiologist Amy Nelson, PhD - Consultant Sarah Pfau, MPH - Consultant Amy Sayle, PhD, MPH - Consultant Michelle Torok, MPH - Doctoral Candidate Drew Voetsch, MPH - Doctoral Candidate Aaron Wendelboe, MSPH - Doctoral student

    6. Future PHTIN Sessions October 12th. . . . . . . . . Analyzing Data December 14th. . . . . . . Risk Communication Each session will be on a Tuesday from 10:00 a.m. - 12:00 p.m. (with time for discussion) There are only two remaining sessions in the 2004 PHTIN series on outbreak investigation methods. Those are: Analyzing Data and Risk CommunicationThere are only two remaining sessions in the 2004 PHTIN series on outbreak investigation methods. Those are: Analyzing Data and Risk Communication

    7. Session I VI Slides After the airing of each session, NCCPHP will post PHTIN Outbreak Investigation Methods series slides on the following two web sites: NCCPHP Training web site: http://www.sph.unc.edu/nccphp/phtin/ North Carolina Division of Public Health, Office of Public Health Preparedness and Response http://www.epi.state.nc.us/epi/phpr/

    8. Session IV Designing Questionnaires

    9. Todays Presenters Sarah Pfau, MPH Consultant NC Center for Public Health Preparedness Kay Sanford, MSPH Head, Injury and Violence Prevention Branch, Division of Public Health,NC Department of Health and Human Services,

    10. Designing Questionnaires Learning Objectives Upon completion of this session, you will: Understand the role of questionnaire design in an outbreak investigation Know how to develop a hypothesis generating questionnaire Recognize key planning strategies for successful questionnaire design Upon completion of this session, you will: Upon completion of this session, you will:

    11. Designing Questionnaires Learning Objectives Recognize key characteristics of well-designed questions Recognize three broad question types and when to use them Understand what different question types measure, and the type of data (quantitative versus qualitative) they yield

    12. Designing Questionnaires Learning Objectives Know how to format questionnaires for interviewer administered or self-administered settings Recognize the impact of questionnaire design on View design and data analysis in Epi Info software

    13. Designing Questionnaires

    14. Basic Steps of an Outbreak Investigation Verify the diagnosis and confirm the outbreak Define a case and conduct case finding Tabulate and orient data: time, place, person Take immediate control measures Formulate and test hypothesis Plan and execute additional studies Implement and evaluate control measures Communicate findings Before beginning this session, lets review the basic steps of an outbreak investigation. Remember that in reality, some of these steps may occur simultaneously or in a different order. First, we must verify the diagnosis and confirm that there is actually an outbreak occurring. Next we need to create a preliminary case definition and conduct active case finding. Once we have information from some cases we need to compile and review it. Then we implement preliminary control measures. After that we formulate and test a hypothesis, and plan and execute additional studies based on the preliminary results. Finally, we implement and evaluate control measures and communicate our findings. In sessions 1 and 2, we addressed steps 1 through 7 on this list. In Session III, we re-visited item 5. Today we will re-visit numbers 3 and 5 above. The formulation and testing of hypotheses are actually two distinct steps. We formulate a hypothesis based on the clinical, laboratory and epidemiological data that are available to us. Hypothesis generating interviews tend to be loosely structured information gathering interviews with persons who are ill. In the hypothesis testing phase of the outbreak investigation, we conduct structured interviews with predefined questionnaires talking to both people who are ill and those who are not ill. This PHTIN session will focus on questionnaires primarily for hypothesis generating during an outbreak investigation. Before beginning this session, lets review the basic steps of an outbreak investigation. Remember that in reality, some of these steps may occur simultaneously or in a different order. First, we must verify the diagnosis and confirm that there is actually an outbreak occurring. Next we need to create a preliminary case definition and conduct active case finding. Once we have information from some cases we need to compile and review it. Then we implement preliminary control measures. After that we formulate and test a hypothesis, and plan and execute additional studies based on the preliminary results. Finally, we implement and evaluate control measures and communicate our findings. In sessions 1 and 2, we addressed steps 1 through 7 on this list. In Session III, we re-visited item 5. Today we will re-visit numbers 3 and 5 above. The formulation and testing of hypotheses are actually two distinct steps. We formulate a hypothesis based on the clinical, laboratory and epidemiological data that are available to us. Hypothesis generating interviews tend to be loosely structured information gathering interviews with persons who are ill. In the hypothesis testing phase of the outbreak investigation, we conduct structured interviews with predefined questionnaires talking to both people who are ill and those who are not ill. This PHTIN session will focus on questionnaires primarily for hypothesis generating during an outbreak investigation.

    15. Designing Questionnaires I want to begin by quickly reviewing a list of sub-topics that we will cover during todays session. I want to begin by quickly reviewing a list of sub-topics that we will cover during todays session.

    16. Session Overview The importance of questionnaire design Hypothesis generating versus hypothesis testing questionnaires Preparing for questionnaire design Question design Question type Questionnaire format In addition to these sub-topics, you will have an opportunity to participate in one activity that you can complete at your sites. The activity will challenge you to critique sample questions for sound design principles. And remember that in addition to discussion about the theory and methods behind sound questionnaire design, you will hear from Kay Sanford, Head of the Injury and Violence Prevention Branch of the North Carolina Division of Public Health. Ms. Sanford will tell you about her practical, field-based experience with the development of the 2003 Hurricane Isabel Rapid Needs Assessment survey instrument. If you joined us for the previous PHTIN session, Interviewing Techniques, you will recognize that this is the same survey instrument that Martha Salyers discussed implementing for the door-to-door, face-to-face Rapid Needs Assessment interviews. So lets now begin the session with a consideration of the importance of questionnaire design. . . In addition to these sub-topics, you will have an opportunity to participate in one activity that you can complete at your sites. The activity will challenge you to critique sample questions for sound design principles. And remember that in addition to discussion about the theory and methods behind sound questionnaire design, you will hear from Kay Sanford, Head of the Injury and Violence Prevention Branch of the North Carolina Division of Public Health. Ms. Sanford will tell you about her practical, field-based experience with the development of the 2003 Hurricane Isabel Rapid Needs Assessment survey instrument. If you joined us for the previous PHTIN session, Interviewing Techniques, you will recognize that this is the same survey instrument that Martha Salyers discussed implementing for the door-to-door, face-to-face Rapid Needs Assessment interviews. So lets now begin the session with a consideration of the importance of questionnaire design. . .

    17. Why is Questionnaire Design Important? The quality of the data will be no better than the most error-prone feature of the survey design. - Fowler, F.J. (1993). Survey Research Methods: Second Edition. Sage Publications: Newbury Park. Before we talk about question design, we should address the broader topic of questionnaire design. Why is questionnaire design important? Well, as this slide points out, the quality of your data will be no better than the most error-prone feature of the survey design. Questionnaire design is a skill to be learned. You can and will improve upon your skills each time you implement a survey. Some of you may have experience or lessons learned to date by which you discovered that you may have needed to include different or more questions, or that you should have collected information in greater detail than you did. After todays session, you should be able to recognize the characteristics of sound question and questionnaire design that help you avoid compromising data quality. Before we talk about question design, we should address the broader topic of questionnaire design. Why is questionnaire design important? Well, as this slide points out, the quality of your data will be no better than the most error-prone feature of the survey design. Questionnaire design is a skill to be learned. You can and will improve upon your skills each time you implement a survey. Some of you may have experience or lessons learned to date by which you discovered that you may have needed to include different or more questions, or that you should have collected information in greater detail than you did. After todays session, you should be able to recognize the characteristics of sound question and questionnaire design that help you avoid compromising data quality.

    18. Why is Questionnaire Design Important? With an understanding of good questionnaire design principles, you will ask only about what you need to meet your research objectives. It can be tempting to ask a variety of questions that may yield interesting (and even relevant) information. However, if questions do not address the research objectives, you should not include them. Use an existing hypothesis, or a well-defined case definition, or some other guiding factor in the investigation process to help you identify research objectives on which to base survey questions. I will talk more specifically about preparing for questionnaire design in a few minutes. It can be tempting to ask a variety of questions that may yield interesting (and even relevant) information. However, if questions do not address the research objectives, you should not include them. Use an existing hypothesis, or a well-defined case definition, or some other guiding factor in the investigation process to help you identify research objectives on which to base survey questions. I will talk more specifically about preparing for questionnaire design in a few minutes.

    19. Ask Only About What You Need. . . Example: You ask respondents to list all medications that they are taking (difficult in terms of respondent recall / accuracy), when you really only need to know if they are taking antibiotics. Here is an example of asking for more detail than you need to meet your research objectives: [READ SLIDE]. Here is an example of asking for more detail than you need to meet your research objectives: [READ SLIDE].

    20. Why is Questionnaire Design Important? Question type and response option formatting impact how you can design an on-screen data entry form and / or analyze variables in your software program Later in todays session, I will discuss specific examples of the interrelatedness of question type and response option formatting and View design and data analysis in Epi Info software. We have included this sub-topic in the session because since fall 2003, the North Carolina Center for Public Health Preparedness has been conducting state-wide trainings on the use of Epi Info software for outbreak investigations. As with the 2004 Public Health Training and Information Network series on outbreak investigation methods, these trainings have been sponsored by the Division of Public Healths Office of Public Health Preparedness and Response. Later in todays session, I will discuss specific examples of the interrelatedness of question type and response option formatting and View design and data analysis in Epi Info software. We have included this sub-topic in the session because since fall 2003, the North Carolina Center for Public Health Preparedness has been conducting state-wide trainings on the use of Epi Info software for outbreak investigations. As with the 2004 Public Health Training and Information Network series on outbreak investigation methods, these trainings have been sponsored by the Division of Public Healths Office of Public Health Preparedness and Response.

    21. Why is Questionnaire Design Important? Survey answers are not of interest intrinsically; rather, the answers are important because of their relationship to what they are supposed to help you measure. In direct relation to the concept of asking only about what you need to know, 1. A survey instrument and each question within it should be thoughtfully and precisely constructed. Make each question count. Well developed questions are key to the success of collecting data that will allow you to either hypothesize or measure a relationship between exposure and outcome in an outbreak investigation. [A hypothesis generating questionnaire will yield data that allow you to form a hypothesis. And a follow up, hypothesis testing questionnaire will yield data that allow you to measure a relationship between exposure and outcome]. In direct relation to the concept of asking only about what you need to know, 1. A survey instrument and each question within it should be thoughtfully and precisely constructed. Make each question count. Well developed questions are key to the success of collecting data that will allow you to either hypothesize or measure a relationship between exposure and outcome in an outbreak investigation. [A hypothesis generating questionnaire will yield data that allow you to form a hypothesis. And a follow up, hypothesis testing questionnaire will yield data that allow you to measure a relationship between exposure and outcome].

    22. Hypothesis Generating Questionnaires Questionnaire design is a skill that is taught under the umbrella of social science research. And many of the techniques or methods carry over into an application as specific as a public health outbreak investigation. However, in the context of this Public Health Training and Information Network series on Outbreak Investigation Methods, we want to convey the distinction between Hypothesis Generating and Hypothesis Testing questionnaires. And even more specifically, I am going to provide you with examples of questions that should be included on a standardized hypothesis generating questionnaire for a food borne illness outbreak investigation. In an outbreak investigation, the first type of questionnaire that you will implement will be one that allows you to generate a hypothesis, or clearly defined research question that you can test with further research (e.g., an analytic study like a case-control study). So lets talk about this type of questionnaire first. Questionnaire design is a skill that is taught under the umbrella of social science research. And many of the techniques or methods carry over into an application as specific as a public health outbreak investigation. However, in the context of this Public Health Training and Information Network series on Outbreak Investigation Methods, we want to convey the distinction between Hypothesis Generating and Hypothesis Testing questionnaires. And even more specifically, I am going to provide you with examples of questions that should be included on a standardized hypothesis generating questionnaire for a food borne illness outbreak investigation. In an outbreak investigation, the first type of questionnaire that you will implement will be one that allows you to generate a hypothesis, or clearly defined research question that you can test with further research (e.g., an analytic study like a case-control study). So lets talk about this type of questionnaire first.

    23. Which Questionnaire Type Should You Use? Hypothesis Generating? Hypothesis Testing? The goal of a hypothesis generating questionnaire is to explore all potential sources of infection, but with a small number of cases rule of thumb maybe 10 cases. And cases only no comparison group is involved during hypothesis generating data collection. In contrast, the primary purpose of a hypothesis testing questionnaire is to literally test a research hypothesis that has been formulated based on data collected via a hypothesis generating questionnaire. In order to carry out an analytic study, one needs to include both cases and controls in the survey sample population. You will use a hypothesis generating questionnaire to conduct exploratory data collection about possible sources of the outbreak under investigation. Keep in mind that the questions included in the questionnaire should not only represent known food items or exposures that have been linked to the pathogen [assuming that the pathogen has been identified]; the questions should also include any potential new exposures within the context of the outbreak. Ideally, the data collected will allow you the investigator to narrow down the list of potential sources of infection. Furthermore, the information collected might help you narrow down demographic stratum of cases (e.g., they are all toddlers versus elderly; they all live in the same county; or they are all the same gender). -------------------------------------------- The primary focus of todays instruction will be the development of hypothesis generating questionnaires. However, the principles for sound question design and questionnaire formatting can (and should be!) applied to either hypothesis generating or hypothesis testing questionnaires. The goal of a hypothesis generating questionnaire is to explore all potential sources of infection, but with a small number of cases rule of thumb maybe 10 cases. And cases only no comparison group is involved during hypothesis generating data collection. In contrast, the primary purpose of a hypothesis testing questionnaire is to literally test a research hypothesis that has been formulated based on data collected via a hypothesis generating questionnaire. In order to carry out an analytic study, one needs to include both cases and controls in the survey sample population. You will use a hypothesis generating questionnaire to conduct exploratory data collection about possible sources of the outbreak under investigation. Keep in mind that the questions included in the questionnaire should not only represent known food items or exposures that have been linked to the pathogen [assuming that the pathogen has been identified]; the questions should also include any potential new exposures within the context of the outbreak. Ideally, the data collected will allow you the investigator to narrow down the list of potential sources of infection. Furthermore, the information collected might help you narrow down demographic stratum of cases (e.g., they are all toddlers versus elderly; they all live in the same county; or they are all the same gender). -------------------------------------------- The primary focus of todays instruction will be the development of hypothesis generating questionnaires. However, the principles for sound question design and questionnaire formatting can (and should be!) applied to either hypothesis generating or hypothesis testing questionnaires.

    24. Lets look at some key differences between hypothesis generating and hypothesis testing questionnaires. [READ THROUGH TABLE] Because the hypothesis generating questionnaire is so exploratory by nature, it should include both closed and open-ended questions. However, there are two challenges that come with collecting qualitative responses with open-ended questions: 1. Its best to use the same interviewer for the hypothesis generating interviews so things said in passing by multiple respondents do not go unnoticed. 2. Entering, coding, and analyzing data are not as straightforward as with quantitative data (e.g., data collected via questions with categorical response options). In todays session you will learn when and how to use closed- and open-ended questions. Keep in mind that even if you use an existing, standardized hypothesis generating questionnaire, it may actually be preferable in some instances for interviewers to diverge from the questionnaire structure if they think that further exploring a particular response or comment might benefit the outbreak investigation process. Lets look at some key differences between hypothesis generating and hypothesis testing questionnaires. [READ THROUGH TABLE] Because the hypothesis generating questionnaire is so exploratory by nature, it should include both closed and open-ended questions. However, there are two challenges that come with collecting qualitative responses with open-ended questions: 1. Its best to use the same interviewer for the hypothesis generating interviews so things said in passing by multiple respondents do not go unnoticed. 2. Entering, coding, and analyzing data are not as straightforward as with quantitative data (e.g., data collected via questions with categorical response options). In todays session you will learn when and how to use closed- and open-ended questions. Keep in mind that even if you use an existing, standardized hypothesis generating questionnaire, it may actually be preferable in some instances for interviewers to diverge from the questionnaire structure if they think that further exploring a particular response or comment might benefit the outbreak investigation process.

    25. Hypothesis Generating Questionnaire Include questions about: Demographics Clinical details of the illness Health care provider visits Water exposure Exposure to other ill persons Exposure to children in day care Exposure to a farm or farm animals Travel outside of the immediate area Demographics include: name, age, gender, occupation, address, phone number, date of birth, etc. Clinical details include: date of onset of illness, duration of illness, and severity of symptoms Health care provider visits: have the cases already been seen at a physicians office or hospital for their symptoms? If yes, are there any laboratory results to review? Water exposure: this refers to both drinking and recreational (e.g., swimming) water exposure Exposure to other ill persons, children in day care, and farm animals: If you know which pathogen you are investigating, you can restrict the time frame of exposure to the incubation period of the specific pathogen. If you have not yet identified the pathogen, you might want to use one week (e.g., 7 days) as a rule of thumb for an exposure history. Travel: Same rule of thumb here: if incubation period is not know, you can use a 7 day recall / exposure history. You might choose to design your questionnaire so it is divided into sections that aggregate some of the key types of information to be collected. (e.g., you could have three main survey sections title, Demographics Clinical Characteristics and Risk Factors.) Demographics include: name, age, gender, occupation, address, phone number, date of birth, etc. Clinical details include: date of onset of illness, duration of illness, and severity of symptoms Health care provider visits: have the cases already been seen at a physicians office or hospital for their symptoms? If yes, are there any laboratory results to review? Water exposure: this refers to both drinking and recreational (e.g., swimming) water exposure Exposure to other ill persons, children in day care, and farm animals: If you know which pathogen you are investigating, you can restrict the time frame of exposure to the incubation period of the specific pathogen. If you have not yet identified the pathogen, you might want to use one week (e.g., 7 days) as a rule of thumb for an exposure history. Travel: Same rule of thumb here: if incubation period is not know, you can use a 7 day recall / exposure history. You might choose to design your questionnaire so it is divided into sections that aggregate some of the key types of information to be collected. (e.g., you could have three main survey sections title, Demographics Clinical Characteristics and Risk Factors.)

    26. Hypothesis Generating Questionnaire If the pathogen can be spread through food or beverages, include questions about: Food eaten in the home Food eaten in the homes of friends, family Food eaten at any restaurant Dates and times of food consumption and any suspicious observations If the pathogen is known and you know the incubation period for the illness, you can use that incubation period as your frame of reference for respondent recall. Just to review a point that Anjum discussed in the Interviewing Techniques PHTIN session in August: You may find it helpful to facilitate respondent recall by asking a respondent to look at a personal calendar and even collect food receipts for the most accurate data collection. Ms. Sanford will talk later today about the importance of including scripting and explanatory information for interviewers in a questionnaire. This type of prompt for receipts might be one example of interviewer scripting. If the pathogen is known and you know the incubation period for the illness, you can use that incubation period as your frame of reference for respondent recall. Just to review a point that Anjum discussed in the Interviewing Techniques PHTIN session in August: You may find it helpful to facilitate respondent recall by asking a respondent to look at a personal calendar and even collect food receipts for the most accurate data collection. Ms. Sanford will talk later today about the importance of including scripting and explanatory information for interviewers in a questionnaire. This type of prompt for receipts might be one example of interviewer scripting.

    27. Hypothesis Generating Questionnaire Here is an example of a hypothesis generating questionnaire used for a food borne illness outbreak investigation among 76 cases of Hepatitis in neighboring counties in the 1960s. Some of these sample questions are taken from an Epidemic Intelligence Service case study. They represent a larger set of hypothesis generating questions. This slide contains food and beverage consumption questions that were included on the hypothesis generating questionnaire for the Hepatitis outbreak. [READ THROUGH QUESTIONS AND RESPONSE OPTIONS] In following with sound research methods, this questionnaire includes questions about known food items that have been previously linked to Hepatitis infection. However, note that the questionnaire also includes an open-ended question to allow respondents to report additional food products outside of the usual suspects. This is a good questionnaire design element to incorporate into hypothesis generating questionnaires. Here is an example of a hypothesis generating questionnaire used for a food borne illness outbreak investigation among 76 cases of Hepatitis in neighboring counties in the 1960s. Some of these sample questions are taken from an Epidemic Intelligence Service case study. They represent a larger set of hypothesis generating questions. This slide contains food and beverage consumption questions that were included on the hypothesis generating questionnaire for the Hepatitis outbreak. [READ THROUGH QUESTIONS AND RESPONSE OPTIONS] In following with sound research methods, this questionnaire includes questions about known food items that have been previously linked to Hepatitis infection. However, note that the questionnaire also includes an open-ended question to allow respondents to report additional food products outside of the usual suspects. This is a good questionnaire design element to incorporate into hypothesis generating questionnaires.

    28. Hypothesis Generating Questionnaire Another standard component of a hypothesis generating questionnaire during an outbreak investigation will be the travel history of case-patients. Not only will you want to ask whether or not they have traveled; you will also need to determine when and to where. As in this example, keep in mind that it might be necessary to document multiple travel destinations or times since some professionals travel weekly for business purposes. Another standard component of a hypothesis generating questionnaire during an outbreak investigation will be the travel history of case-patients. Not only will you want to ask whether or not they have traveled; you will also need to determine when and to where. As in this example, keep in mind that it might be necessary to document multiple travel destinations or times since some professionals travel weekly for business purposes.

    29. Hypothesis Testing Questionnaires Now lets consider the function and key question attributes of hypothesis testing questionnaires. . . Now lets consider the function and key question attributes of hypothesis testing questionnaires. . .

    30. Hypothesis Testing Questionnaire Include detailed questions about the suspected source of infection. Example: The local bakery is suspected as the source of a Hepatitis outbreak in multiple counties. The hypothesis testing questionnaire is used with both cases and non-cases, and includes only a food history for all possible items on the bakery menu to pinpoint the exact food item that is contaminated. Remember that while a hypothesis generating questionnaire will be exploratory, a hypothesis testing questionnaire will be more targeted and specific. Lets look at an example: [READ SLIDE]. Remember that while a hypothesis generating questionnaire will be exploratory, a hypothesis testing questionnaire will be more targeted and specific. Lets look at an example: [READ SLIDE].

    31. Hypothesis Testing Questionnaire After investigators implemented the hypothesis generating questionnaire including a question about which restaurants cases had visited, they discovered that 63 out of 76 cases (84%) shared eating at the local bakery as a common exposure. During the analytic investigation with cases and non-cases, this hypothesis testing questionnaire was used to pinpoint the contaminated food item in the local bakery. You can see how short and targeted this questionnaire is. By the way, in case you are interested in knowing: the culprit ended up being the glaze. After investigators implemented the hypothesis generating questionnaire including a question about which restaurants cases had visited, they discovered that 63 out of 76 cases (84%) shared eating at the local bakery as a common exposure. During the analytic investigation with cases and non-cases, this hypothesis testing questionnaire was used to pinpoint the contaminated food item in the local bakery. You can see how short and targeted this questionnaire is. By the way, in case you are interested in knowing: the culprit ended up being the glaze.

    32. Hypothesis Generating versus Testing Questions: E. coli 0157:H7 Hypothesis Generating Account for consumption of ground beef, lettuce, alfalfa sprouts, un-pasteurized milk or juice, and swimming in or drinking sewage-contaminated water Hypothesis Testing [beef suspected]: Beef brand, date of purchase, and grocery store or restaurant where beef was purchased or eaten Lets look at one more example of the different levels of detail in information collected via hypothesis generating versus hypothesis testing questionnaires during the same outbreak investigation. This example is for a hypothetical investigation for Escherichia coli [strain 0157:H7] . In the hypothesis generating questionnaire, you would want to include questions that account for possible exposure to sprouts, lettuce, salami, unpasteurized milk and juice, and swimming in or drinking sewage-contaminated water [these are the known sources of infection that have been documented in previous outbreak investigations]. Lets say that upon preliminary information from the hypothesis generating questionnaire, ground beef is suspected as the source of infection. Your hypothesis testing questionnaire would include questions to assist investigators with locating the source of the beef and taking short- and long-term control measures to prevent further infection. [READ BEEF INFORMATION TO BE COLLECTED] Lets look at one more example of the different levels of detail in information collected via hypothesis generating versus hypothesis testing questionnaires during the same outbreak investigation. This example is for a hypothetical investigation for Escherichia coli [strain 0157:H7] . In the hypothesis generating questionnaire, you would want to include questions that account for possible exposure to sprouts, lettuce, salami, unpasteurized milk and juice, and swimming in or drinking sewage-contaminated water [these are the known sources of infection that have been documented in previous outbreak investigations]. Lets say that upon preliminary information from the hypothesis generating questionnaire, ground beef is suspected as the source of infection. Your hypothesis testing questionnaire would include questions to assist investigators with locating the source of the beef and taking short- and long-term control measures to prevent further infection. [READ BEEF INFORMATION TO BE COLLECTED]

    33. Preparing for Questionnaire Design Now that you understand the importance of a methodological approach to questionnaire design and the distinctive functions of hypothesis generating and hypothesis testing questionnaires, lets look at some related items that you should consider before drafting questions for your survey instrument. Now that you understand the importance of a methodological approach to questionnaire design and the distinctive functions of hypothesis generating and hypothesis testing questionnaires, lets look at some related items that you should consider before drafting questions for your survey instrument.

    34. Preparing for Questionnaire Design There is more to questionnaire design than writing questions. Ideally, you should first: Have a clear purpose and research objectives List variables to be measured Have an analysis plan Consider cost and other logistical aspects There is more to developing a questionnaire than just writing questions. Lets consider each of these individually on the next few slides. . . There is more to developing a questionnaire than just writing questions. Lets consider each of these individually on the next few slides. . .

    35. Questionnaire Design Have a clear purpose and research objectives. Is the purpose of your outbreak investigation survey to generate a hypothesis or to test a refined hypothesis? Part of having a clear purpose and research objectives is understanding whether or not the questionnaire should serve the purpose of collecting data for hypothesis generating versus hypothesis testing. You should now understand how to make that distinction. In an outbreak investigation, you will begin with a hypothesis generating questionnaire, and then follow up with a second questionnaire to collect data pertinent to a refined hypothesis. Part of having a clear purpose and research objectives is understanding whether or not the questionnaire should serve the purpose of collecting data for hypothesis generating versus hypothesis testing. You should now understand how to make that distinction. In an outbreak investigation, you will begin with a hypothesis generating questionnaire, and then follow up with a second questionnaire to collect data pertinent to a refined hypothesis.

    36. Questionnaire Design List variables to be measured Whether you are generating or testing a hypothesis, determine your variables of interest before you develop questions. You will avoid asking unnecessary questions or asking for unnecessary details. Be sure that you have a reason for asking each question that you include in your questionnaire, and know what you will do with the data that you collect. If applicable, base your list of variables to be measured on the hypothesis to be tested. If you are designing a hypothesis generating questionnaire, refer to an existing food borne outbreak investigation survey instrument for sample variables. Remember that the key pieces of information to be included are: person, place, time, and risk factor variables. And if you have already identified the pathogen and need to pinpoint the source of infection, you can identify which food item variables to include by conducting some quick, online research on the CDCs web site.Be sure that you have a reason for asking each question that you include in your questionnaire, and know what you will do with the data that you collect. If applicable, base your list of variables to be measured on the hypothesis to be tested. If you are designing a hypothesis generating questionnaire, refer to an existing food borne outbreak investigation survey instrument for sample variables. Remember that the key pieces of information to be included are: person, place, time, and risk factor variables. And if you have already identified the pathogen and need to pinpoint the source of infection, you can identify which food item variables to include by conducting some quick, online research on the CDCs web site.

    37. Questionnaire Design Go to: http://www.cdc.gov Under Health & Safety Topics in the left margin, click on, Diseases & Conditions Select a link to either an alphabetized list of all diseases or diseases by topic [e.g., if you already know that a pathogen is water-borne versus food-borne] If you have a known pathogen, you can include items related to a suspected source of infection as response options on your questionnaire. For example, food versus drinking water versus swimming water versus a blood supply, and even specific food items identified as sources of infection in previous outbreak investigations. If you have a known pathogen, you can include items related to a suspected source of infection as response options on your questionnaire. For example, food versus drinking water versus swimming water versus a blood supply, and even specific food items identified as sources of infection in previous outbreak investigations.

    38. Questionnaire Design Once you click on a link for a specific disease (I chose Listeriosis for this example), you can see that you have immediate access to information to help you identify USUAL SUSPECT food items to include on your hypothesis generating questionnaire. On this screen, you can verify right away that Listeriosis is a food-borne illness (SEE RED UNDERLINE), and you can read about exactly which foods may be contaminated. Those foods then become RESPONSE OPTIONS in your hypothesis generating questionnaire. There are other online and published resources in which you could find similar reference information. However, this is a good place to start. Once you click on a link for a specific disease (I chose Listeriosis for this example), you can see that you have immediate access to information to help you identify USUAL SUSPECT food items to include on your hypothesis generating questionnaire. On this screen, you can verify right away that Listeriosis is a food-borne illness (SEE RED UNDERLINE), and you can read about exactly which foods may be contaminated. Those foods then become RESPONSE OPTIONS in your hypothesis generating questionnaire. There are other online and published resources in which you could find similar reference information. However, this is a good place to start.

    39. Questionnaire Design Have an analysis plan. Guides the question types and response option categories used on the questionnaire Helps assure that the data collection leading up to analysis yields variable coding that your analysis software program can use efficiently. The analysis plan should be based on the research question(s), variables, study design, and sampling method. An analysis plan guides the selection of question types and response option categories when you know: a. Your hypothesis (e.g., what are you testing? What do you need to answer or prove?) b. The variables involved in your analytic study (e.g., exposure and outcome, and confounding) Basically, what food items or environmental exposures do you need to account for with the survey? Which food items or environmental exposures may not be suspected as the cause of illness, but may be related in some way that make them important to consider during data analysis? For example, you suspect exposure to a toxin in a 10-story office building, but you do not know exactly which floor or in which department on a floor that the point source originated. Including questions about occupation and office location within the building might be helpful. c. How you will collect and code information obtained from respondents Whenever possible, pre-code any categorical response options for closed-ended questions particularly to facilitate View design and data entry in Epi Info relational database software! I will illustrate what I mean by pre-coding later in todays session. d. How you will need to analyze the data within your analysis software (for example, is it sufficient to collect DOB versus current age? Will you need to calculate age later?) Will you need to create questions with dichotomous (2-part, e.g., yes versus No or ill versus not ill or positive versus negative) response options for the purpose of later generating 2 x 2 tables for an analytic study to test a hypothesis with data from both cases and controls?The analysis plan should be based on the research question(s), variables, study design, and sampling method. An analysis plan guides the selection of question types and response option categories when you know: a. Your hypothesis (e.g., what are you testing? What do you need to answer or prove?) b. The variables involved in your analytic study (e.g., exposure and outcome, and confounding) Basically, what food items or environmental exposures do you need to account for with the survey? Which food items or environmental exposures may not be suspected as the cause of illness, but may be related in some way that make them important to consider during data analysis? For example, you suspect exposure to a toxin in a 10-story office building, but you do not know exactly which floor or in which department on a floor that the point source originated. Including questions about occupation and office location within the building might be helpful. c. How you will collect and code information obtained from respondents Whenever possible, pre-code any categorical response options for closed-ended questions particularly to facilitate View design and data entry in Epi Info relational database software! I will illustrate what I mean by pre-coding later in todays session. d. How you will need to analyze the data within your analysis software (for example, is it sufficient to collect DOB versus current age? Will you need to calculate age later?) Will you need to create questions with dichotomous (2-part, e.g., yes versus No or ill versus not ill or positive versus negative) response options for the purpose of later generating 2 x 2 tables for an analytic study to test a hypothesis with data from both cases and controls?

    40. Questionnaire Design Consider cost and other logistical aspects What is the survey sample size? What is the geographic distribution of the survey sample? Will questionnaires be interviewer administered or self-administered? What is your staff capacity to work within the parameters of a c above? While you might have the luxury of using a long, exploratory hypothesis generating questionnaire, a hypothesis testing questionnaire may need to be much more succinct. Particularly if the number of cases and controls to be interviewed is large. And, as you learned in the August PHTIN session on Interviewing Techniques, each interview method has different staff capacity and cost implications. Also remember that you need to design your questionnaire so interviewers can be self-sufficient (e.g., not have any questions about the meanings of words in the questionnaire or which questions to ask within a skip pattern). Remember, as you learned in the August PHTIN session on Interviewing Techniques that interviewer training upon the completion of your questionnaire will be paramount to a successful data collection effort. In todays discussion of questionnaire formatting, I will provide examples of text items and formatting that you can include in a questionnaire to facilitate interviewer training and survey implementation. Kay Sanford will also discuss the practical application of these methods in her presentation on the development of the post-Hurricane Isabel Rapid Needs Assessment survey instrument. While you might have the luxury of using a long, exploratory hypothesis generating questionnaire, a hypothesis testing questionnaire may need to be much more succinct. Particularly if the number of cases and controls to be interviewed is large. And, as you learned in the August PHTIN session on Interviewing Techniques, each interview method has different staff capacity and cost implications. Also remember that you need to design your questionnaire so interviewers can be self-sufficient (e.g., not have any questions about the meanings of words in the questionnaire or which questions to ask within a skip pattern). Remember, as you learned in the August PHTIN session on Interviewing Techniques that interviewer training upon the completion of your questionnaire will be paramount to a successful data collection effort. In todays discussion of questionnaire formatting, I will provide examples of text items and formatting that you can include in a questionnaire to facilitate interviewer training and survey implementation. Kay Sanford will also discuss the practical application of these methods in her presentation on the development of the post-Hurricane Isabel Rapid Needs Assessment survey instrument.

    41. 5 minute break I am going to pause for a 5-minute break here. When we resume, I will talk about question design. I am going to pause for a 5-minute break here. When we resume, I will talk about question design.

    42. Question Design Finally after careful planning, you can begin the task of drafting questions for your survey instrument! I stress the term draft here because even though you may feel rushed to release a survey instrument into the field after you develop it, I encourage you to FIRST field test the instrument with colleagues. The most effective way to make sure that field testing gets done is to develop a standardized outbreak investigation questionnaire that can be modified only where applicable For example, you may have interchangeable question sections for food borne versus water borne versus vector born diseases. If you have a standardized questionnaire in advance that you can field test / practice using during field exercises or drills, you will be ahead of the game when a real outbreak occurs!Finally after careful planning, you can begin the task of drafting questions for your survey instrument! I stress the term draft here because even though you may feel rushed to release a survey instrument into the field after you develop it, I encourage you to FIRST field test the instrument with colleagues. The most effective way to make sure that field testing gets done is to develop a standardized outbreak investigation questionnaire that can be modified only where applicable For example, you may have interchangeable question sections for food borne versus water borne versus vector born diseases. If you have a standardized questionnaire in advance that you can field test / practice using during field exercises or drills, you will be ahead of the game when a real outbreak occurs!

    43. Question Design Elements of good question design: Reliability Validity Specificity versus ambiguity Simplicity Only one question asked Mutually exclusive answer choices Refers respondents to specific dates / times for recall When feasible, make sure data can be compared to existing sources of information Today I will discuss these eight elements of good question design: [READ SLIDE]Today I will discuss these eight elements of good question design: [READ SLIDE]

    44. Question Design 1. A question that is designed to be reliable will assure that the words are interpreted the same way in any setting, and that respondents answer the same way in any setting. You may have heard of the research terms, Reliability and Validity before. [READ SLIDE] To assure that your questions are reliable, avoid ambiguous terms or words that might be interpreted differently by different people. A closed-ended question with pre-determined, categorical response options may provide more reliable responses than an open-ended question it leaves less room for misinterpretation of the question stem. You may have heard of the research terms, Reliability and Validity before. [READ SLIDE] To assure that your questions are reliable, avoid ambiguous terms or words that might be interpreted differently by different people. A closed-ended question with pre-determined, categorical response options may provide more reliable responses than an open-ended question it leaves less room for misinterpretation of the question stem.

    45. Reliable Question Design Question: Are you experiencing diarrhea? Interviewer then adds: For the purposes of this survey, we consider diarrhea to be 3 or more loose bowel movements in a 24 hour period. Here is an example in which clarification for a potentially ambiguous term has been provided for an interviewer administered survey. You could place the exact same text in italics below the question stem on a self-administered survey to be sure that the respondent has a clear understanding of the word, diarrhea.Here is an example in which clarification for a potentially ambiguous term has been provided for an interviewer administered survey. You could place the exact same text in italics below the question stem on a self-administered survey to be sure that the respondent has a clear understanding of the word, diarrhea.

    46. Question Design 2. A question that is designed to be valid will always yield information that can be used as a true measure of what you, the researcher, are looking for. READ SLIDE In other words, does the question measure what it is supposed to measure? READ SLIDE In other words, does the question measure what it is supposed to measure?

    47. Valid Question Design Less Useful Which is your source of drinking water at home? Tap water Bottled water Better Which is your source of drinking water at home? Municipal tap water Municipal tap water with additional filtration Well water Commercially bottled water In this example, you will be able to gain a much more accurate measurement of the true water source by using the second set of response options. The differentiation among water sources could mean the difference between identifying a source of infection and not identifying the source of infection!In this example, you will be able to gain a much more accurate measurement of the true water source by using the second set of response options. The differentiation among water sources could mean the difference between identifying a source of infection and not identifying the source of infection!

    48. Question Design 3. Avoid ambiguity in question wording. Less useful When did you have Disease X? Better How old were you when you had Disease X? The first version of this question could yield a variety of different responses, and you might not get the response you actually need! Consider the following possibilities: When I was 10 When I started college In 1995 Right after I moved to North CarolinaThe first version of this question could yield a variety of different responses, and you might not get the response you actually need! Consider the following possibilities: When I was 10 When I started college In 1995 Right after I moved to North Carolina

    49. Question Design 3. Avoid ambiguity in question wording. Less useful Have you been examined by a physician in the past seven days? Better Have you been examined by a physician for these symptoms in the past seven days? Here is one more example of making a question as specific as possible in order to avoid ambiguous interpretation on behalf of the respondent. Here is one more example of making a question as specific as possible in order to avoid ambiguous interpretation on behalf of the respondent.

    50. Question Design 4. Use simple language and keep questions short. Less useful Were you exposed to the fomite at the dinner party? Better Did you use a shared hand towel at the dinner party?

    51. Question Design 5. Ask only one question. Two questions in one Did you eat mashed potatoes and giblet gravy? One question at a time Did you eat mashed potatoes? Yes No If Yes, did you eat them: a. Plain or with butter b. With giblet gravy Particularly in an outbreak investigation in which you want to be able to pinpoint a source of infection as specifically as possible (e.g., bean sprouts versus just the salad bar), be sure that you separate each food item even when things are traditionally served together. A similar example might be coffee with cream. - You might break the beverage down into two response options related to coffee: a) coffee black b) coffee with creamParticularly in an outbreak investigation in which you want to be able to pinpoint a source of infection as specifically as possible (e.g., bean sprouts versus just the salad bar), be sure that you separate each food item even when things are traditionally served together. A similar example might be coffee with cream. - You might break the beverage down into two response options related to coffee: a) coffee black b) coffee with cream

    52. Question Design 6. For closed-ended questions, make sure that response options are mutually exclusive. Not useful What is your age? 18 years old or younger 18 years old or older Useful What is your age? 17 years old or younger 18 years old or older This seems like a very simple example, but always be sure to double check your response option categories for any overlap. Not only will they yield inaccurate data they will have implications for entering and analyzing data within a software program. This seems like a very simple example, but always be sure to double check your response option categories for any overlap. Not only will they yield inaccurate data they will have implications for entering and analyzing data within a software program.

    53. Question Design 7. Use specific date / time references to improve respondent recall. Less useful Have you been swimming in a public pool recently? Better Did you swim in a public pool between Monday, June 2nd and Monday, June 9th, 2004? Here, the better survey question provides a specific range of dates to which the respondent can refer, potentially making recall easier. Here, the better survey question provides a specific range of dates to which the respondent can refer, potentially making recall easier.

    54. Question Design When feasible, make sure data can be compared to existing sources of information. Example: Case-patient age Less than 2 years 2 5 years The U.S. Census reports population data for age groups less than 1, 1 2, 3, 4, 5, etc. This example might pertain more to ongoing surveillance databases than during an outbreak investigation. For example, you might initially need to look at every value and corresponding frequency for age in a line listing to look for trends in an outbreak. If you administer surveillance surveys and need to use those for reporting purposes to state or federal offices, you might want to parallel the U.S. Census data category cut-off points for a variety of demographic or health status indicators. Similarly, if you want to conduct a needs assessment and have local data that you then want to compare to national trends, you will need to use response options that allow you to compare information to existing data. NOTE: One nice thing about data analysis software is that it can be forgiving on this front. For example, Even if you initially collect age as a continuous variable, you can recode new categories that match the state or federal ones, thereby allowing you to quickly compare data anyway. However, if you can research common data categories ahead of time, it is most efficient to design your questionnaire with pre-coded response options that parallel existing data sources. This example might pertain more to ongoing surveillance databases than during an outbreak investigation. For example, you might initially need to look at every value and corresponding frequency for age in a line listing to look for trends in an outbreak. If you administer surveillance surveys and need to use those for reporting purposes to state or federal offices, you might want to parallel the U.S. Census data category cut-off points for a variety of demographic or health status indicators. Similarly, if you want to conduct a needs assessment and have local data that you then want to compare to national trends, you will need to use response options that allow you to compare information to existing data. NOTE: One nice thing about data analysis software is that it can be forgiving on this front. For example, Even if you initially collect age as a continuous variable, you can recode new categories that match the state or federal ones, thereby allowing you to quickly compare data anyway. However, if you can research common data categories ahead of time, it is most efficient to design your questionnaire with pre-coded response options that parallel existing data sources.

    55. Question Design: General Guidelines Design clear, specific questions. Use simple language and keep questions short. Ask only one question! So lets just quickly review the question design concepts that I just illustrated. . . So lets just quickly review the question design concepts that I just illustrated. . .

    56. Question Design: General Guidelines For closed-ended questions, make sure that response options are mutually exclusive. Use specific date / time references to improve respondent recall. Make sure data can be compared to existing sources of information if you need to compare local, state, and national statistics. Always remember, too, that you can borrow questions from other questionnaires that have been tested (e.g., ones used by other health departments, neighboring research institutes, and CDC). Particularly for food borne outbreak investigations, the Centers for Disease Control and Prevention has several questionnaire templates available online for your reference. The URLs for those templates are provided in the reference slides at the end of this presentation. Always remember, too, that you can borrow questions from other questionnaires that have been tested (e.g., ones used by other health departments, neighboring research institutes, and CDC). Particularly for food borne outbreak investigations, the Centers for Disease Control and Prevention has several questionnaire templates available online for your reference. The URLs for those templates are provided in the reference slides at the end of this presentation.

    57. Question Type Now we are ready to move on to discuss the three main types of questions that you might include on a questionnaire. Now we are ready to move on to discuss the three main types of questions that you might include on a questionnaire.

    58. Question Type Questions fall into three broad categories: Closed-ended Open-ended Fill-in-the-blank Data collected and analysis options may differ by question type. There are three broad categories of question type: Closed-ended Open-ended Fill-in-the-blank When you design a questionnaire, you may use only one type (most likely closed-ended only on a self administered or hypothesis testing questionnaire) or a combination of several types of questions. There is no real right answer in terms of which question type you use. However, several factors can influence your decision to use a particular question type. Those factors include, but are not limited to: Whether you are designing a hypothesis generating versus a hypothesis testing questionnaire Your research objectives The type of information that you need to collect and analyze (e.g., qualitative versus quantitative data) this really goes hand-in-hand with whether or not you are designing a hypothesis generating versus testing questionnaire Whether or not you have enough information to provide specific, categorical response options for respondents Whether or not responses will be objective versus subjectiveThere are three broad categories of question type: Closed-ended Open-ended Fill-in-the-blank When you design a questionnaire, you may use only one type (most likely closed-ended only on a self administered or hypothesis testing questionnaire) or a combination of several types of questions. There is no real right answer in terms of which question type you use. However, several factors can influence your decision to use a particular question type. Those factors include, but are not limited to: Whether you are designing a hypothesis generating versus a hypothesis testing questionnaire Your research objectives The type of information that you need to collect and analyze (e.g., qualitative versus quantitative data) this really goes hand-in-hand with whether or not you are designing a hypothesis generating versus testing questionnaire Whether or not you have enough information to provide specific, categorical response options for respondents Whether or not responses will be objective versus subjective

    59. Closed-ended Questions

    60. Question Type Closed-Ended Questions: Provide answer choices in pre-coded categories that represent counts, ranges, or demographic information. Yield quantitative data. Are preferable for self-administered and hypothesis testing questionnaires. READ SLIDE FIRST Quantitative data can be counted and used to generate a frequency distribution (either stratified or unstratified), thereby giving you the ability to quickly assess any trends in the sample of case-patients. Closed-ended questions are preferable for a self-administered survey because they leave less room for misinterpretation of the question stem, and no interviewer is available to clarify the meaning of a question. Closed-ended questions are preferable for hypothesis testing questionnaires because the data will be used to test for a statistically significant association between exposure and outcome. READ SLIDE FIRST Quantitative data can be counted and used to generate a frequency distribution (either stratified or unstratified), thereby giving you the ability to quickly assess any trends in the sample of case-patients. Closed-ended questions are preferable for a self-administered survey because they leave less room for misinterpretation of the question stem, and no interviewer is available to clarify the meaning of a question. Closed-ended questions are preferable for hypothesis testing questionnaires because the data will be used to test for a statistically significant association between exposure and outcome.

    61. Question Type There are two ways in which categorical measurement is carried out in closed-ended questions: Nominal Ordinal

    62. Question Type Nominal response options result in the classification of a respondent into a category for race, gender, marital status, etc. - Response options can be mutually exclusive (Select only one response)

    63. Question Type Example: Nominal response options in a closed-ended question: Please select the one response that describes your marital status. Single Married Divorced Widowed Nominal response options are unordered categories that do NOT fall along a continuum (as you will see, ordinal response options for closed-ended questions DO fall along a subjective continuum). The response options therefore require respondents to process more information about themselves before selecting the most appropriate response. NOTE: When you use closed-ended questions with nominal response options, you may sometimes choose to include an Other response option, for which the respondent then essentially fills in the blank with the response that most accurately describes him or her. This could be particularly helpful in an outbreak investigation if you have overlooked an exposure in your list of response options it helps you retain an exploratory data collection mechanism even if you think you know what disease or exposure you are investigating. Depending on the purpose of a question, if you find that respondents repeatedly use this other response option, you may want to re-evaluate your question design. Most likely, the response options that you have provided are inadequate. Nominal response options are unordered categories that do NOT fall along a continuum (as you will see, ordinal response options for closed-ended questions DO fall along a subjective continuum). The response options therefore require respondents to process more information about themselves before selecting the most appropriate response. NOTE: When you use closed-ended questions with nominal response options, you may sometimes choose to include an Other response option, for which the respondent then essentially fills in the blank with the response that most accurately describes him or her. This could be particularly helpful in an outbreak investigation if you have overlooked an exposure in your list of response options it helps you retain an exploratory data collection mechanism even if you think you know what disease or exposure you are investigating. Depending on the purpose of a question, if you find that respondents repeatedly use this other response option, you may want to re-evaluate your question design. Most likely, the response options that you have provided are inadequate.

    64. Question Type Ordinal response options result in classification into an ordered category that may or may not rank values that are equidistant. - Ordinal responses have a numeric value.

    65. Question Type Example: Ordinal response options in a closed-ended question: When you turn off the frozen dairy dessert machine at the restaurants closing time, how frequently do you sanitize the dispenser? 1 Every time 4 Once a month 2 Every other time 5 Less than once a month 3 Once a week Ordinal responses tend to run along a subjective continuum, and include the complete range of possible answers. Note that each of the response option categories are correctly mutually exclusive. This type of question format is easy for respondents to answer because it is so specific. It is also easier for you, the researcher, to code, enter, and analyze when you number (or pre-code) the response options versus use check boxes! Ordinal responses tend to run along a subjective continuum, and include the complete range of possible answers. Note that each of the response option categories are correctly mutually exclusive. This type of question format is easy for respondents to answer because it is so specific. It is also easier for you, the researcher, to code, enter, and analyze when you number (or pre-code) the response options versus use check boxes!

    66. Question Type Likert scales contain pre-coded ordinal responses with assigned values. You can then calculate averages to determine the most prevalent response. Example: Please rate the severity of your abdominal cramp pain (circle one): 1 minimal 3 moderate 5 very painful 2 mild 4 painful You are probably familiar with scales that ask you to strongly agree, versus somewhat agree , et cetera. This example uses a Likert scale to allow respondents to select a pain rating along a subjective continuum. You are probably familiar with scales that ask you to strongly agree, versus somewhat agree , et cetera. This example uses a Likert scale to allow respondents to select a pain rating along a subjective continuum.

    67. Question Type: Likert Scales Remember, too, that the number of options that you include on a subjective continuum of a scale like this will influence respondents choices! Unbalanced response choices can lead to biased responses. For example: [READ SLIDE] Respondents may react to the options differently and even select a different answer. In this example, the status of moderate in the three-point versus five-point continuum differs significantly! Remember, too, that the number of options that you include on a subjective continuum of a scale like this will influence respondents choices! Unbalanced response choices can lead to biased responses. For example: [READ SLIDE] Respondents may react to the options differently and even select a different answer. In this example, the status of moderate in the three-point versus five-point continuum differs significantly!

    68. Question Type: Likert Scales Example: Please rate the severity of your abdominal cramp pain: 1 minimal 3 moderate 5 very painful 2 mild 4 painful Calculate the average of responses, where N = 35: 1 = 10 people; 2 = 5 people; 3 = 20 people; 4 = 0; 5 = 0 10 x 1 = 10; 2 x 5 = 10; 3 x 20 = 60 10 + 10 + 60 = 80 80 / 35 = 2.29 (mild pain) While I dont want to stray from the focus of question design and get into data analysis, I want to quickly show you how to hand calculate an average for a Likert scale question in case you find this question type to be intimidating. READ THROUGH CALCULATION STEPS 1 - 4. In the data analysis phase of your study, you should be cautious about how you interpret data collected via Likert scales. A Likert scale really helps you order respondents and allows you to compare trends over similar questions or across multiple study populations (e.g., if you were to implement the same survey in three neighboring counties during an outbreak investigation). But be sure that your summary statements about these data are comparative versus absolute. While I dont want to stray from the focus of question design and get into data analysis, I want to quickly show you how to hand calculate an average for a Likert scale question in case you find this question type to be intimidating. READ THROUGH CALCULATION STEPS 1 - 4. In the data analysis phase of your study, you should be cautious about how you interpret data collected via Likert scales. A Likert scale really helps you order respondents and allows you to compare trends over similar questions or across multiple study populations (e.g., if you were to implement the same survey in three neighboring counties during an outbreak investigation). But be sure that your summary statements about these data are comparative versus absolute.

    69. Question Type Using Dont Know or Not Applicable response options When you have categorical, dichotomous response options such as, Yes and No, you may sometimes choose to add a Dont Know or Not Applicable option. Particularly with a hypothesis generating survey in an outbreak investigation that might rely upon a respondents recall of exposure to certain illnesses or foods within several days or weeks, the respondent may legitimately need the option of a Dont Know response. Similarly, if you are waiting for laboratory results for case confirmation, you may want to incorporate a not yet available response option along with positive & negative or high and low. Ideally, you should not need to include a not applicable response option for a question if you have used skip patterns for question navigation where appropriate. *NOTE: These alternative response options to seemingly dichotomous response options have an impact when you design a View for data entry in Epi Info software. I will tell you more about this later in todays session. Particularly with a hypothesis generating survey in an outbreak investigation that might rely upon a respondents recall of exposure to certain illnesses or foods within several days or weeks, the respondent may legitimately need the option of a Dont Know response. Similarly, if you are waiting for laboratory results for case confirmation, you may want to incorporate a not yet available response option along with positive & negative or high and low. Ideally, you should not need to include a not applicable response option for a question if you have used skip patterns for question navigation where appropriate. *NOTE: These alternative response options to seemingly dichotomous response options have an impact when you design a View for data entry in Epi Info software. I will tell you more about this later in todays session.

    70. Open-ended Questions The next question type that I will discuss today is Open-Ended.The next question type that I will discuss today is Open-Ended.

    71. Question Type Open-Ended Questions: Allow respondents to provide answers in their own words. Yield qualitative data. May yield unanticipated answers that contribute to the study. Are most appropriate for hypothesis generating versus testing questionnaires. READ SLIDE FIRST Qualitative data analysis, when done properly, involves multiple stages of categorizing and coding responses. Because qualitative data can be the most difficult to analyze, you will use primarily closed-ended questions on a hypothesis testing questionnaire that is part of an analytic study during an outbreak investigation. However, you may use open-ended questions for exploratory data collection with a hypothesis generating questionnaire particularly since you will be interviewing only a small number of case-patients during the hypothesis generating phase of an investigation. READ SLIDE FIRST Qualitative data analysis, when done properly, involves multiple stages of categorizing and coding responses. Because qualitative data can be the most difficult to analyze, you will use primarily closed-ended questions on a hypothesis testing questionnaire that is part of an analytic study during an outbreak investigation. However, you may use open-ended questions for exploratory data collection with a hypothesis generating questionnaire particularly since you will be interviewing only a small number of case-patients during the hypothesis generating phase of an investigation.

    72. Question Type Example: Open-ended questions What restaurants did you patronize in the past seven days? Please list the two main symptoms you are having with this illness: To get you warmed up for an activity that is at the end of this discussion, lets briefly discuss some design flaws in these two open ended questions: 1. This first question may not be clear enough. Particularly with an open-ended question format, you should specify the geographic proximity of restaurants: e.g., What restaurants in Chapel Hill. . . Or What restaurants in North Carolina. . . Etc. With this geographic limitation, it would also be important to include questions about travel outside of Chapel Hill [or North Carolina] on your survey. Another potential problem with the first question: If all people in the survey sample are interviewed on the same date, then . . .in the past seven days may be sufficient. However, if you survey hundreds or thousands of people over several days, you will need to make the question as specific as possible to the suspected date / time of exposure and relevant to the incubation period for the illness under investigation. So you might change the wording to, . . . From June 2 through June 9, 2004? With the second question, assuming that you would be employing the question in an interviewer administered setting, you might have the interviewer briefly define the term, symptom to the respondent. Also, in a real outbreak investigation, you would probably NOT want to use an open ended question to obtain this information. It would be much better to use a closed-ended question with categorical responses that include any possible symptoms that are part of criteria for the clinical case definition. If you used this open-ended question, you might miss cases! To get you warmed up for an activity that is at the end of this discussion, lets briefly discuss some design flaws in these two open ended questions: 1. This first question may not be clear enough. Particularly with an open-ended question format, you should specify the geographic proximity of restaurants: e.g., What restaurants in Chapel Hill. . . Or What restaurants in North Carolina. . . Etc. With this geographic limitation, it would also be important to include questions about travel outside of Chapel Hill [or North Carolina] on your survey. Another potential problem with the first question: If all people in the survey sample are interviewed on the same date, then . . .in the past seven days may be sufficient. However, if you survey hundreds or thousands of people over several days, you will need to make the question as specific as possible to the suspected date / time of exposure and relevant to the incubation period for the illness under investigation. So you might change the wording to, . . . From June 2 through June 9, 2004? With the second question, assuming that you would be employing the question in an interviewer administered setting, you might have the interviewer briefly define the term, symptom to the respondent. Also, in a real outbreak investigation, you would probably NOT want to use an open ended question to obtain this information. It would be much better to use a closed-ended question with categorical responses that include any possible symptoms that are part of criteria for the clinical case definition. If you used this open-ended question, you might miss cases!

    73. Fill-in-the-blank Questions The third and final question type that I will discuss today is Fill-in-the-blank. The third and final question type that I will discuss today is Fill-in-the-blank.

    74. Question Type Fill-in-the-blank Questions: Allow respondents to provide short answers in their own words. Yield qualitative data. Are most appropriate when possible response categories are too numerous to list. Are most appropriate when the question is measuring respondent characteristics versus attitudes, beliefs, or behaviors.

    75. Question Type Example: Fill-in-the-blank Questions 1. County of residence _____ 2. Age in years ____ 3. Number of children under age 18 in your household ____ Here, a fill-in-the-blank question about county of residence is both appropriate and ideal in a state like North Carolina with its 100 counties. A survey would look very crowded with 100 response options for county of residence! Also, in Epi Info software, each data table associated with a View or data entry interface has a limit of 256 fields. And if you needed a field for each possible county, you would use up 100 fields with only one question! Here, a fill-in-the-blank question about county of residence is both appropriate and ideal in a state like North Carolina with its 100 counties. A survey would look very crowded with 100 response options for county of residence! Also, in Epi Info software, each data table associated with a View or data entry interface has a limit of 256 fields. And if you needed a field for each possible county, you would use up 100 fields with only one question!

    76. Question and Answer Opportunity What questions do you have about the three broad question types that you can incorporate into a survey instrument?What questions do you have about the three broad question types that you can incorporate into a survey instrument?

    77. 5 minute break We are going to take another 5 minute break. When we reconvene, I will facilitate a five-minute activity that you can complete at your sites. We are going to take another 5 minute break. When we reconvene, I will facilitate a five-minute activity that you can complete at your sites.

    78. Activity Completion Time: 5 minutes Now that you have learned about the elements of good question design and the three broad categories of question type, I want to ask you to participate in a short activity. Now that you have learned about the elements of good question design and the three broad categories of question type, I want to ask you to participate in a short activity.

    79. Activity Instructions: Three sample questions appear on the following slides. Your task is to critique those questions and write down one element that could be improved or modified to make the question better. During this activity, you are going to critique three questions. As you critique a question, remember that there are many elements that might make it a well or poorly designed one. You may need to recommend modification of the actual question type used (e.g, closed-ended versus open-ended). Or, you may suggest better or more complete wording. Or, you may recommend modification of only the response options (e.g., perhaps not all options are mutually exclusive). I will give you all one minute per question to critique and write comments. Then I will show you my suggested answers for possible improvements. During this activity, you are going to critique three questions. As you critique a question, remember that there are many elements that might make it a well or poorly designed one. You may need to recommend modification of the actual question type used (e.g, closed-ended versus open-ended). Or, you may suggest better or more complete wording. Or, you may recommend modification of only the response options (e.g., perhaps not all options are mutually exclusive). I will give you all one minute per question to critique and write comments. Then I will show you my suggested answers for possible improvements.

    80. Activity Question 1: Did you take any prescription medications for this illness? Yes No If yes, what medications? ___________ The first part of this question is all right. For example, the wording is specific enough to distinguish between prescription versus over-the-counter medication. However, the follow up question / part B of this could be formatted differently. Rather than fill-in-the-blank, this question might be best formatted as a closed-ended question with pre-determined, categorical responses (for example, the medications most commonly used to treat symptoms of illness X that is under investigation). And you could have one final, other category with a fill-in-the-blank option. There are a few reasons for my recommendation to alter question format here: First, based on research objectives, is there a particular medication that you, as the researcher are interested in identifying among respondents? If yes, then you will want to include it in a list of categorical response options versus leaving it up to respondents to possibly come up with it. If respondents took multiple medications, they might remember one but not necessarily the one you are looking for! Second, respondents may have trouble remembering the name of a prescription medication particularly if they have never taken it before. And finally, particularly in a self-administered setting in which you want the questionnaire to be as easy (and quick) to fill out as possible, the closed-ended format would be easier for a respondent versus sitting and trying to recall the name(s) of medications. The first part of this question is all right. For example, the wording is specific enough to distinguish between prescription versus over-the-counter medication. However, the follow up question / part B of this could be formatted differently. Rather than fill-in-the-blank, this question might be best formatted as a closed-ended question with pre-determined, categorical responses (for example, the medications most commonly used to treat symptoms of illness X that is under investigation). And you could have one final, other category with a fill-in-the-blank option. There are a few reasons for my recommendation to alter question format here: First, based on research objectives, is there a particular medication that you, as the researcher are interested in identifying among respondents? If yes, then you will want to include it in a list of categorical response options versus leaving it up to respondents to possibly come up with it. If respondents took multiple medications, they might remember one but not necessarily the one you are looking for! Second, respondents may have trouble remembering the name of a prescription medication particularly if they have never taken it before. And finally, particularly in a self-administered setting in which you want the questionnaire to be as easy (and quick) to fill out as possible, the closed-ended format would be easier for a respondent versus sitting and trying to recall the name(s) of medications.

    81. Activity Question 2: Did you travel anywhere during the seven days before your illness? Yes No If airline travel, what airline? __________ Outgoing flight number ____ Returning flight number ____ Now here is an example of asking for too many details that may or may not contribute to you meeting your research objectives. In a hypothesis generating survey, it is absolutely appropriate (in fact, standard) to ask about travel history, as I illustrated at the beginning of this session with the EIS questionnaire. However, particularly in a hypothesis generating versus testing / follow up survey, information about which airline and outgoing and returning flight numbers are probably not necessary. And even if this were a hypothesis testing survey, how realistic is it to expect a person to remember (or even have on file) their outgoing and returning flight numbers? And what would you DO with this information? Well, you would probably want it for the purpose of taking short-term control measures so you could contact all passengers who had been on board the same plane as your responding case. But again expecting recall on an item like this might be unrealistic. You, as the researcher, could probably track down flight numbers if you asked easier follow up questions such as, destination city and linked that up with the airline and date of travel. I also want to point out that for both Question 1 and this one, instructions for a skip pattern should be in place whether the questionnaire is self-administered or interviewer administered. A skip pattern directs the respondent to a new series of questions if the response at the beginning of a related series of questions is, no. Now here is an example of asking for too many details that may or may not contribute to you meeting your research objectives. In a hypothesis generating survey, it is absolutely appropriate (in fact, standard) to ask about travel history, as I illustrated at the beginning of this session with the EIS questionnaire. However, particularly in a hypothesis generating versus testing / follow up survey, information about which airline and outgoing and returning flight numbers are probably not necessary. And even if this were a hypothesis testing survey, how realistic is it to expect a person to remember (or even have on file) their outgoing and returning flight numbers? And what would you DO with this information? Well, you would probably want it for the purpose of taking short-term control measures so you could contact all passengers who had been on board the same plane as your responding case. But again expecting recall on an item like this might be unrealistic. You, as the researcher, could probably track down flight numbers if you asked easier follow up questions such as, destination city and linked that up with the airline and date of travel. I also want to point out that for both Question 1 and this one, instructions for a skip pattern should be in place whether the questionnaire is self-administered or interviewer administered. A skip pattern directs the respondent to a new series of questions if the response at the beginning of a related series of questions is, no.

    82. Activity Question 3: During the seven days before your illness, did you have any pets at home, have contact with household pets elsewhere, or visit a household with pets (including reptiles)? Yes No There are several problems with this question. The respondent is really being asked three questions. Lets break them down: Did you have any pets at home? Did you have contact with household pets elsewhere? Did you visit a household with pets? First, pets and household pets are used in the same question do they define different things? The respondent cannot be sure. This would be a good place for a clear definition in parenthesis for either the respondent or an interviewer to read. (e.g., By household pet, we mean. . .). Also, no clear definition or clarification of which animals should be included is provided, other than the fact that they should consider reptiles to be included in these categories. But if you are going to provide a suggested exposure, why not include all / other animals that can transmit salmonella in a categorical list of response options, with the addition of a fill in the blank other category? [I am assuming that the researchers are conducting a Salmonella outbreak investigation]. And finally, this question is probably too complex and asking more than it needs to to meet the research objectives. If my assumption is correct and epidemiologists are investigating a salmonella outbreak, are researchers really only interested in learning whether or not a respondent has come in contact with a reptile in the seven days before their illness? Why not just ask THAT? The question as it is currently written will yield a yes response about exposure to cats, dogs, hamsters, etc. . . At a minimum, this question as it is worded would require a follow up question specifically about reptile exposure. There are several problems with this question. The respondent is really being asked three questions. Lets break them down: Did you have any pets at home? Did you have contact with household pets elsewhere? Did you visit a household with pets? First, pets and household pets are used in the same question do they define different things? The respondent cannot be sure. This would be a good place for a clear definition in parenthesis for either the respondent or an interviewer to read. (e.g., By household pet, we mean. . .). Also, no clear definition or clarification of which animals should be included is provided, other than the fact that they should consider reptiles to be included in these categories. But if you are going to provide a suggested exposure, why not include all / other animals that can transmit salmonella in a categorical list of response options, with the addition of a fill in the blank other category? [I am assuming that the researchers are conducting a Salmonella outbreak investigation]. And finally, this question is probably too complex and asking more than it needs to to meet the research objectives. If my assumption is correct and epidemiologists are investigating a salmonella outbreak, are researchers really only interested in learning whether or not a respondent has come in contact with a reptile in the seven days before their illness? Why not just ask THAT? The question as it is currently written will yield a yes response about exposure to cats, dogs, hamsters, etc. . . At a minimum, this question as it is worded would require a follow up question specifically about reptile exposure.

    83. Epi Info Impact of Question Design on View Design and Data Analysis Many of you in the audience may have participated in one of the Epi Info software trainings that have been sponsored by the Office of Public Health Preparedness and Response throughout North Carolina. I therefore wanted to include a short segment in todays session on the implications of question type used in a survey on View Design and data analysis in Epi Info software. For those of you who are unfamiliar with Epi Info, a View is simply the on-screen data entry form (it can be created to look identical to a hard copy survey). As you design a View, you set up a database structure for data entry. Many of you in the audience may have participated in one of the Epi Info software trainings that have been sponsored by the Office of Public Health Preparedness and Response throughout North Carolina. I therefore wanted to include a short segment in todays session on the implications of question type used in a survey on View Design and data analysis in Epi Info software. For those of you who are unfamiliar with Epi Info, a View is simply the on-screen data entry form (it can be created to look identical to a hard copy survey). As you design a View, you set up a database structure for data entry.

    84. Epi Info Implications Overview: Field type and properties Check code in MakeView Categorical data Dichotomous response options So I want to talk about three broad aspects of database design and analysis options that you will want to keep in mind as you design each question on your survey instrument particularly if you plan to enter data into an Epi Info database. These aspects are: [READ SLIDE] So lets begin by looking at field type and properties.So I want to talk about three broad aspects of database design and analysis options that you will want to keep in mind as you design each question on your survey instrument particularly if you plan to enter data into an Epi Info database. These aspects are: [READ SLIDE] So lets begin by looking at field type and properties.

    85. Tables To understand what field type and properties are, we need to back up a bit and quickly review the three key components of a database: Tables versus Records versus Fields. Tables are the core of your database the largest unit of information where data are stored by rows and Columns. To understand what field type and properties are, we need to back up a bit and quickly review the three key components of a database: Tables versus Records versus Fields. Tables are the core of your database the largest unit of information where data are stored by rows and Columns.

    86. Records Each record contains a complete set of information on one specific subject or case-patient E.g., record #1 = John Does name, address, date of onset of illness, lab results, etc. E.g., record #2 = Jane Does name address, date of onset of illness, lab results, etc. Each record contains a complete set of information on one specific subject or case-patient. Each completed hard copy survey that you have represents ONE RECORD. Each record contains a complete set of information on one specific subject or case-patient. Each completed hard copy survey that you have represents ONE RECORD.

    87. Fields The smallest unit of information in your database. Each field contains one piece of information about a subject / case-patient. E.g., field 1= first name E.g., field 2= street address Each field contains one piece of information about a subject / case-patient. Therefore, each piece of information such as the line where you enter age or survey date on your hard copy survey represents ONE FIELD. Each field contains one piece of information about a subject / case-patient. Therefore, each piece of information such as the line where you enter age or survey date on your hard copy survey represents ONE FIELD.

    88. Sample Database Tables Here is a visual representation of the three database components: tables, records, and fields. [READ FOOTER TEXT ON SLIDE] Remember that as you design each question for a questionnaire, each response option will become an individual field in your database. For example, on a food borne illness outbreak investigation hypothesis generating questionnaire, each food or beverage will be represented by an individual field in your database structure. Here is a visual representation of the three database components: tables, records, and fields. [READ FOOTER TEXT ON SLIDE] Remember that as you design each question for a questionnaire, each response option will become an individual field in your database. For example, on a food borne illness outbreak investigation hypothesis generating questionnaire, each food or beverage will be represented by an individual field in your database structure.

    89. Sample Epi Info View Here you can see how each food or beverage is represented by an individual field in this sample View screen in Epi Info. The blue boxes you see are simply where the fields have been grouped together for faster analysis programming options. Here you can see how each food or beverage is represented by an individual field in this sample View screen in Epi Info. The blue boxes you see are simply where the fields have been grouped together for faster analysis programming options.

    90. Field Type and Properties You know that there may be many different TYPES of information that you collect with a questionnaire. Regardless of whether a question is closed-ended, open-ended, or fill-in-the-blank, the data that the question yields may be text (e.g., first name, city, grade) or numeric (age, body weight). Epi Info software has SEVENTEEN data types that you can assign to a field when you create a data entry interface for your database. The most basic or commonly used ones include text and number. The image on this slide shows the Field Definition Window that you use in Epi Infos MakeView component to add one field at a time to the design of your data entry interface screen. The place where you have the option to select the type of data that the field will store is circled in red. You know that there may be many different TYPES of information that you collect with a questionnaire. Regardless of whether a question is closed-ended, open-ended, or fill-in-the-blank, the data that the question yields may be text (e.g., first name, city, grade) or numeric (age, body weight). Epi Info software has SEVENTEEN data types that you can assign to a field when you create a data entry interface for your database. The most basic or commonly used ones include text and number. The image on this slide shows the Field Definition Window that you use in Epi Infos MakeView component to add one field at a time to the design of your data entry interface screen. The place where you have the option to select the type of data that the field will store is circled in red.

    91. Field Type and Properties [Number Type] Age in years _____ Number of children living in this household _____ [Text Type] County of residence _____ First name _____ If you use a fill-in-the-blank question format, you will have text or perhaps a number to enter into a field during data entry. For example, you might be asking for age in years [requiring a number format] or number of children living in this household [requiring a number format] or county of residence [requiring a text format]. These fill-in-the-blank responses are very short, and therefore are not at risk of exceeding the 126 character limit for a text type field in Epi Info. If you use an open-ended question format, you might still get only a few words in a response, but you might also get a paragraph of text to enter into your database. If you know that the response length exceeds the 126 character limit for a text type field, you can use a modified version of the text type, which is called, multiline. Multiline field type allows you to enter as much text as you want data storage capacity is virtually unlimited. If you use a fill-in-the-blank question format, you will have text or perhaps a number to enter into a field during data entry. For example, you might be asking for age in years [requiring a number format] or number of children living in this household [requiring a number format] or county of residence [requiring a text format]. These fill-in-the-blank responses are very short, and therefore are not at risk of exceeding the 126 character limit for a text type field in Epi Info. If you use an open-ended question format, you might still get only a few words in a response, but you might also get a paragraph of text to enter into your database. If you know that the response length exceeds the 126 character limit for a text type field, you can use a modified version of the text type, which is called, multiline. Multiline field type allows you to enter as much text as you want data storage capacity is virtually unlimited.

    92. Field Type and Properties Here is the Epi Info View with individual fields for each food or beverage item that I showed to you a moment ago. For each food or beverage item, you would have one of two formatting options during field programming in Epi Info: a. Checkbox b. Yes/ No Either format looks similar on the data entry screen. During data entry for a check box field, you click on a check box with the mouse to indicate that data should be recorded for a response option. For yes / no boxes, you can type in the full word or simply y or n. So WHY do you need to enter data for each food item separately? Well, you need to be able to analyze each item as a potential risk factor. By separating items (e.g, milk, coffee, and water versus beverage), you can specifically pinpoint a suspected risk factor and generate a hypothesis for further analytic study. You can use the field-specific data to quickly generate two pieces of analysis output that may allow you to form a hypothesis: Line Listing Frequency table I want to quickly show you the output that you can generate with this field format, but we will not focus on data analysis in-depth until the upcoming October 12th PHTIN session on, Analyzing Data. Here is the Epi Info View with individual fields for each food or beverage item that I showed to you a moment ago. For each food or beverage item, you would have one of two formatting options during field programming in Epi Info: a. Checkbox b. Yes/ No Either format looks similar on the data entry screen. During data entry for a check box field, you click on a check box with the mouse to indicate that data should be recorded for a response option. For yes / no boxes, you can type in the full word or simply y or n. So WHY do you need to enter data for each food item separately? Well, you need to be able to analyze each item as a potential risk factor. By separating items (e.g, milk, coffee, and water versus beverage), you can specifically pinpoint a suspected risk factor and generate a hypothesis for further analytic study. You can use the field-specific data to quickly generate two pieces of analysis output that may allow you to form a hypothesis: Line Listing Frequency table I want to quickly show you the output that you can generate with this field format, but we will not focus on data analysis in-depth until the upcoming October 12th PHTIN session on, Analyzing Data.

    93. Line Listing Here is an image of a sample line listing in the Analysis component of Epi Info software. You can see that you can sort by age in ascending order, and then quickly scan the data table to note any clusters of a particular age or gender within the ill population that have consumed milk, coffee, or water. For that matter, you could make this list even easier to look at by simultaneously sorting by gender. Here is an image of a sample line listing in the Analysis component of Epi Info software. You can see that you can sort by age in ascending order, and then quickly scan the data table to note any clusters of a particular age or gender within the ill population that have consumed milk, coffee, or water. For that matter, you could make this list even easier to look at by simultaneously sorting by gender.

    94. Frequency Table Similarly, you can generate a frequency table and even stratify it by illness status, gender, or age for each food item as a means of looking for a suspicious food source. In this example, it is pretty clear that milk is NOT the source of infection in this outbreak under investigation. This frequency table tells you both in writing and graphically that 95.7% of the ill people in your study did NOT consume milk. You can see why it is helpful to program response options specifically food and beverage items as individual fields in a database structure if you want to maximize your ability to analyze data. Similarly, you can generate a frequency table and even stratify it by illness status, gender, or age for each food item as a means of looking for a suspicious food source. In this example, it is pretty clear that milk is NOT the source of infection in this outbreak under investigation. This frequency table tells you both in writing and graphically that 95.7% of the ill people in your study did NOT consume milk. You can see why it is helpful to program response options specifically food and beverage items as individual fields in a database structure if you want to maximize your ability to analyze data.

    95. Dichotomous Response Options Which of the following food items did you consume in March or April of this year? Commercial milk purchased locally Yes No Fresh clams purchased locally Yes No Fresh oysters purchased locally Yes No None of the above Yes No Here is a sample question stem and its dichotomous [e.g., two-part] response options. This is one of the questions from the hypothesis generating survey that I showed you at the beginning of todays session. Your View programming options for this response option field type would be: Either checkbox field type for each of the four response options (4 fields) or yes / no field type for each of the four response options. Here is a sample question stem and its dichotomous [e.g., two-part] response options. This is one of the questions from the hypothesis generating survey that I showed you at the beginning of todays session. Your View programming options for this response option field type would be: Either checkbox field type for each of the four response options (4 fields) or yes / no field type for each of the four response options.

    96. Contingency Table When you use dichotomous response options such as Yes and No, you can also generate contingency or 2 x 2 tables to evaluate the relationship between exposure and outcome variables in a study. Keep in mind that you would generate a 2 x 2 table with data from a hypothesis testing versus generating questionnaire. Remember that you would need to include both cases and non-cases in an analytic study survey sample to accurately measure any association between an exposure and the outcome of illness. Lets look at this sample contingency table: [see slide] This 2 x 2 table gives you even more information than the frequency table. Now you can compare the dichotomous response options for two fields (Illness status and milk consumption status) in a cross-tabulated format. We will discuss the interpretation of row and column frequencies and percents in analysis output during the October 12th PHTIN session on Analyzing Data. When you use dichotomous response options such as Yes and No, you can also generate contingency or 2 x 2 tables to evaluate the relationship between exposure and outcome variables in a study. Keep in mind that you would generate a 2 x 2 table with data from a hypothesis testing versus generating questionnaire. Remember that you would need to include both cases and non-cases in an analytic study survey sample to accurately measure any association between an exposure and the outcome of illness. Lets look at this sample contingency table: [see slide] This 2 x 2 table gives you even more information than the frequency table. Now you can compare the dichotomous response options for two fields (Illness status and milk consumption status) in a cross-tabulated format. We will discuss the interpretation of row and column frequencies and percents in analysis output during the October 12th PHTIN session on Analyzing Data.

    97. Check Code in MakeView Eight Check Code commands in the Field Definition window: Repeat Last Legal Values Required Codes Read Only Comment Legal Soundex Range* * Use only with number formatted fields So far I have talked about question design and the implications for View design and data analysis for field type and properties and dichotomous response options. The last consideration that I want to address is your option to use check code programming for fields that represent mutually exclusive (e.g., the respondent can select only one response option), categorical response options. Check Code in Epi Info software allows you to pre-program field parameters or restrictions. The code that you assign to a field in MakeView impacts data entry. I do not have time to teach you about each one of these types of Check Code available via the Field Definition Window, but do want to point out their function and show you some examples of a couple that you might use most frequently. The eight check code options visible in the Field Definition dialogue window are: [READ SLIDE] So far I have talked about question design and the implications for View design and data analysis for field type and properties and dichotomous response options. The last consideration that I want to address is your option to use check code programming for fields that represent mutually exclusive (e.g., the respondent can select only one response option), categorical response options. Check Code in Epi Info software allows you to pre-program field parameters or restrictions. The code that you assign to a field in MakeView impacts data entry. I do not have time to teach you about each one of these types of Check Code available via the Field Definition Window, but do want to point out their function and show you some examples of a couple that you might use most frequently. The eight check code options visible in the Field Definition dialogue window are: [READ SLIDE]

    98. Check Code in MakeView If you look at the lower half of the Field Definition Window, you see the eight check code options that I just listed. I will show you some illustrations of their use in just a moment. But I do not want to forget to point out the blue, Program button on the left portion of the MakeView screen. This Program button is what you use to enter Skip Pattern programming that will be reflected on the data entry screen. You may recall that I have mentioned skip patterns a few times today. If you build skip patterns into your hard copy survey, you can also build them into your data entry screen. Just as an interviewer will skip over a series of questions if the answer to the first question in the series is, No, Epi Info can make the cursor skip over a series of response options if the data entered for the first response option in a series of related ones is, No. Overall, skip pattern programming in Epi Info increases the efficiency with which a person can enter data, and also assists with preventing data entry errors. Now lets look at some examples of how you can pre-program nominal or ordinal categorical response options using Check Code buttons in the Field Definition window. . . If you look at the lower half of the Field Definition Window, you see the eight check code options that I just listed. I will show you some illustrations of their use in just a moment. But I do not want to forget to point out the blue, Program button on the left portion of the MakeView screen. This Program button is what you use to enter Skip Pattern programming that will be reflected on the data entry screen. You may recall that I have mentioned skip patterns a few times today. If you build skip patterns into your hard copy survey, you can also build them into your data entry screen. Just as an interviewer will skip over a series of questions if the answer to the first question in the series is, No, Epi Info can make the cursor skip over a series of response options if the data entered for the first response option in a series of related ones is, No. Overall, skip pattern programming in Epi Info increases the efficiency with which a person can enter data, and also assists with preventing data entry errors. Now lets look at some examples of how you can pre-program nominal or ordinal categorical response options using Check Code buttons in the Field Definition window. . .

    99. Check Code in MakeView If you use a closed-ended question format with nominal categorical responses, you can create a field with the categorical responses from the survey pre-programmed for data entry. The person entering data then merely needs to select a response option from a dropdown menu and click on it. The data type that you select for this specific field formatting is referred to as, Text with Legal Values. You select text in the type option box of the Field Definition window, but you also pre-program response options using the Legal Values button. During data entry, a person can only enter one of the pre-programmed response options For example, there is no chance that a county in the western part of the state will be included in your database if you have all eastern counties in the legal values list of pre-programmed response options. You may recall that earlier in todays session I noted that using Yes No and Dont Know or Not Applicable response options has implications for View design in Epi Info. Even though Yes and No are among response options, you cannot use a Yes / No field type in View design. You will actually need to use this Text with Legal Values programming option. If you use a closed-ended question format with nominal categorical responses, you can create a field with the categorical responses from the survey pre-programmed for data entry. The person entering data then merely needs to select a response option from a dropdown menu and click on it. The data type that you select for this specific field formatting is referred to as, Text with Legal Values. You select text in the type option box of the Field Definition window, but you also pre-program response options using the Legal Values button. During data entry, a person can only enter one of the pre-programmed response options For example, there is no chance that a county in the western part of the state will be included in your database if you have all eastern counties in the legal values list of pre-programmed response options. You may recall that earlier in todays session I noted that using Yes No and Dont Know or Not Applicable response options has implications for View design in Epi Info. Even though Yes and No are among response options, you cannot use a Yes / No field type in View design. You will actually need to use this Text with Legal Values programming option.

    100. Field Type and Properties Please select the one response that describes your marital status. Single Married Divorced Widowed Now lets look at a question with a restriction placed on the response options. Here, rather than a check all that apply scenario, we have a please select only one response scenario response options are mutually exclusive. In addition to being mutually exclusive, these response options are nominal. When you add a field to your View screen, you can assign the field type, Text with Legal Values that will accept only one of the four response options. When you are ready to analyze the data, you can generate frequency tables for each of the four possible marital status response options. Now lets look at a question with a restriction placed on the response options. Here, rather than a check all that apply scenario, we have a please select only one response scenario response options are mutually exclusive. In addition to being mutually exclusive, these response options are nominal. When you add a field to your View screen, you can assign the field type, Text with Legal Values that will accept only one of the four response options. When you are ready to analyze the data, you can generate frequency tables for each of the four possible marital status response options.

    101. Check Code in MakeView Here is how you program your response options into Epi Info. Note that I have selected the do not sort option. If you allow Epi Info to sort pre-programmed response options, it will alphabetize them. Even though my response options are nominal versus ordinal, I still prefer to keep the response options in the same order in which they appear on the hard copy of the survey instrument. It just saves time during data entry. Here is how you program your response options into Epi Info. Note that I have selected the do not sort option. If you allow Epi Info to sort pre-programmed response options, it will alphabetize them. Even though my response options are nominal versus ordinal, I still prefer to keep the response options in the same order in which they appear on the hard copy of the survey instrument. It just saves time during data entry.

    102. Field Type and Properties This slide shows you two ways to format the same question about marital status. Image A is what the Text with Legal Values formatting will look like on the data entry screen. One has to click on a dropdown menu arrow to see the list of response options. There is also a second alternative for field type with mutually exclusive response options called, option. You can see in image B how this type of field formatting yields radio buttons with all response options visible at the same time. Either one of the field formatting images shown on this slide will give you the same effect during data entry: only one option can be entered, because response options are mutually exclusive. This slide shows you two ways to format the same question about marital status. Image A is what the Text with Legal Values formatting will look like on the data entry screen. One has to click on a dropdown menu arrow to see the list of response options. There is also a second alternative for field type with mutually exclusive response options called, option. You can see in image B how this type of field formatting yields radio buttons with all response options visible at the same time. Either one of the field formatting images shown on this slide will give you the same effect during data entry: only one option can be entered, because response options are mutually exclusive.

    103. Check Code in MakeView Comment Legal Comment Legal check code is similar to Text with Legal Values. The primary distinction is that you not only enter pre-programmed response options each response option is coded with a number that is associated with it. During data analysis, only that number versus the word, agree or disagree will appear as a row header in a frequency table. Hopefully you also recognize the formatting of these response options as a Likert Scale that I discussed earlier in todays session. For these response options, it is critical to select the do not sort option we do not want our ordinal response options along a subjective continuum to be alphabetized! Comment Legal check code is similar to Text with Legal Values. The primary distinction is that you not only enter pre-programmed response options each response option is coded with a number that is associated with it. During data analysis, only that number versus the word, agree or disagree will appear as a row header in a frequency table. Hopefully you also recognize the formatting of these response options as a Likert Scale that I discussed earlier in todays session. For these response options, it is critical to select the do not sort option we do not want our ordinal response options along a subjective continuum to be alphabetized!

    104. Question and Answer Opportunity There are so many things that I want to teach you about transferring your hard copy survey onto an on screen, data entry interface in Epi Info. But since this is a training on Designing Questionnaires versus Epi Info software I am going to limit myself to what I just covered. What questions do you have about the impact of question design on the ability to code, enter, and analyze data in Epi Info software? Can I further clarify any field formatting considerations that I discussed today? There are so many things that I want to teach you about transferring your hard copy survey onto an on screen, data entry interface in Epi Info. But since this is a training on Designing Questionnaires versus Epi Info software I am going to limit myself to what I just covered. What questions do you have about the impact of question design on the ability to code, enter, and analyze data in Epi Info software? Can I further clarify any field formatting considerations that I discussed today?

    105. Questionnaire Format Now that I have talked about sound questionnaire design, the principles of good question design, and when and why to use different types of questions, lets consider the big picture: formatting your questionnaire. While question design is certainly a very critical element of successful survey research, you should not neglect the format of your questionnaire how you piece together those carefully constructed research questions. Now that I have talked about sound questionnaire design, the principles of good question design, and when and why to use different types of questions, lets consider the big picture: formatting your questionnaire. While question design is certainly a very critical element of successful survey research, you should not neglect the format of your questionnaire how you piece together those carefully constructed research questions.

    106. Questionnaire Format We will now consider layout guidelines: In general For interviewer administered questionnaires For self-administered questionnaires

    107. General Formatting Guidelines First, some general formatting guidelines that you can apply to all questionnaire types. . . First, some general formatting guidelines that you can apply to all questionnaire types. . .

    108. General Guidelines A well-designed questionnaire: Minimizes respondent burden Minimal time to think about each response Minimal time to complete entire survey READ SLIDE FIRST So far in todays session, I have discussed specific ways in which you can design individual questions with the goal of minimizing respondent burden. Now I am going to talk about ways in which you can format the questionnaire as a whole to make questionnaire completion as easy as possible for respondents. READ SLIDE FIRST So far in todays session, I have discussed specific ways in which you can design individual questions with the goal of minimizing respondent burden. Now I am going to talk about ways in which you can format the questionnaire as a whole to make questionnaire completion as easy as possible for respondents.

    109. General Guidelines: Respondent Perspective Format page layout with a vertical flow from question to question and from response option to response option. Maintain white space on the page. Number every question.

    110. General Guidelines: Respondent Perspective Use consistent codes for response options. Arrange questions in a logical order; even within sub-sections if possible. Use differentiating font for questions and response options. 5. For example, all A,B,C,D or 1,2,3,4 OR, if you use Likert Scale response options, you may even be able to group together those types of questions one after the other, with the Likert scale out to the right of the questions it makes it easier for respondents to maintain the flow of thinking in those measurement terms for a series of questions rather than having to respond in that format intermittently throughout a survey. 6. Then within sub-sections on related topics, you may also want to group question types together so respondents get into a flow of answering a certain way. For example, lump all yes / no questions together first, then all a. b. c. d. questions next, then finish up with open-ended questions. In relation to the question ordering theme, do not begin your survey with an open-ended or the most difficult (or sensitive in nature) question. 7. For example, italicize question stems and use normal font for response options, or make one bolded, one not bolded. 5. For example, all A,B,C,D or 1,2,3,4 OR, if you use Likert Scale response options, you may even be able to group together those types of questions one after the other, with the Likert scale out to the right of the questions it makes it easier for respondents to maintain the flow of thinking in those measurement terms for a series of questions rather than having to respond in that format intermittently throughout a survey. 6. Then within sub-sections on related topics, you may also want to group question types together so respondents get into a flow of answering a certain way. For example, lump all yes / no questions together first, then all a. b. c. d. questions next, then finish up with open-ended questions. In relation to the question ordering theme, do not begin your survey with an open-ended or the most difficult (or sensitive in nature) question. 7. For example, italicize question stems and use normal font for response options, or make one bolded, one not bolded.

    111. General Guidelines: Respondent Perspective Provide clear but brief instructions for survey completion. Indicate units for fill-in-the-blank questions. Use clear cues (text or arrows) to guide respondents through skip patterns. Use transitional cues for each subsection of questions (The next series of questions will ask about . . . ). 10. Particularly for self-administered questionnaires. And remember: well-designed questions that are reliable and valid in their measurement capacity should not require extensive instructions. Particularly for self-administered questionnaires, but you will need to include text or symbols for questionnaire navigation for interviewers also. This could be helpful for either self-administered or interviewer administered questionnaires. 10. Particularly for self-administered questionnaires. And remember: well-designed questions that are reliable and valid in their measurement capacity should not require extensive instructions. Particularly for self-administered questionnaires, but you will need to include text or symbols for questionnaire navigation for interviewers also. This could be helpful for either self-administered or interviewer administered questionnaires.

    112. General Guidelines: Researcher Perspective Include a unique identification number so you can maintain confidentiality and link data when applicable (e.g., in Epi Info). When applicable, use numbers versus check boxes for pre-coded response options so they can be used for data entry and analysis (e.g., in Epi Info). Some questionnaire formatting strategies benefit the researcher more than the respondent. [READ THIS SLIDE AND NEXT TWO SLIDES] Some questionnaire formatting strategies benefit the researcher more than the respondent. [READ THIS SLIDE AND NEXT TWO SLIDES]

    113. General Guidelines: Researcher Perspective To reduce the difficulty of distinguishing between missing data and simply all no responses to a categorical list of response options, consider alternative formatting. Example: standard question format Where did you see published information about this free HIV screening? Billboard on subway Doctors office Local newspaper

    114. General Guidelines: Researcher Perspective Example: alternative question format Where did you see published information about this free HIV screening? Billboard on subway Yes No Doctors office Yes No Local newspaper Yes No

    115. Format Guidelines: Interviewer Administered Questionnaire Interviewer administered questionnaires may be used over the telephone or in a face-to-face setting during outbreak investigations. Lets look at some specific examples of how to implement the best formatting structure. Interviewer administered questionnaires may be used over the telephone or in a face-to-face setting during outbreak investigations. Lets look at some specific examples of how to implement the best formatting structure.

    116. Questionnaire Format Interviewer administered questionnaire: Differentiate between text instructions for the interviewer and questions to be asked (e.g., italicize or bold instructions). Establish a clear system for skip patterns (e.g., lines and arrows or clear instructions).

    117. Questionnaire Format Interviewer administered questionnaire: Put optional wording in parentheses. Example: (his / her) Put all words for the interviewer to say in writing to assure standardized interviewing (e.g., introductions, transitions, explanations, or definitions).

    118. Interviewer Administered Questionnaires Suggested Supporting Documentation for Interviewers Whether in a face-to-face or telephone interview setting, interviewers will be accessible to respondents for clarification or definitions, for information about the purpose of the survey, or other questions. Therefore, interviewers need to be able to answer any anticipated respondent questions that may arise. Today our guest lecturer, Kay Sanford, will talk to you about several pieces of documentation that she helped develop to accompany the 2003 post-Hurricane Isabel Rapid Needs Assessment. A part of the interviewer training protocol that Ms. Anjum Hajat discussed in the August Interviewing Techniques PHTIN session includes providing background information about the survey to interviewers. This information serves a dual purpose of informing interviewers and serving as a reference in an interviewer manual. Whether in a face-to-face or telephone interview setting, interviewers will be accessible to respondents for clarification or definitions, for information about the purpose of the survey, or other questions. Therefore, interviewers need to be able to answer any anticipated respondent questions that may arise. Today our guest lecturer, Kay Sanford, will talk to you about several pieces of documentation that she helped develop to accompany the 2003 post-Hurricane Isabel Rapid Needs Assessment. A part of the interviewer training protocol that Ms. Anjum Hajat discussed in the August Interviewing Techniques PHTIN session includes providing background information about the survey to interviewers. This information serves a dual purpose of informing interviewers and serving as a reference in an interviewer manual.

    119. Supporting Documentation: Interviewers Manual Background Fieldwork Interviewing Techniques Telephone Interviewer Instructions (if applicable) Terms and Definitions Details of the Questionnaire Description of the Survey Area As Ms. Anjum Hajat discussed in the August Interviewing Techniques PHTIN session, interviewer manuals are part of a well-implemented interviewer training program. One of the references cited at the end of this slide set suggests the following components for an interviewer manual: [READ SLIDE] Background: This includes: the purpose of the survey; topics covered; how the survey data will be used; the sample design; and confidentiality information. Fieldwork: This includes: how to locate respondents; scheduling interviews; and how to turn in completed surveys to a supervisor. Interviewing Techniques:This includes: how to read questions; probing (please see the August PHTIN slide set online to review this concept); refusals; entering responses and making notes; and what to do when respondents say, I dont know or provide an otherwise inadequate answer. Telephone Interviewer Instructions: I will discuss some sample components that you could include with telephone interviewer instructions in a few minutes. Terms and Definitions: Defines any potentially esoteric or confusing terms for the benefit of both the interviewer and respondents. Details of the Questionnaire: Sort of a map of the surveys overall layout and sub-sections. Description of the Survey Area: This information for interviewers might include: safety concerns (e.g., road access, flooded areas, etc. in the event of a natural or manmade disaster); maps (perhaps both road maps and topographical maps) and directions to interview sites; and even the demographic composition of neighborhoods. For example, for the post-Hurricane Isabel Rapid Needs Assessment efforts in 2003, some interviewers needed to use a Spanish version of the survey instrument to effectively interview a segment of the sample population. As Ms. Anjum Hajat discussed in the August Interviewing Techniques PHTIN session, interviewer manuals are part of a well-implemented interviewer training program. One of the references cited at the end of this slide set suggests the following components for an interviewer manual: [READ SLIDE] Background: This includes: the purpose of the survey; topics covered; how the survey data will be used; the sample design; and confidentiality information. Fieldwork: This includes: how to locate respondents; scheduling interviews; and how to turn in completed surveys to a supervisor. Interviewing Techniques:This includes: how to read questions; probing (please see the August PHTIN slide set online to review this concept); refusals; entering responses and making notes; and what to do when respondents say, I dont know or provide an otherwise inadequate answer. Telephone Interviewer Instructions: I will discuss some sample components that you could include with telephone interviewer instructions in a few minutes. Terms and Definitions: Defines any potentially esoteric or confusing terms for the benefit of both the interviewer and respondents. Details of the Questionnaire: Sort of a map of the surveys overall layout and sub-sections. Description of the Survey Area: This information for interviewers might include: safety concerns (e.g., road access, flooded areas, etc. in the event of a natural or manmade disaster); maps (perhaps both road maps and topographical maps) and directions to interview sites; and even the demographic composition of neighborhoods. For example, for the post-Hurricane Isabel Rapid Needs Assessment efforts in 2003, some interviewers needed to use a Spanish version of the survey instrument to effectively interview a segment of the sample population.

    120. Supporting Documentation Standard format for interviewer introductions for telephone or face-to-face interviews: Interviewers name Health department (or organization) represented One sentence description of the surveys purpose (e.g., There has been an outbreak of disease X in your county and we are calling you to . . .) A conservative estimate of how long the interview will take When an interviewer approaches someones front door or phones a potential respondents home, the first thing that he (or she) needs to do is introduce him / herself. You might choose to include this introductory information in a narrative form a standardized script at the top of a questionnaire so interviewers can read through it. When an interviewer approaches someones front door or phones a potential respondents home, the first thing that he (or she) needs to do is introduce him / herself. You might choose to include this introductory information in a narrative form a standardized script at the top of a questionnaire so interviewers can read through it.

    121. Supporting Documentation: Interviewer Information Who is the interviewer? Who is sponsoring / implementing the survey? Who is the primary contact? What is the purpose of the survey / study? How will results be used, and how will they help the respondent? On this and the next slide, I will review some pieces of information that you could type up on one page to assist either a telephone or face-to-face interviewer with providing standardized and accurate answers to respondent questions about the surveys implementation. This type of one-page document might be referred to as a help sheet. In the context of outbreak investigations, you will most likely provide information about your local health department in item #2. In the context of outbreak investigations, it should not be difficult to explain the purpose of the survey and the potential benefit to respondents! On this and the next slide, I will review some pieces of information that you could type up on one page to assist either a telephone or face-to-face interviewer with providing standardized and accurate answers to respondent questions about the surveys implementation. This type of one-page document might be referred to as a help sheet. In the context of outbreak investigations, you will most likely provide information about your local health department in item #2. In the context of outbreak investigations, it should not be difficult to explain the purpose of the survey and the potential benefit to respondents!

    122. Supporting Documentation: Interviewer Information How many people will be participating? How did the interviewer obtain the respondents name? Will responses be confidential? Will survey results be available to respondents?

    123. Supporting Documentation: Telephone Interviewer Instructions What information to have available during an interview How to use the telephone system (and the CATI* software, if applicable) How to log calls Here are some suggested pieces of information that you might include in an instructions sheet specifically for telephone interviewers. Here are some suggested pieces of information that you might include in an instructions sheet specifically for telephone interviewers.

    124. Supporting Documentation: Telephone Interviewer Instructions How to handle angry respondents When to seek supervisor assistance What to do once a telephone interview has been completed

    125. Format Guidelines: Self-Administered Questionnaire Lets now consider some formatting strategies that are specific to self-administered questionnaires. . .Lets now consider some formatting strategies that are specific to self-administered questionnaires. . .

    126. Questionnaire Format Self administered questionnaire: Completion of the questionnaire should be self-explanatory; do not use detailed instructions, because not all respondents will read them. Restrict the design to closed-ended questions (e.g., checking a box or circling a response option). Order questions from most topic-related and easy to answer at the beginning, to most likely to be objectionable or of a sensitive nature at the end. Do NOT begin your survey with long, detailed instructions that may not even be needed until later on in the survey. Even when you try your hardest to keep a survey instrument simple, you may still need to incorporate skip patterns (e.g., to direct the respondent to a new question or series of questions if s/he answers no to the first in a series of questions). Do NOT begin your survey with long, detailed instructions that may not even be needed until later on in the survey. Even when you try your hardest to keep a survey instrument simple, you may still need to incorporate skip patterns (e.g., to direct the respondent to a new question or series of questions if s/he answers no to the first in a series of questions).

    127. Questionnaire Format Self administered questionnaire: Strive for consistency in question type and response option formatting whenever possible. Use clean formatting with plenty of white space so you do not overwhelm the respondent with clutter on the page. Use skip patterns only where necessary; use visual cues versus written instructions where needed. The more consistent the question type and tasks expected of the respondent, the less likely he / she is to become confused. 5. Its best to use lines / arrows to direct a respondent around the skipped question(s) versus relying on him / her to read additional instructions. The more consistent the question type and tasks expected of the respondent, the less likely he / she is to become confused. 5. Its best to use lines / arrows to direct a respondent around the skipped question(s) versus relying on him / her to read additional instructions.

    128. Format Guidelines Interviewer versus Self-Administered Questionnaires Lets now review a side-by-side comparison of the distinct formatting guideline differences for interviewer administered versus self-administered questionnaires. . .[SEE NEXT SLIDE] Lets now review a side-by-side comparison of the distinct formatting guideline differences for interviewer administered versus self-administered questionnaires. . .[SEE NEXT SLIDE]

    129. Format Guidelines To review, interviewer administered questionnaires can be used for either telephone or face-to-face interviews, while self-administered questionnaires can be distributed in a clinic waiting room, via mail, or via the Internet (e.g., emailed or posted on a Web site). The table on this slide summarizes some key differences in survey formatting between the two broad categories of questionnaire type. Overall, the two types will probably look and sound different when read aloud, because: Self-administered questionnaires have a visual element Interviewer administered questionnaires have to be understandable from a listeners perspective particularly for telephone interviews. Remember that question stems and response options must be succinct and simple, and to not use too many response categories. The person on the other end of the telephone line must be able to follow and remember everything that the interviewer says. [READ TABLE]To review, interviewer administered questionnaires can be used for either telephone or face-to-face interviews, while self-administered questionnaires can be distributed in a clinic waiting room, via mail, or via the Internet (e.g., emailed or posted on a Web site). The table on this slide summarizes some key differences in survey formatting between the two broad categories of questionnaire type. Overall, the two types will probably look and sound different when read aloud, because: Self-administered questionnaires have a visual element Interviewer administered questionnaires have to be understandable from a listeners perspective particularly for telephone interviews. Remember that question stems and response options must be succinct and simple, and to not use too many response categories. The person on the other end of the telephone line must be able to follow and remember everything that the interviewer says. [READ TABLE]

    130. You are not finished yet. . . Even after you have carefully constructed your survey questions and placed them in a user-friendly order and format on paper, you still have one task remaining: pilot testing your questionnaire. It can be helpful to have standardized questionnaires on hand to use for outbreak investigations whenever possible. One of the benefits of pilot testing a survey instrument is that you will have assessed whether or not the questions mean the same thing to every respondent. One way to field test / pilot test a survey instrument is to solicit the help of colleagues. [SEE NEXT SLIDE FOR DETAILS. . . ] Even after you have carefully constructed your survey questions and placed them in a user-friendly order and format on paper, you still have one task remaining: pilot testing your questionnaire. It can be helpful to have standardized questionnaires on hand to use for outbreak investigations whenever possible. One of the benefits of pilot testing a survey instrument is that you will have assessed whether or not the questions mean the same thing to every respondent. One way to field test / pilot test a survey instrument is to solicit the help of colleagues. [SEE NEXT SLIDE FOR DETAILS. . . ]

    131. Pilot Test Questionnaires Are questions yielding the information that they are supposed to yield? Do respondents understand all wording? Do respondents interpret the questions the same way? Do closed-ended questions have a response option that applies to each respondent? Are skip patterns followed correctly? When you pilot test your questionnaires during field or table top exercises, here are some elements that you can critique: [READ SLIDE] More specifically, if you pilot test a self-administered questionnaire, stick around to watch your colleagues complete the survey. You will then have an opportunity to observe body language Do people need to erase responses and change answers, possibly because they misread instructions? Do people have a confused expression on their face? You may also want to time the completion of the questionnaire particularly if you plan to mail it out to respondents. But its really not a bad idea to time it anyway for telephone interviews so you can inform potential respondents of the time commitment involved in participating in an interview. If you pilot test an interviewer-administered questionnaire for telephone interviews, have two colleagues not only practice the interview (one person serves as interviewer, one as respondent), but sit with backs to each other so they can only listen to what is being said. This puts them in the same position in which they would be as a real respondent during a telephone interview. When you pilot test your questionnaires during field or table top exercises, here are some elements that you can critique: [READ SLIDE] More specifically, if you pilot test a self-administered questionnaire, stick around to watch your colleagues complete the survey. You will then have an opportunity to observe body language Do people need to erase responses and change answers, possibly because they misread instructions? Do people have a confused expression on their face? You may also want to time the completion of the questionnaire particularly if you plan to mail it out to respondents. But its really not a bad idea to time it anyway for telephone interviews so you can inform potential respondents of the time commitment involved in participating in an interview. If you pilot test an interviewer-administered questionnaire for telephone interviews, have two colleagues not only practice the interview (one person serves as interviewer, one as respondent), but sit with backs to each other so they can only listen to what is being said. This puts them in the same position in which they would be as a real respondent during a telephone interview.

    132. Question and Answer Opportunity What questions do you have about questionnaire formatting? What questions do you have about questionnaire formatting?

    133. Guest Expert Lecturer Kay Sanford, MSPH Head, Injury Epidemiology Unit Injury and Violence Prevention Branch NC DHHS Division of Public Health kay.sanford@ncmail.net 919.715.6444 Today our guest expert lecturer, Kay Sanford, is here to talk about her experience in developing a survey instrument and conducting interviewer training for the interviewer administered, face-to-face [door-to-door] during the September 2003 Rapid Needs Assessment response in northeastern North Carolina in the aftermath of Hurricane Isabel. Today our guest expert lecturer, Kay Sanford, is here to talk about her experience in developing a survey instrument and conducting interviewer training for the interviewer administered, face-to-face [door-to-door] during the September 2003 Rapid Needs Assessment response in northeastern North Carolina in the aftermath of Hurricane Isabel.

    134. NC Rapid Needs Assessment: Hurricane Isabel, September 2003 North Carolina, and the rest of the western hemisphere, have experienced the devastating effects of hurricanes for millennia. Today, were going to focus on one, Hurricane Isabel, and one public health component, the designing of surveys that can be used to rapidly assess the public health needs of our state after a natural disaster. North Carolina, and the rest of the western hemisphere, have experienced the devastating effects of hurricanes for millennia. Today, were going to focus on one, Hurricane Isabel, and one public health component, the designing of surveys that can be used to rapidly assess the public health needs of our state after a natural disaster.

    135. Rapid Needs Assessment Background 3rd week September 2003, Hurricane Isabel en route to NC coast (Beaufort County) RNA process had been used in other disasters, e.g., Ankara earthquake 1999; Maine Ice Storm, 1998; Hurricane Opal, 1995, Decision to perform RNA made as Isabel approached the NC coast Last September, almost exactly a year ago, Hurricane Isabel was churning out in the Atlantic. The prediction was it would hit North Carolina, somewhere along the coast. Several days before Isabel was expected to make landfall, the states Public Health Preparedness and Response Team decided the state now had sufficient expertise and staffing to do a public health rapid needs assessment. Part of that process would be to develop the survey instruments. Last September, almost exactly a year ago, Hurricane Isabel was churning out in the Atlantic. The prediction was it would hit North Carolina, somewhere along the coast. Several days before Isabel was expected to make landfall, the states Public Health Preparedness and Response Team decided the state now had sufficient expertise and staffing to do a public health rapid needs assessment. Part of that process would be to develop the survey instruments.

    136. Development of Questionnaires

    137. Who needed the RNA data? In 2003, no mandate for RAPID hurricane data within the state PHP&R - DPH initiative - useful to Division Emergency Management Local emergency operations programs Local Health Departments RNA now written into State Hurricane Plan One of the first factors that have to be taken into consideration when developing a new survey is the target audience, in other words, who will be using the information that is collected. Once the target audience is identified, it is often easier to decide what information needs to be collected. In 2003, there was no mandate for the rapid collection of hurricane data in North Carolina. Post event surveillance had been done for Fran and Floyd, but that had been based on emergency room data that came in slowly over the course of several weeks after the wind and the flooding had subsided. The state Division of Public Health expected a rapid needs assessment would be useful in multiple venues such as the state Division of Emergency Management, local emergency operations programs and local health departments. We were right. The RNA data proved so useful, that its collection is now written into the State Hurricane Plan. But last year, as we sat down to design our surveys, we werent quite sure of our audience. So, our target audience would have to be based in part, on who we could invite to the table to help us select our questions. One of the first factors that have to be taken into consideration when developing a new survey is the target audience, in other words, who will be using the information that is collected. Once the target audience is identified, it is often easier to decide what information needs to be collected. In 2003, there was no mandate for the rapid collection of hurricane data in North Carolina. Post event surveillance had been done for Fran and Floyd, but that had been based on emergency room data that came in slowly over the course of several weeks after the wind and the flooding had subsided. The state Division of Public Health expected a rapid needs assessment would be useful in multiple venues such as the state Division of Emergency Management, local emergency operations programs and local health departments. We were right. The RNA data proved so useful, that its collection is now written into the State Hurricane Plan. But last year, as we sat down to design our surveys, we werent quite sure of our audience. So, our target audience would have to be based in part, on who we could invite to the table to help us select our questions.

    138. Who was involved in developing the RNA Questionnaire? Epidemiology Section Public Health Preparedness & Response Injury and Violence Prevention Branch Division of Environmental Health Division of Mental Health CDC specialists in mass trauma surveillance PHRST members As we had two days in which to create the surveys, we had the luxury of calling in folks with backgrounds in surveillance in environmental health, injury prevention, mental health, and public health from the Raleigh area. CDC sent three seasoned epidemiologists from Atlanta with many years of field experience in collecting data following natural and man-made disasters. We also had the advantage of having several PHRST members join us since they were going to be the ones administering the surveys in the field, it was important that they had input into content and form. Although we didnt realize it until Day 2, someone was conspicuously absent from the table. There is a significant Latino, possibly non-English speaking, population in the coastal region of North Carolina. However, we had not asked anyone fluent in Spanish or the Latino Community Coordinator from the DHHS Office of Communication to work with us from the beginning. Translations of survey instruments were graciously provided at the last minute, but without the benefit of cultural awareness that could have enhanced the quality of the introductory script and the wording of the response categories. As we had two days in which to create the surveys, we had the luxury of calling in folks with backgrounds in surveillance in environmental health, injury prevention, mental health, and public health from the Raleigh area. CDC sent three seasoned epidemiologists from Atlanta with many years of field experience in collecting data following natural and man-made disasters. We also had the advantage of having several PHRST members join us since they were going to be the ones administering the surveys in the field, it was important that they had input into content and form. Although we didnt realize it until Day 2, someone was conspicuously absent from the table. There is a significant Latino, possibly non-English speaking, population in the coastal region of North Carolina. However, we had not asked anyone fluent in Spanish or the Latino Community Coordinator from the DHHS Office of Communication to work with us from the beginning. Translations of survey instruments were graciously provided at the last minute, but without the benefit of cultural awareness that could have enhanced the quality of the introductory script and the wording of the response categories.

    139. How we developed the RNA Questionnaire by committee (1) Selected a TEAM leader Identified examples to serve as prototypes Agreed upon goals of data collection Promoted our individual areas So what we had was survey design by committee. Not an easy task when you have a group of experts, all with strong convictions, and all with limited time. To make the process work, we selected a TEAM leader. Team is capitalized because that is the only way the process could work as a team. We identified and reviewed examples of surveys that had been used in previous RNA exercises. We discussed at length and then agreed on the goals of our data collection; then we re-reviewed the prototype forms and selected the one used for Hurricane Opal as the basis for modification. Everyone provided and defended a list of data items from their own specialties. This took time, but helped to build mutual respect and solidify the team approach for the difficult task of selecting which questions would make the final cut. So what we had was survey design by committee. Not an easy task when you have a group of experts, all with strong convictions, and all with limited time. To make the process work, we selected a TEAM leader. Team is capitalized because that is the only way the process could work as a team. We identified and reviewed examples of surveys that had been used in previous RNA exercises. We discussed at length and then agreed on the goals of our data collection; then we re-reviewed the prototype forms and selected the one used for Hurricane Opal as the basis for modification. Everyone provided and defended a list of data items from their own specialties. This took time, but helped to build mutual respect and solidify the team approach for the difficult task of selecting which questions would make the final cut.

    140. Strategies for Designing the RNA Survey Instrument Limit forms to one-page Number all questions Preference for closed-ended questions Omitted extremely sensitive questions Allowed for Dont Know as a legitimate answer Header completed by interviewer Questions read verbatim by interviewer We had to set some upfront ground rules. We knew the amount of time an interviewer could spend with a respondent would be limited. Although only 210 surveys needed to be completed to complete the sample, we knew the houses would be far apart, and the traveling might be hazardous. Time would be of the essence. So, we agreed that the forms would be limited to one page. To make data entry easier, every question would be numbered. Whenever possible, we would use closed-ended questions with specific response categories. Again to facilitate the speed of data collection and data analysis. Because of the difficulty in administering emotionally charged questions, we decided to omit extremely sensitive questions, such as did any member of this household killed because of the hurricane? It may, or may not, have been a good decision. We carefully allowed for Dont Know as a legitimate response category, as it is never possible for a respondent to answer all questions, and data analysis is greatly enhanced when there is no question between deliberately skipped and unintentionally missed response. We decided that it would be quicker if the header information (mostly demographics of the household and the respondent) was completed by the interviewer and not the respondent. And, last but not least, we agreed that all questions would be carefully scripted and read verbatim by the interviewer to maintain standardization and to reduce interviewer bias. We had to set some upfront ground rules. We knew the amount of time an interviewer could spend with a respondent would be limited. Although only 210 surveys needed to be completed to complete the sample, we knew the houses would be far apart, and the traveling might be hazardous. Time would be of the essence. So, we agreed that the forms would be limited to one page. To make data entry easier, every question would be numbered. Whenever possible, we would use closed-ended questions with specific response categories. Again to facilitate the speed of data collection and data analysis. Because of the difficulty in administering emotionally charged questions, we decided to omit extremely sensitive questions, such as did any member of this household killed because of the hurricane? It may, or may not, have been a good decision. We carefully allowed for Dont Know as a legitimate response category, as it is never possible for a respondent to answer all questions, and data analysis is greatly enhanced when there is no question between deliberately skipped and unintentionally missed response. We decided that it would be quicker if the header information (mostly demographics of the household and the respondent) was completed by the interviewer and not the respondent. And, last but not least, we agreed that all questions would be carefully scripted and read verbatim by the interviewer to maintain standardization and to reduce interviewer bias.

    141. How we developed the RNA Questionnaire by committee (2) Wrote / revised individual questions Prepared QxQ instructions & procedures COMROMISED on questions, definitions, and protocol. With the game plan fully agreed upon, the team leader was selected to write and revise the individual questions. During Day 1, multiple iterations were circulated before the final version was accepted. The team leader also prepared drafts of the Question by Question instructions, after consulting with each of the area specialists. The penultimate version on Day 2 was based on COMPROMISE. We decided that this version was penultimate because it still had not been pilot tested. Because of the limited time, pilot testing would have to be the response of the PHRST members who practiced reading and entering data during the first day of training. If a question, the definitions or the response categories were too cumbersome, then one last revision could be made before the second day of training. This is and wasnt optimal pilot testing, but it was better than nothing. With the game plan fully agreed upon, the team leader was selected to write and revise the individual questions. During Day 1, multiple iterations were circulated before the final version was accepted. The team leader also prepared drafts of the Question by Question instructions, after consulting with each of the area specialists. The penultimate version on Day 2 was based on COMPROMISE. We decided that this version was penultimate because it still had not been pilot tested. Because of the limited time, pilot testing would have to be the response of the PHRST members who practiced reading and entering data during the first day of training. If a question, the definitions or the response categories were too cumbersome, then one last revision could be made before the second day of training. This is and wasnt optimal pilot testing, but it was better than nothing.

    142. How many questionnaires were needed? 1. Tracking Form 2. Survey Form 3. Referral Form Although we talk about THE survey, it is a misnomer. The data collection required three forms: the tracking form; the survey form, and the referral form. Although we talk about THE survey, it is a misnomer. The data collection required three forms: the tracking form; the survey form, and the referral form.

    143. Tracking Form This is a snapshot of the upper third of the Tracking Form. Well go over it in detail during the next couple of slides. A tracking form is necessary in order to document the success (or failure) of the sample, and the response rates. It orients the survey to time and space. To spend things up, it was generally completed by the team member who wasnt administering the survey form. This is a snapshot of the upper third of the Tracking Form. Well go over it in detail during the next couple of slides. A tracking form is necessary in order to document the success (or failure) of the sample, and the response rates. It orients the survey to time and space. To spend things up, it was generally completed by the team member who wasnt administering the survey form.

    144. What data needed to be collected for Hurricane Isabel? TRACKING FORM: Sampling information county cluster number block number Interview team Sample sequence No., Address Date, Time The recorder documented the sampling information, e.g., the county, census cluster number and block number, that were provided to the survey team prior to deployment. The name of the Interview Team was also recorded, as was the specific information that would identify the sequence number, address of the household that was to be interviewed, and the date and the time the interview began. The recorder documented the sampling information, e.g., the county, census cluster number and block number, that were provided to the survey team prior to deployment. The name of the Interview Team was also recorded, as was the specific information that would identify the sequence number, address of the household that was to be interviewed, and the date and the time the interview began.

    145. What data needed to be collected for Hurricane Isabel? TRACKING FORM: Response status Interview Conducted Interview started but not completed Interview refused Language barrier Residence not occupied Evacuated Vacant Seasonal occupant Dwelling destroyed Reason Unknown Other (specify) Equally important, the recorder documented whether or not the interview took place, and if it didnt, why the interview could not be conducted or couldnt be completed. This establishes the basis for calculating the response rate, and for evaluating the effectiveness of sampling procedures and the deployment protocol. Equally important, the recorder documented whether or not the interview took place, and if it didnt, why the interview could not be conducted or couldnt be completed. This establishes the basis for calculating the response rate, and for evaluating the effectiveness of sampling procedures and the deployment protocol.

    146. Survey Instrument We show the Survey Form to you, not to illustrate how not to make a slide, but to indicate that the restrictions for a one-page form resulted in small print and minimal margins. There are actually 3 more questions at the foot of the page that could not be captured on this slide. Starting with the next slide, well begin looking at the types of questions that the Team agreed to include. We show the Survey Form to you, not to illustrate how not to make a slide, but to indicate that the restrictions for a one-page form resulted in small print and minimal margins. There are actually 3 more questions at the foot of the page that could not be captured on this slide. Starting with the next slide, well begin looking at the types of questions that the Team agreed to include.

    147. What data needed to be collected for Hurricane Isabel? The SURVEY FORM Damage to homes Availability of usable utilities Incidence of hurricane related illness / injury Access to medical care Access to food and water Self-identified immediate needs The Team decided there should be 6 major categories of data collected from the RNA survey, all of which could be the basis for a public health referral: 1. Damage to homes from the hurricane. 2. The availability of usable utilities in the home, including water, electricity, gas, toilet, telephone, radio, a generator or the use of charcoal. 3. The incidence of hurricane related illness or injury. 4. Access to medical care or medications, regardless of the source of illness or injury. 5. Access to food and water. 6. And any other self-identified immediate needs of the respondent. And, in the best of all possible worlds, each area would be covered by one or two simply worded questions with standardized response categories. Well discuss these more in a few minutes when we talk about interview training. There is only one question that allows a narrative response. Free text responses are difficult to record, especially when data collection is done electronically. Free text fields are also more difficult to analyze than categorical responses, especially when time is of the essence. We were sure that in order for the RNA to be perceived by the public health and emergency services communities as an essential service, the results would have to be available the morning after data were collected. The Team decided there should be 6 major categories of data collected from the RNA survey, all of which could be the basis for a public health referral: 1. Damage to homes from the hurricane. 2. The availability of usable utilities in the home, including water, electricity, gas, toilet, telephone, radio, a generator or the use of charcoal. 3. The incidence of hurricane related illness or injury. 4. Access to medical care or medications, regardless of the source of illness or injury. 5. Access to food and water. 6. And any other self-identified immediate needs of the respondent. And, in the best of all possible worlds, each area would be covered by one or two simply worded questions with standardized response categories. Well discuss these more in a few minutes when we talk about interview training. There is only one question that allows a narrative response. Free text responses are difficult to record, especially when data collection is done electronically. Free text fields are also more difficult to analyze than categorical responses, especially when time is of the essence. We were sure that in order for the RNA to be perceived by the public health and emergency services communities as an essential service, the results would have to be available the morning after data were collected.

    148. Respondent Referral Form The goal of the RNA was not only the collection of data, but the provision of reassurance, in part through the referral of households with immediate needs to the local emergency management and public health departments. In order to match referrals to surveys, sampling and ID data were recorded at the top of the form. The name, address and contact information of the person in the household who would serve as the contact person, not necessarily the respondent, was recorded. Each PHRST Unit documented whether or not assistance was provided for example, each vehicle had a small supply of bottled water. Packs of environmental hazards and injury prevention literature had been prepared for each Unit to distribute as needed. If a referral was needed, the source of assistance and who made the call was documented on the form. A narrative description of the assistance was recorded by one of the Unit members either during the visit or afterwards. The goal of the RNA was not only the collection of data, but the provision of reassurance, in part through the referral of households with immediate needs to the local emergency management and public health departments. In order to match referrals to surveys, sampling and ID data were recorded at the top of the form. The name, address and contact information of the person in the household who would serve as the contact person, not necessarily the respondent, was recorded. Each PHRST Unit documented whether or not assistance was provided for example, each vehicle had a small supply of bottled water. Packs of environmental hazards and injury prevention literature had been prepared for each Unit to distribute as needed. If a referral was needed, the source of assistance and who made the call was documented on the form. A narrative description of the assistance was recorded by one of the Unit members either during the visit or afterwards.

    149. Strategies for collecting the data Paper form - delayed data entry in the field; more delayed data transmission Access database; Epi Info for data analysis Electronic form on laptop/iPAQ - direct data entry; delayed data transmission Hurricane Isabel: Paper form w/ Delayed DE-Delayed DT Hurricane Charley: Direct DE w/ Delayed DT Next time: Direct DE/Direct DT The Team spent much time discussing how the data would be collected. Each method has its advantages and disadvantages. Paper forms are often easy to use in the field, but require delayed data entry, and delayed data transmission. It is a well-established principle in epidemiology that every time data are recorded and re-recorded, the potential for error increases. An ACCESS database was selected for data management and was installed and programmed on each of the laptops assigned to the PHRST units. The ideal was to enter the data directly into the laptop and to use the paper forms as backup. This proved to be too cumbersome in the field, and the Teams used the paper form and then entered the survey data into the laptop at the end of the day. After the data had been entered at the end of two very long days, the management team analyzed it during the wee hours of the night for release the following morning. When Hurricane Charley went through NC several weeks ago, PHRST 5 was able to use handheld computers for survey data entry and for GIS coding. Its a tradeoff. The hand-helds are expensive, and direct data entry onto the iPAQ took more time than transcription onto a paper form. The advantage was that once the data were entered, they were there, without the need for re-entry resulting in an ultimate saving of time and increased accuracy. The Team spent much time discussing how the data would be collected. Each method has its advantages and disadvantages. Paper forms are often easy to use in the field, but require delayed data entry, and delayed data transmission. It is a well-established principle in epidemiology that every time data are recorded and re-recorded, the potential for error increases. An ACCESS database was selected for data management and was installed and programmed on each of the laptops assigned to the PHRST units. The ideal was to enter the data directly into the laptop and to use the paper forms as backup. This proved to be too cumbersome in the field, and the Teams used the paper form and then entered the survey data into the laptop at the end of the day. After the data had been entered at the end of two very long days, the management team analyzed it during the wee hours of the night for release the following morning. When Hurricane Charley went through NC several weeks ago, PHRST 5 was able to use handheld computers for survey data entry and for GIS coding. Its a tradeoff. The hand-helds are expensive, and direct data entry onto the iPAQ took more time than transcription onto a paper form. The advantage was that once the data were entered, they were there, without the need for re-entry resulting in an ultimate saving of time and increased accuracy.

    150. What else was needed? Introductory script English and Spanish Question by Question instructions Standard in formal epidemiologic research Often only form of procedure manual when preparation time limited RNA situations Often deciding factor in complicated analysis Just having the tracking and survey instruments isnt enough. The Team prepared introductory scripts in English, that was subsequently translated into Spanish. Question by Question instructions (QxQs) were prepared for each item on the survey. QxQs are standard in formal epidemiologic studies in addition to full-scale procedure manuals. For Hurricane Isabel, this one page document served as the procedures manual for the PHRST Units in the field. Just having the tracking and survey instruments isnt enough. The Team prepared introductory scripts in English, that was subsequently translated into Spanish. Question by Question instructions (QxQs) were prepared for each item on the survey. QxQs are standard in formal epidemiologic studies in addition to full-scale procedure manuals. For Hurricane Isabel, this one page document served as the procedures manual for the PHRST Units in the field.

    151. Scripting: English Introductory Script The script is deceptively simple, and reads The script is deceptively simple, and reads

    152. Introductory Script Identifies and legitimizes the interviewer States reasons for conducting the survey Safe living conditions Need within household for medical care Assures respondent of the confidentiality of responses Selects the household respondent The Introductory Script is often the basis for the first impression, and the deciding factor to the first point of contact in the household on whether or not a survey will be allowed. In addition to the initials on the PHRST vehicle and the PHRST clothing, the script identifies and legitimizes the interviewer. It immediately states the reasons for conducting the survey as the health and safety of the household. It assures the first contact person that whatever information is given will be confidential, and the introductory script selects the household respondent. The Introductory Script is often the basis for the first impression, and the deciding factor to the first point of contact in the household on whether or not a survey will be allowed. In addition to the initials on the PHRST vehicle and the PHRST clothing, the script identifies and legitimizes the interviewer. It immediately states the reasons for conducting the survey as the health and safety of the household. It assures the first contact person that whatever information is given will be confidential, and the introductory script selects the household respondent.

    153. Question by Question Instructions Refresher guide to interviewers after training Standardized definitions Instructions for probing Instructions for recording or reading response categories Instructions for when to refer The QxQ instructions are the most complicated component of writing questionnaires and serve as the basis for the initial training and as a refresher guide to the interviewers after training. They provide the standardized definitions of all terms that are used in the questions, so that variation among responses can be ascribed to real differences in individual households, and not differences in interviewer interpretation of what was asked. The QxQs often provide the wording to use for probes. Although the best probes are often silence or the quiet repetition of the question, a standardized definition can be used to explain what the critical terms in a question mean. QxQs provide the instructions for recording or reading response categories. Forms are deceptively simple; it is often important that the interviewers memorize the instructions for recording answers that are different than the usual Yes, No and Dont Know categories. Lastly, the QxQs for this RNA needed to indicate when a referral for services was required. Well briefly review the parts of the QxQ instructions over the next few slides. There isnt enough time to talk about each of these in detail. However, a copy of this document can be made available to you, if youd like to send me an e-mail. The QxQ instructions are the most complicated component of writing questionnaires and serve as the basis for the initial training and as a refresher guide to the interviewers after training. They provide the standardized definitions of all terms that are used in the questions, so that variation among responses can be ascribed to real differences in individual households, and not differences in interviewer interpretation of what was asked. The QxQs often provide the wording to use for probes. Although the best probes are often silence or the quiet repetition of the question, a standardized definition can be used to explain what the critical terms in a question mean. QxQs provide the instructions for recording or reading response categories. Forms are deceptively simple; it is often important that the interviewers memorize the instructions for recording answers that are different than the usual Yes, No and Dont Know categories. Lastly, the QxQs for this RNA needed to indicate when a referral for services was required. Well briefly review the parts of the QxQ instructions over the next few slides. There isnt enough time to talk about each of these in detail. However, a copy of this document can be made available to you, if youd like to send me an e-mail.

    154. Survey Instrument Header Date Interviewer Sampling information Respondent information Like the Tracking and Referral forms, the Survey form begins with information that documents when the survey is being done, with whom, by whom, and the sampling information. Like the Tracking and Referral forms, the Survey form begins with information that documents when the survey is being done, with whom, by whom, and the sampling information.

    155. Survey Instrument Household information type usual number of residents residents during the storm persons at increased risk The survey begins by ascertaining information about the household, the number of people in the household who experienced the hurricane, and the number of people who could have been, by age alone, at increased risk. The survey begins by ascertaining information about the household, the number of people in the household who experienced the hurricane, and the number of people who could have been, by age alone, at increased risk.

    156. Survey Instrument Damage to house from the hurricane habitability of structure tarp needed amount of flood water in the house missing: amount of mud or debris The next series of questions concern damage to the house from the hurricane; whether the respondent considers the house to be habitable, whether a tarp is needed for the roof, and whether there was any flood water in the house. A question on whether there was any household damage due to mud or debris, either from flooding or landslides, is currently being discussed by the PHP&R team as we prepare for the next hurricane. The next series of questions concern damage to the house from the hurricane; whether the respondent considers the house to be habitable, whether a tarp is needed for the roof, and whether there was any flood water in the house. A question on whether there was any household damage due to mud or debris, either from flooding or landslides, is currently being discussed by the PHP&R team as we prepare for the next hurricane.

    157. Survey Instrument Current status of household utilities running water power toilet telephone radio use of generator use of charcoal Eleven questions were included in the Survey on the current status of the utilities in the household. The use of several of substitutes for non-working utilities (unsanitary toilets, or the unsafe placement of generators and charcoal burners) were known by the Team to create a public health emergency. Special instructions were included in the QxQs on how or when to provide a safety message or to refer the household for assistance. Eleven questions were included in the Survey on the current status of the utilities in the household. The use of several of substitutes for non-working utilities (unsanitary toilets, or the unsafe placement of generators and charcoal burners) were known by the Team to create a public health emergency. Special instructions were included in the QxQs on how or when to provide a safety message or to refer the household for assistance.

    158. Interviewer Training In addition to preparing the survey and supporting forms and instructions, the Team developed the materials for interviewer training. Interviewer training was held twice in Raleigh in the Cooper Building, on the floor below the Public Health Command Center. Each session was 2 hours long on day one for all of the PHRST Units that had arrived and on day 2 for the latecomers and as a refreshers for those who had attended the day before. Training on safety and logistics was provided by CDC staff from Atlanta and several PHP&R staff, all of whom have years of field experience. Training on interviewing techniques was provided by the Questionnaire Design Team Leader from the Injury and Violence Prevention Branch. In addition to preparing the survey and supporting forms and instructions, the Team developed the materials for interviewer training. Interviewer training was held twice in Raleigh in the Cooper Building, on the floor below the Public Health Command Center. Each session was 2 hours long on day one for all of the PHRST Units that had arrived and on day 2 for the latecomers and as a refreshers for those who had attended the day before. Training on safety and logistics was provided by CDC staff from Atlanta and several PHP&R staff, all of whom have years of field experience. Training on interviewing techniques was provided by the Questionnaire Design Team Leader from the Injury and Violence Prevention Branch.

    159. Interviewer Training Recap from August Lecture As we were reminded during the August set of lectures, training is never optional. Training emphasizes that the safety of the interviewer is always the first consideration, the accuracy and completeness of the data collection is the second consideration, and that none of it is possible without an esprit de corps. Those out in the field, and those back at the command centers are a team. If at all possible, and for Hurricane Isabel, it was possible, trainings must be interactive. Those who are doing the training and those who are being trained must be able to communicate. This was particularly important because the training served as the pilot testing. Questions and instructions that were unintelligible in the comfort of the 3rd floor training room would be useless in the field. There is no better way to assess the quality of a survey than by practicing. All field team members were given time during the training to read the surveys out loud and to record the responses on a practice form. Field team members were encouraged to read the QxQs out loud on the drive to the coast. As we were reminded during the August set of lectures, training is never optional. Training emphasizes that the safety of the interviewer is always the first consideration, the accuracy and completeness of the data collection is the second consideration, and that none of it is possible without an esprit de corps. Those out in the field, and those back at the command centers are a team. If at all possible, and for Hurricane Isabel, it was possible, trainings must be interactive. Those who are doing the training and those who are being trained must be able to communicate. This was particularly important because the training served as the pilot testing. Questions and instructions that were unintelligible in the comfort of the 3rd floor training room would be useless in the field. There is no better way to assess the quality of a survey than by practicing. All field team members were given time during the training to read the surveys out loud and to record the responses on a practice form. Field team members were encouraged to read the QxQs out loud on the drive to the coast.

    160. Interviewer Training Planning and implementation logistics Explanatory notes for interviewers Scripting Terms defined Instructions for probing Rapport and feedback Instructions for disaster relief referrals Other. . . The topics that were covered at the Interviewer training included the benefits of scripting and the reasons for not deviating from the written word. This applies not only to the first introduction but to every question in the surveys. Well look at some examples of how terms were defined in the QxQs over the course of the next several slides. During the interviewer training, every question was discussed, and all questions from the field interviewers were resolved and incorporated into the final version of the survey form and the QxQ instructions. Much of the training dealt with how to probe when a respondent cannot provide an answer, gives an incomplete answer or an irrelevant response to the question. As a general rule, the best probe is several extra seconds of silence. It is always difficult to know just how much information the respondent has to process before an answer can be given. The second best probe is to simply repeat the question, often by placing the onus of miscommunication on oneself. I may not have read the question well, let me repeat it for you Without leading the respondent into the a more precise response, all of the response categories can be read, such as would you say there was none or minimal damage to the house, damage but the house is habitable, or damage and the house is uninhabitable? Rapport and feedback techniques were reviewed and will be discussed in greater detail on the next slide. Examples of feedback techniques were discussed, emphasizing that approval should be made for responding to a question and not for the behavior that is described by the respondent. And the protocol and timing for referring the household for assistance was discussed, and ways in which it should be done after the interview so as not to interrupt the flow of the questionnaire. The topics that were covered at the Interviewer training included the benefits of scripting and the reasons for not deviating from the written word. This applies not only to the first introduction but to every question in the surveys. Well look at some examples of how terms were defined in the QxQs over the course of the next several slides. During the interviewer training, every question was discussed, and all questions from the field interviewers were resolved and incorporated into the final version of the survey form and the QxQ instructions. Much of the training dealt with how to probe when a respondent cannot provide an answer, gives an incomplete answer or an irrelevant response to the question. As a general rule, the best probe is several extra seconds of silence. It is always difficult to know just how much information the respondent has to process before an answer can be given. The second best probe is to simply repeat the question, often by placing the onus of miscommunication on oneself. I may not have read the question well, let me repeat it for you Without leading the respondent into the a more precise response, all of the response categories can be read, such as would you say there was none or minimal damage to the house, damage but the house is habitable, or damage and the house is uninhabitable? Rapport and feedback techniques were reviewed and will be discussed in greater detail on the next slide. Examples of feedback techniques were discussed, emphasizing that approval should be made for responding to a question and not for the behavior that is described by the respondent. And the protocol and timing for referring the household for assistance was discussed, and ways in which it should be done after the interview so as not to interrupt the flow of the questionnaire.

    161. Interviewer Training The complexities of providing a distressed family reassurance and concomitantly serving as a public health interviewer were reviewed. The perfect interviewer establishes professional rapport by being nonjudgmental, noncommittal and objective. This is done through body language, the use of probes, the types of feedback given the respondent, and the accuracy of the recording of responses. The interviewer training included all of the excellent examples of appropriate demeanor, probes, feedback and interviewer bias were provided in the first set of lectures in August, and should be reviewed whenever local training is developed. The complexities of providing a distressed family reassurance and concomitantly serving as a public health interviewer were reviewed. The perfect interviewer establishes professional rapport by being nonjudgmental, noncommittal and objective. This is done through body language, the use of probes, the types of feedback given the respondent, and the accuracy of the recording of responses. The interviewer training included all of the excellent examples of appropriate demeanor, probes, feedback and interviewer bias were provided in the first set of lectures in August, and should be reviewed whenever local training is developed.

    162. Interviewer Instructions This slide is to show the density and the format of the QxQ instructions. You are not expected to read this. But from a quick look, you can see that every question on the RNA Survey form is addressed. The question on the form is highlighted in bold. Well look at several examples over the next several slides. This slide is to show the density and the format of the QxQ instructions. You are not expected to read this. But from a quick look, you can see that every question on the RNA Survey form is addressed. The question on the form is highlighted in bold. Well look at several examples over the next several slides.

    163. Interviewer Training Terms Defined in QxQs 10. [INTERVIEWER] What type of dwelling is this? ___ 1= single family 2= mobile home 3= 2-5 family 4= 6 or more family 5= other QxQ: Selection based on type of housing, not the number of families living in the premises Well start with question 10. This is one of the few questions that is not asked specifically of the respondent. Therefore, it begins with the word INTERVIEWER in brackets. And just so that there is standardization for the response, a definition of what type of dwelling means is provided to the interviewer. Well start with question 10. This is one of the few questions that is not asked specifically of the respondent. Therefore, it begins with the word INTERVIEWER in brackets. And just so that there is standardization for the response, a definition of what type of dwelling means is provided to the interviewer.

    164. Interviewer Training Terms Defined in QxQs 11. How many people lived in this residence before the hurricane? ____ QxQ: Enter the number of people who routinely slept and ate at least one of their meals at this residence prior to 9/18/03, e.g., exclude students or military personnel living away from home, overnight guests, etc. Enter 99 for Dont Know. At issue: simple question; complex definition; probing likely; provision for Dont Know. This question is addressed to the respondent. The question is simple. The definition of the time frame, and who to count is complex, but is only provided if the respondent asks for more clarification. This is a constant dilemma in designing questions. If the desire is to keep the question simple, how much information should be included, and how often should clarification be given to the respondent. It is also essential to provide the recorder with a mechanism for indicating the respondent couldnt/wouldnt given an answer, but the question was not unintentionally skipped. This question is addressed to the respondent. The question is simple. The definition of the time frame, and who to count is complex, but is only provided if the respondent asks for more clarification. This is a constant dilemma in designing questions. If the desire is to keep the question simple, how much information should be included, and how often should clarification be given to the respondent. It is also essential to provide the recorder with a mechanism for indicating the respondent couldnt/wouldnt given an answer, but the question was not unintentionally skipped.

    165. Interviewer Training Terms Defined in QxQs 13. How much external damage was there from the hurricane to the residence? ___ 1= none or minimal 2= damaged, but habitable 3= damaged, but uninhabitable At issue: categories are read aloud; habitable means safe to live in; based on respondents answer; skip pattern. The QxQs provide a list of the response categories and defines the term habitable as safe to live in. The interviewer is instructed that the response categories are read aloud, but the selection is based on the respondents perception. The rules for the skip pattern are listed. The following question (Item 14) about the need for a tarp is only asked if the response to this question is that the house is damaged. The QxQs provide a list of the response categories and defines the term habitable as safe to live in. The interviewer is instructed that the response categories are read aloud, but the selection is based on the respondents perception. The rules for the skip pattern are listed. The following question (Item 14) about the need for a tarp is only asked if the response to this question is that the house is damaged.

    166. Interviewer Training Terms Defined in QxQs 16.e Does the indoor toilet work? ___ 1= yes 2= no 3= never had 9= Dont know Enter YES (1) even if waste material can be flushed by pouring water into commode. Enter Never Had (3) when the residences does not have any kind of indoor standard toilet. At issue: complex definitions; skip pattern; potential for referral. Many of the questions on the status of household utilities only have three response categories: Yes, No and Dont Know. However, when a utility isnt working, other options needed to be offered. For example, the response categories to the question Does the toilet work? are deceptively complex. The No response, to be useful, has to be distinguished from the lack of the utility. The Yes response to the working status of the toilet can include both the operation of the regular flushing mechanism or manual flushing by pouring water in the commode. A positive response results in a skip pattern. A negative response should be flagged for referral. These are just examples. The logistics of each question had to be carefully worked out and tested prior to releasing a survey form for implementation. Although interviewer training is not the optimal venue in which to pilot a new form, it provided the field team with an opportunity to appreciate the complexities of creating a survey, and the opportunity to feel a certain level of ownership of and commitment to the instrument. Many of the questions on the status of household utilities only have three response categories: Yes, No and Dont Know. However, when a utility isnt working, other options needed to be offered. For example, the response categories to the question Does the toilet work? are deceptively complex. The No response, to be useful, has to be distinguished from the lack of the utility. The Yes response to the working status of the toilet can include both the operation of the regular flushing mechanism or manual flushing by pouring water in the commode. A positive response results in a skip pattern. A negative response should be flagged for referral. These are just examples. The logistics of each question had to be carefully worked out and tested prior to releasing a survey form for implementation. Although interviewer training is not the optimal venue in which to pilot a new form, it provided the field team with an opportunity to appreciate the complexities of creating a survey, and the opportunity to feel a certain level of ownership of and commitment to the instrument.

    167. Lessons Learned Developing the questions for the three survey forms was not error free. There were many lessons learned. Developing the questions for the three survey forms was not error free. There were many lessons learned.

    168. Lessons Learned (1) Include representative from Latino (minority) community from the beginning Cultural sensitivity / correctness Translation of forms, scripts Include question on the presence of mud or debris in survey questionnaire Appropriate for mountains and flatlands Ive mentioned all of these at some point, but they bear repeating. First, it is vital to have people who represent all members of the community in which the survey is to be conducted. This is particularly important if there is a substantial minority population within that community. As we were reminded at the 11th hour, there is considerably more to creating a form for a special population, for example a Spanish form for the local Latino community, than just providing an accurate translation. The wording, the introductions, the location and mechanism of conducting the interview can vary by culture. Any slip can result in the loss of valuable information and the perceived or actual disenfranchisement of the minority population. The addition of even a single question to the form requires reformatting of the paper and electronic versions of the form and the database. All of it costs time and money. At the heart of the debate, who would use this information and what would we do with it, if we had it? We did not include a question(s) on damage due to mud or debris. This omission is currently being discussed as we await Hurricane Ivan, the next hurricane predicted to drench the already rain-soaked mountains, and as we assess the damage from Hurricane Frances. Ive mentioned all of these at some point, but they bear repeating. First, it is vital to have people who represent all members of the community in which the survey is to be conducted. This is particularly important if there is a substantial minority population within that community. As we were reminded at the 11th hour, there is considerably more to creating a form for a special population, for example a Spanish form for the local Latino community, than just providing an accurate translation. The wording, the introductions, the location and mechanism of conducting the interview can vary by culture. Any slip can result in the loss of valuable information and the perceived or actual disenfranchisement of the minority population. The addition of even a single question to the form requires reformatting of the paper and electronic versions of the form and the database. All of it costs time and money. At the heart of the debate, who would use this information and what would we do with it, if we had it? We did not include a question(s) on damage due to mud or debris. This omission is currently being discussed as we await Hurricane Ivan, the next hurricane predicted to drench the already rain-soaked mountains, and as we assess the damage from Hurricane Frances.

    169. Lessons Learned (2) Discuss at outset need for including questions on current conditions versus risk factors for potential problems Landslides often occur days after hurricane Floods often occur days or weeks after event The addition of a question on damage due to mud and debris also suggests that there are other issues that have yet been discussed. Do we focus on current damage at the time of the rapid needs assessment, or should we look for extant risk factors that predispose a household/community to future problems. For example, there are many established risk factors for floods and landslides. The damage from these often dont occur for days or weeks after an event; however, if at risk status can be identified, preventative measures could be taken. Another lesson learned was how, or in this case perhaps, how not to assess mental trauma after a natural disaster. Data from Hurricane Isabel indicated that 30% of the households. Data from terrorist events have suggested that 90% or more people closely associated with such an event experience some form of emotional trauma. However, almost all recover. The challenge is how to identify those whose mental trauma will not resolve over time. One suggestion this year has been to use a more emotionally charged question a strategy originally rejected by the questionnaire design team, but one that should be reconsidered. I would be more than happy to answer any questions, either now if time permits, or later. My contact information is on the first slide. The appendices provide copies of the forms in Spanish. All of the survey instruments and scripts are available. The addition of a question on damage due to mud and debris also suggests that there are other issues that have yet been discussed. Do we focus on current damage at the time of the rapid needs assessment, or should we look for extant risk factors that predispose a household/community to future problems. For example, there are many established risk factors for floods and landslides. The damage from these often dont occur for days or weeks after an event; however, if at risk status can be identified, preventative measures could be taken. Another lesson learned was how, or in this case perhaps, how not to assess mental trauma after a natural disaster. Data from Hurricane Isabel indicated that 30% of the households. Data from terrorist events have suggested that 90% or more people closely associated with such an event experience some form of emotional trauma. However, almost all recover. The challenge is how to identify those whose mental trauma will not resolve over time. One suggestion this year has been to use a more emotionally charged question a strategy originally rejected by the questionnaire design team, but one that should be reconsidered. I would be more than happy to answer any questions, either now if time permits, or later. My contact information is on the first slide. The appendices provide copies of the forms in Spanish. All of the survey instruments and scripts are available.

    170. Lessons Learned (3) Emotional / mental stress is frequent after mass trauma how to craft questions that will identify stress that is not easily abated Cost-benefit of asking about highly emotionally charged events, e.g., death of family member(s)

    171. Appendix

    172. Spanish Introductory Script

    173. Spanish RNA Survey Instrument

    174. Session Summary Lets review the main points of this session on questionnaire design that were presented to you today.Lets review the main points of this session on questionnaire design that were presented to you today.

    175. Session Summary With an understanding of good questionnaire design principles, you will ask only about what you need to meet your research objectives. In preparation for questionnaire design, you should: have clear research objectives; list variables to be measured; identify an analysis plan; and consider cost and logistical aspects.

    176. Session Summary Use a hypothesis generating questionnaire to explore all potential sources of infection, but with a small number of cases and no comparison group. A hypothesis testing questionnaire literally tests a research hypothesis based on data collected via a hypothesis generating questionnaire. Include both cases and controls in the survey sample population.

    177. Session Summary The three broad question types are: closed-ended; open-ended; and fill-in-the-blank. You may use only one or a combination of all, depending on the purpose of your survey. Key elements of question design are: reliability; validity; specificity; simplicity; asking only one question; mutually exclusive answer choices; providing date / time references; and using response options that parallel existing data sources.

    178. Session Summary A well-formatted questionnaire minimizes respondent burden by: using question order, spacing, distinctive font, symbols, and simple, short instructions to facilitate navigation; and using customized components for interviewer administered versus self-administered settings. Question type and response option formatting impact how you can design an on-screen data entry form and / or analyze variables in your software program.

    179. References and Resources American Statistical Association (1999). What Is a Survey? Designing a Questionnaire. Alexandria, VA: Section on Survey Research Methods. American Statistical Association (1997). What Is a Survey? More About Mail Surveys. Alexandria, VA: Section on Survey Research Methods, American Statistical Association. Centers for Disease Control and Prevention, Food borne and Diarrheal Diseases Branch, Food borne Outbreak Response and Surveillance Unit (2003). Outbreak Investigation Toolkit: Standard Questionnaire. Online resource: http://www.cdc.gov/foodborneoutbreaks/standard_questionnaire.htm Data Skills Online web site, Office of Continuing Education, Institute for Public Health, UNC Chapel Hill. An Overview of Primary Data Collection Instruments and Designing Questionnaires online self-instructional tools. http://www.sph.unc.edu/toolbox/.

    180. References and Resources Centers for Disease Control and Prevention. Standard Food Borne Disease Outbreak Questionnaire. (PDF file) at: http://www.cdc.gov/foodborneoutbreaks/question/standard_questionnaire.pdf Fowler, F.J. (1993). Survey Research Methods: Second Edition. Sage Publications: Newbury Park. Reingold, A. (1998). Outbreak Investigations A Perspective. Emerging Infectious Diseases. Vol. 4 No. 1: January March 1998.

    181. References and Resources Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. John Wiley & Sons, Inc.: New York. Stehr-Green, J.K. (2002). A Multi-state Outbreak of E. coli 0157:H7 Infection: Case Study Instructors Guide. Atlanta, GA: U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention. http://www.phppo.cdc.gov/phtn/casestudies/classroom/ecoli.htm

    182. References and Resources Stehr-Green, J. and Stehr-Green, P. (2004). Hypothesis Generating Interviews. Module 3 of a Field Epidemiology Methods course being developed in the NC Center for Public Health Preparedness, UNC Chapel Hill. Torok, M. (2004). FOCUS on Field Epidemiology. Case Finding and Line Listing: A Guide for Investigators. Volume 1, Issue 4. NC Center for Public Health Preparedness. Wiggins, B. and Deeb-Sossa, N. (2000). Conducting Telephone Surveys. Chapel Hill, NC: Odum Institute for Research in Social Science.

    183. Slides from Todays Session Following this program, please visit one of the web sites below to access and download a copy of todays slides: NCCPHP Training web site: http://www.sph.unc.edu/nccphp/training/index.html North Carolina Division of Public Health, Office of Public Health Preparedness and Response http://www.epi.state.nc.us/epi/phpr/

    184. Next Session October 12th 10:00 a.m. - Noon Topic: Analyzing Data

    185. Site Sign-in Sheet Please mail or fax your sites sign-in sheet to: Linda White NC Office of Public Health Preparedness and Response Cooper Building 1902 Mail Service Center Raleigh, NC 27699 FAX: (919) 715 - 2246

More Related