1 / 22

DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT

DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT. Kathleen A. Martin, D.P.E. Office of Institutional Research & Assessment Bucknell University. POINT OF CLARIFICATION. Terms “survey” and “questionnaire” are often used interchangeably

Samuel
Télécharger la présentation

DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT Kathleen A. Martin, D.P.E. Office of Institutional Research & Assessment Bucknell University

  2. POINT OF CLARIFICATION • Terms “survey” and “questionnaire” are often used interchangeably • “Survey” refers to a descriptive research technique or methodology (also includes interviews, and focus groups) • “Questionnaire” refers to a paper and pencil instrument through which information is obtained • For the purposes of outcomes assessment, questionnaires are frequently developed to survey a population

  3. QUESTIONNAIRES & ASSESSMENT • Useful for surveying groups of any size • Typically designed to determine opinions, attitudes or present practices • Limitation: rely on self report information v. observed behavior • Challenges to validity become a concern

  4. VALIDITY CHALLENGES • Validity: the extent to which the test measures what it purports to measure • Internal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically? • External validity: can the results be generalized? • Measurement error: difference between the characteristics of your sample and the characteristics of the population • Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.

  5. MAXIMIZING VALIDITY • Create a questionnaire that is clear and logical • Support content relevance and content representativeness • Content Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated. • Content Representativeness: Are the questions an adequate representation of the universe of possible questions? • Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation • Select a large, representative random sample from the population of interest or census the finite population • Increase response rate using formal follow up procedures

  6. INITIAL CONSIDERATIONS • Is the development of a new questionnaire . . . • Necessary? • Feasible? • Select mode of administration • Web Based? • Paper and pencil? • Combination?

  7. QUESTIONNAIRE PLANNING • 1. Determine the Objective • What do you want to ask and of whom? • Consider how the data will be analyzed. What information is needed? • 2. Determine the Mode of Administration • Paper and pencil • Cost effective, but low response rate • Web / E-mail based • Cost effective, higher response rate, some suspicion of the medium • Combination • Can be administratively cumbersome

  8. QUESTIONNAIRE PLANNING • Determine the Sampling Methodology • To whom will you send the questionnaire? Who can supply the information? • Will everyone be surveyed? • ALL current students? ALL alumni? • If yes, this is a census of the finite population • The selection of a representative sample of the population may be more effective based on time and cost of administration • Decision depends on the size of the finite population

  9. CONSTRUCTING the QUESTIONNAIRE • Consider what objective each question measures • Three to five questions per objective is recommended • How many questions? • Find a balance: long enough to obtain the necessary information but short enough so respondents won’t lose interest. • Questionnaires that are too long won’t be completed • Goal: 25-30 minutes MAX

  10. DESIGNING THE QUESTIONNAIRE • Question Formats: Open Ended Questions • Allow respondents to express feelings and expand on ideas. • Question should be phrased to avoid a one word response. • “What aspects of the Sociology program best prepared you for your current position?” • Open ended questions at the beginning of the questionnaire may be discouraging to respondent – include one or two at the end • Analysis of responses can be challenging

  11. QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Ranking • Forces the respondent to rank order responses based on a value judgment. • Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time). _____ Studying _____ Socializing _____ Volunteering _____ Working at a part time job _____ Exercising

  12. QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Checklist • A number of possible answers are provided and the respondent is asked to choose one or choose all that apply. • How did you finance your Bucknell University education? Choose all that apply. _____ Parental contribution _____ Student Loan _____ Work Study _____ Off campus employment _____ Other (please specify): _____________

  13. Neutral / No Opinion Strongly Agree Strongly Disagree Disagree Agree QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Scaled items • Very common • Indicate strength of agreement or disagreement with a statement. • Numbers are assigned to each response on the continuum • The English curriculum at Bucknell University adequately prepared me for my current position. 1 2 3 4 5

  14. QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Suggestions for writing scaled items • Use positively worded or negatively worded items consistently • Use an odd number of responses along the continuum to allow for a neutral response • Anchors may be different based on focus of the item (Strongly Agree – Strongly Disagree; Frequently – Never; Very Good – Very Poor) • Keep items short

  15. QUESTIONNAIRE DEVELOPMENT • Question Formats: Categorical • Typical demographic questions: • Yes/No/?, Male/Female, F/S/J/S/G • Answers fall into certain categories – respondents pick the one that applies to them • Analyzed using frequency, proportion, percentages • 2 considerations: • Questions on a continuum should stay on a continuum if they will be directly analyzed • Example: Age – allow respondent to write in their age rather than check off a box for their age range • Categories must be collectively exhaustive & mutually exclusive (next slide)

  16. Problematic: Number of hours spent studying per week: 18 – 21 21 – 24 24 – 27 Categories are not mutually exclusive Corrected: Number of hours spent studying per week: 18 – 21 22 – 25 26 – 29 QUESTIONNAIRE DEVELOPMENT

  17. IMPORTANT CONSIDERATIONS • APPEARANCE AND DESIGN • Have clear directions that are age appropriate & easy to follow • Format, size, reproduction should facilitate completion • No less than 12 pt • Times, Arial, Courier • PILOT STUDY • Give the questionnaire to a few people to try it out • Have them evaluate readability and clarity of questions • Estimate time required to complete the questionnaire • Suggestions for revisions

  18. SENDING IT OUT • INITIAL CONTACT / COVER LETTER • Short & to the point • Identify person by name • Convince respondent that participation is important • Ensure confidentiality • Deadline for return of questionnaire • SASE for return (mailed questionnaires only)

  19. Institutional approval • Institutional Review Board • Research involving human subjects must be approved by the Bucknell University IRB • For more information, visit the IRB website at http://www.departments.bucknell.edu/inst_research/IRB/irb.htm • Campus Survey Coordination • The timing of all university-wide surveys must be coordinated through the Campus Survey Coordination Group. • For more information, visit the CSCG website at http://www.departments.bucknell.edu/inst_research/cscg.htm

  20. FINAL THOUGHTS • FOLLOW UP PROCEDURES • Send to all participants to preserve confidentiality (“If you haven’t already returned the questionnaire…..”) • May include a second copy if available • Usually done in 2 week intervals • RESPONSE RATE FOR MAILED QUESTIONNAIRES • Typically low - 30% is considered good • Higher response rate with web-based questionnaires • Good follow-up increases response rate

  21. SUMMARY OF STEPS • Determine the Objective (consider analyses) • Determine the Mode of Administration • Determine the Sampling Methodology • Construct the Questionnaire • Institutional Approval • Conduct the Pilot Study • Write the Initial Communication • Send the Questionnaire • Follow up • Analyze the Results

  22. FOR MORE INFORMATION Contact Kathy Martin @ 71960 or kmartin@bucknell.edu

More Related