Service Evaluation

1. Service Evaluation Gilbert O Ouma Institute of Climate Change and Adaptation University of Nairobi, Kenya

2. Outline • Introduction • Steps in conducting a survey • Establishing goals • Selecting sample population • Methods of data collection • Questionnaire design – the considerations • Question structure • Guidelines for formulating questions • Setting metrics

3. Introduction • Knowing what the client wants is the key factor to success in any type of business • News media, government agencies and political candidates, etc. need to know what the public thinks • We as meteorologists need to know what our users want • The best way to find this information is to conduct surveys

4. Steps in conducting a survey • Establish the goals of the project - What you want to learn • Determine your sample - Who you will interview • Choose interviewing methodology - How you will interview • Create your questionnaire - What you will ask • Pre-test the questionnaire, if practical - Test the questions • Conduct interviews and enter data - Ask the questions • Analyze the data - Produce the reports

6. Establishing Goals • Before you even write the first question, it is important that you have a very clear idea about what you want your questionnaire to achieve • Write down your research goals, and think about what information you need to elicit from respondents to give you evidence that those goals are met • If your program has specific goals and objectives, the needed baseline/monitoring/final evaluation data will be quite obvious • If the goals and objectives are vaguely defined or undefined, you will find it difficult to know what kind of data to gather

7. Examples of goals • The potential market for a new product or service • Ratings of current products or services • User satisfaction levels • User opinions • These goals represent general areas • The more specific you can make your goals, the easier it will be to get usable answers

8. Selecting Your Sample • There are two main components in determining who you will interview • The first is deciding what kind of people to interview - often called the target population • If you conduct an employee attitude survey or an association membership survey, the population is obvious • If you are trying to determine the likely success of a product, the target population may be less obvious • Correctly determining the target population is critical • If you do not interview the right kinds of people, you will not successfully meet your goals

9. The next thing to decide is how many people you need to interview • A small, representative sample can reflect the group from which it is drawn • The larger the sample, the more precisely it reflects the target group • However, the rate of improvement in the precision decreases as your sample size increases • For example, to increase a sample from 250 to 1,000 only doubles the precision • You must make a decision about your sample size based on factors such as • time available • budget • necessary degree of precision

10. Avoid bias in your samples • Totally excluding all bias is almost impossible • however, if you recognize bias exists you can intuitively discount some of the answers • Source of bias depend on the nature of the survey • For example • a survey for a product aimed at retirees will not be as biased by daytime interviews as will a general public opinion survey • A survey about Internet products can safely ignore people who do not use the Internet

12. Methods of data collection • There are many ways of getting information • The most common methods are: • literature searches • talking with people • focus groups • personal interviews • telephone surveys • mail surveys • email surveys • internet surveys

13. Literature search • Involves reviewing all readily available material • These materials can include: • internal company information • relevant publications • Newspapers • magazines • on-line data bases • any other published materials • It is a very inexpensive method of gathering information • But does not often yield timely information

14. Talking with people • A good way to get information during the initial stages of a program/project • Can be used to gather information that is not publicly available, or that is too new to be found in the literature • Basically these involve meetings with different users of your products • Although often valuable, the information has questionable validity because it is highly subjective and might not be representative of the population

15. Focus group • Used as a preliminary tool • to explore peoples ideas and attitudes • to discover customer concerns • A group of 6 to 20 people meet in a conference-room-like setting with a trained moderator • The moderator leads the group's discussion and keeps the focus on the areas you want to explore • Their disadvantage is that the sample is small and may not be representative of the population in general

16. Personal interviews • Good for getting in-depth and comprehensive information • Involves one person interviewing another person for personal or detailed information • Typically, an interviewer will ask questions from a written questionnaire and record the answers verbatim • Sometimes, the questionnaire is simply a list of topics • Personal interviews (because of their expense) are generally used only when subjects are not likely to respond to other survey methods

17. Telephone surveys • The fastest method of gathering information from a relatively large sample (100-400 respondents) • The interviewer follows a prepared script that is essentially the same as a written questionnaire • However, unlike a mail survey, the telephone survey allows the opportunity for some opinion probing

18. Mail surveys • A cost effective method of gathering information • Ideal for • large sample sizes • or when the sample comes from a wide geographic area • Because there is no interviewer, there is no possibility of interviewer bias • The main disadvantage is the inability to probe respondents for more detailed information

19. Email and internet surveys • Relatively new and little is known about the effect of sampling bias in internet surveys • While it is clearly the most cost effective and fastest method of distributing a survey, the demographic profile of the internet user does not represent the general population, although this is changing

20. The choice of survey method will depend on the following

21. Questionnaire design – general considerations • Well-defined goals are the best way to assure a good questionnaire design • When the goals of a study can be expressed in a few clear and concise sentences, the design of the questionnaire becomes considerably easier • The questionnaire is developed to directly address the goals of the study

22. KISS - keep it short and simple • If you present a 20-page questionnaire most potential respondents will give up in horror before even starting • Response rate is the single most important indicator of how much confidence you can place in the results • A low response rate can be devastating to a study

23. Start with an introduction or welcome message • When practical, state who you are and why you want the information in the survey • A good introduction or welcome message will encourage people to complete your questionnaire • Include clear and concise instructions on how to complete the questionnaire • These must be very easy to understand, so use short sentences and basic vocabulary

24. Begin with a few non-threatening and interesting items • If the first items are too threatening or "boring", there is little chance that the person will complete the questionnaire • People generally look at the first few questions before deciding whether or not to complete the questionnaire • Make them want to continue by putting interesting questions first

25. Question Structure • Broadly speaking, survey questions can be classified into three structures: • Closed • open-ended • contingency

26. Closed or multiple choice questions • These ask the respondent to choose, among a possible set of answers, the response that most closely represents his/her viewpoint • May offer simple alternatives such as ‘Yes’ or ‘No’ • They may also require that the respondent chooses among several answer categories, or that he/she uses a frequency scale, an importance scale, or an agreement scale

27. Have you observed delayed onset of rains in these seasons? • Yes • No • Do you receive weather information? • Yes • No • Does this information reach you in good time? • Yes • No

28. Advantages of closed questions are: • the respondent is restricted to a finite (and therefore more manageable) set of responses • they are easy and quick to answer • they have response categories that are easy to code • they permit the inclusion of more variables in a research study because the format enables the respondent to answer more questions in the same time required to answer fewer open-ended questions.

29. The disadvantages of closed questions are: • They can introduce bias • By forcing the respondent to choose between given alternatives • By offering alternatives that otherwise would not have come to mind • Where there is a tendency for the respondent to tick systematically either the first or last category • Where respondents select what may be considered as the most socially desirable response alternative • Where respondents answer all items in a list in the same way • They do not allow for creativity or for the respondent to develop ideas • They do not permit the respondent to qualify the chosen response or express a more complex or subtle meaning • They require skill to write because response categories need to be appropriate, and mutually exclusive

30. Some closed questions may have only two mutually exclusive responses are provided • Gender • Female • Male • Should not be overused in a survey because it elicits much less information than multiple choice formats • Do you read a daily newspaper? - can elicit a yes or no answer without giving a lot of information • May be reworded

31. How many times per week do you read a daily newspaper? • Seven times a week • Five to six times a week • Three to four times a week • One to two times per week • Less than once per week • Never • This would provide more specific and useful information

32. Open-ended or free response questions • These are not followed by any choices • The respondent must answer by supplying a response usually by entering • a number • a word • a short text • Answers are recorded in full, either by the interviewer or, in the case of a self-administered survey, the respondent records his or her own entire response

33. What additional weather information you would like to receive to improve your farming? __________________________________________________________________________ • In which years (in the last two decades) have you had floods _____________________________________ _____________________________________

34. The advantages of open-ended questions are: • They allow respondents to express their ideas spontaneously in their own language • They are less likely to suggest or guide the answer than closed questions because they are free from the format effects associated with closed questions • They can add new information when there is very little existing information available about a topic

35. The disadvantages of open-ended questions are: • They may be difficult to answer and even more difficult to analyze • They require the development of a system of coded categories with which to classify the responses • They require effort and time on behalf of the respondent • They require the respondent to have some degree of writing ability • Respondent handwriting can be illegible

36. There is always the possibility with open-ended questions that responses may come in very different forms • may lead to answers that cannot be systematically coded for analysis • When did you leave school? • Seven years ago • When I got my first job • When my brother started going to high school • When my parents moved into this house • Appropriate probing by the interviewer • Clear instructions for self-administered

37. Contingency questions • This is a special case of a closed-ended question • because it applies only to a subgroup of respondents • The relevance of the question for a subgroup is determined by asking a filter question • The filter question directs the subgroup to answer a relevant set of specialized questions and instructs other respondents to skip to a later section of the questionnaire

38. In the last ten years have you bought or sold land • Yes • No > GO TO Q.21 • The advantage of contingency questions is that detailed data may be obtained from a specific subgroup of the population • Some questions may apply only to, for example, females and not to males • At the base of good contingency questions are clear and specific instructions to respondents

39. Guidelines for writing questions • Keep the vocabulary simple • Use simple words • Avoid acronyms, abbreviations, jargon, technical terms, abstract or general words, words with ambiguous meanings • If a technical term is used, its meaning must be explained • General or abstract words – examples must be provided to clarify meaning

40. Keep the question short • Generally, it is recommended to hold questions to 25 words or less • If a longer sentence is used then it should be broken up so that there will be several shorter sentences • Avoid double-barrelled questions • These are single questions that ask for two things and therefore require two answers • Do you have your own table or your own room to do your homework? • Do you think it is good idea for children to study geography and history in primary school? • In such instances, respondents do not know what to do if they want to say ‘Yes’ to one part of the question but ‘No’ to the other

41. Avoid hypothetical questions • Evidence has shown that hypothetical questions such as “Would you use this resource in your class if it were available?” are not good for the prediction of behaviour • People are generally poor predictors of their own behaviour because of changing circumstances and because so many situational variables intervene • You are able to collect more valid data if you ask the respondents about their past behaviour and present circumstances, attitudes, and opinions

42. Do not overtax the respondent’s memory • It is risky to ask the respondent to recall past behaviour over a long retrospective period • Especially when recurrent events or behaviours are concerned • How many severe El-Nino events have occurred since 1961? • If such a question must be asked, a shorter recall period might be more appropriate

43. Avoid double negatives • Double negatives, either in the question or an answer category (or both), create difficulties for the respondent • Customer satisfaction surveys by meteorological agencies should not be allowed, agree or disagree • This is problematic to answer for respondents who are in favour of the surveys, that is those who do not agree that the surveys should not be allowed

44. Beware of ‘leading’ questions • A leading question is a question phrased in such a way that it seems to the respondent that a particular answer is expected • Do you favor inclusion of indigenous knowledge in deriving weather forecasts? • Could be framed as • You wouldn’t say that you were in favor of inclusion of indigenous knowledge in deriving weather forecasts, would you?” • Or in a more subtle form: • Would you say that you are not in favor of inclusion of indigenous knowledge in deriving weather forecasts?

45. Setting Metrics • One of the important elements of performance management is the use of metrics • A metric is a numerical measure used as baseline measurement for performance • Metrics are used to define quantitative measurements for the quality of your service - they are basically a definition of the performance goals that you want to achieve • They depend on the objectives and goals of your program/projects

46. Overall Objective • The answer to • Why does your agency exist? • What do you want to achieve? • Should have a clearly defined list of objectives • Defines where you are going • If you don't know where you are going then any road will take you there • Also known as desirable outcomes

47. The objectives must be • Doable • Understandable • Manageable • Beneficial

48. Target Metrics are defined at the start of a project to: • indicate the minimum quality which you want to achieve • define how these factors will actually be measured • When used properly these metrics can provide a useful tool • When used irresponsibly they can be a time-wasting distraction • You need to understand what you are measuring, how you are measuring it and why

49. Key Performance Indicator (KPI) • Key Performance Indicators (KPIs) are metrics that help you understand how you are doing against your objectives • For example, if the objective is Awareness about the products, the KPI could be average number of requests for specific products per month

50. Goals • Specific strategies you'll leverage to accomplish your business objectives • Objectives can be quite strategic and high level – support food security • Goals – very specific • Improve forecasts (can be broken down further) • Create awareness, etc.