Construct Evaluations

1. Construct Evaluations

2. True-False • Meaningful true-false items are difficult to compose • Many statements are neither completely true nor completely false • Amount of guessing may be high • Test takers have a 50 percent chance of being correct • Tend to measure recognition only

3. True-False • Precautions • Provide accurate directions • Balance the true and the false items and distribute them so no pattern is evident • Keep true and false statements about equal in length • Used positive rather than negative statements • Avoid the use of none, never, no, or not and always

4. True-False • Construct true-false items that are either true or false, not partially true or false • Provide a space to the left of each item where the student can insert the correct answer for an item which is false • This distinguishes an accidental correct guess from a correct response

5. True-False • Example • Directions: read each of the following statements. If a statement is dramatically correct, circle the T. If the statement is incorrect, circles the F and change the underlined word to make this statement dramatically correct. Place the correct word in the blank beside the F. T F _____ 1. To whom should I give the umbrella? T F She__ 2. Her and I signed up for the trip

6. Multiple Choice • Consists of an incomplete statement or a question followed by four alternatives • Three parts • The incomplete statement-the stem • The correct answer • 3 alternatives-distractors

7. Multiple Choice • Example Directions: in the space provided beside each question, write the letter of the response which best completes the statement. Stem Correct Response Distractors

8. Multiple Choice • 12 guidelines • Word each multiple-choice item as either a statement or a question • What is the period covered by the balance sheet? • Year • Month • Day • Week

9. Multiple Choice • Make sure the stem contains a single clearly worded idea • Good: the rule of 72 is used to determine • Average annual rate of return • Number of years needed to double investment • Total return on investment • Daily rate of return • Poor: what is the chief purpose of using the rule of 72? • Average annual rate of return • Number of years needed to double investment • Total return on investment • Daily rate of return • The term purpose may confuse the reader • The alternatives are not phrased in a way as to answer the question

10. Multiple Choice • Keep the correct response the same length as each distractor • If a negative statement is used, underline it, italicize it, or place it at the end of the stem or do both • Good: an employee whose only employment is in the factory is eligible to invest in all of the following except • An individual retirement account • Keogh plan • Stock option plan • Municipal bonds • Poor: An employed factory worker is not eligible to invest in which of the following?

11. Multiple Choice • Make certain that the correct response is accurate or certainly the best of alternatives • Avoid any clues such as repeating keywords from the stem in the alternatives • Determine that all the distractors are believable • Vary the length of the correct response from item to item to avoid giving clues • Make certain that all alternatives are grammatically correct and written in parallel form

12. Multiple Choice • Avoid alternatives like “none of the above” or “all of the above” • Vary good position of the correct response from question to question • In formatting the questions, put each alternative on a separate line to permit easy reading

13. Matching • Parallel columns are usually provided for matching items • The items below column A are called premises and below column B responses

14. Matching Directions: in this space provided beside column A, write a letter of the statement in column B that best completes each thought expressed. Each item in column B may be used more than once. Responses Premises

15. Matching • Matching items are best suited to large units of subject matter which are sufficiently homogeneous to require discrimination on the part of the students

16. Matching • Guidelines for designing matching questions • Provide clear, simply stated directions • Restrict the number of items in the stem column (column A) (no more than 10) • Include fewer or more alternatives (column B) than items in the stem column (column A) to prevent students from guessing

17. Matching • Arrange items in the stem column in logical order or in ascending or descending order • Arrange alternatives in logical order • Keep all items in a matching question on one page

18. Completion • Simply an incomplete statement • The examinee applies the correct word, amount, or date

19. Completion • Directions: for each of the items below insert the correct response in the blank space at the end of the sentence • The maturity value for an $850, 120-day note at 14 percent is _?_. • The date of the census for which punch cards were first use is _?_.

20. Completion • Use completion items only for those questions for which there is only one correct response • Some guidelines • Give explicit instructions • But the blank at the end of the sentence

21. Completion • Keep all blank lines the same size but provide enough room for the longest answer • Provide no hints • First letter • Number of blank spaces • Length of the blank line • Avoid copying statements directly from the textbook • Only evaluates recognition

22. Short Answer • The examinee is presented with a question and required to supply a word, name, date or amount

23. Short Answer • Directions: Insert the correct response in the space to the left of the number of each test item

24. Short Answer • Use test items for which there is only one correct response • Correct • What is the name of the COBOL program division which gives step-by-step commands for the computer to follow in processing the data? • Answer: procedure division • Incorrect • Explain the divisions of a COBOL program and the function of each • Suggests an essay, which is inappropriate for “understanding-level” outcomes • Not a short answer item

25. Design of Mastery Tests • Chief purpose-discriminate between the student who has fully mastered a competency and one who has not • The progress test • The person identified as non master can seek additional learning experiences and later be retested • Each learner should eventually achieve the minimum level of performance required

26. Design of Mastery Tests • Three critical decisions • Eliminate subjectivity from scoring • Scored each item either as correct (1 point) or incorrect (0 points) • No judgment is made; the answer is either right or wrong

27. Design of Mastery Tests • Set a cutoff score • A student who achieves a 70 percent score and passes the test has not mastered 30 percent of the items • The cutoff score must be set high in order to increase the probability that it will differentiate masters from non masters • Experts suggest a cutoff of 85 to 90 percent

28. Design of Mastery Tests • Decide on number of items per competency • A useful rules found is between 12 and 20 test items are needed for each enabling objectives that you wish to test • A mastery test is appropriate only for competencies which are essential • Nice-to-know competencies should not be tested by mastery test

29. Design of Norm-Referenced Test • The various types of objective items used for mastery test are also appropriate for an norm-referenced test • The norm-referenced test is designed differently

30. Design of Norm-Referenced Test • Four major guidelines • The purpose is to gather information to rank individuals within the group-only a sample of the content in the unit is selected for the test • Seek to avoid questions which all students are likely to answer correctly • A wide variation in scores is sought • Since several competencies are being measured, the norm-referenced test contains more questions than the mastery test

31. Design of Norm-Referenced Test • No decision on mastery can be made from a norm-referenced test

32. Performance Test • Includes essay questions, product performances, and process performances • It is essential that as much objectivity as possible be built into these tests

33. Performance Test • Judgments made on performance test are more likely to be made objectively when the teacher uses • Rubrics • Rating scales • Checklist • Assessment instruments designed at the time to test is constructed

34. Essay Questions • Appropriate for evaluating the competency when a student is required to • Select relevant information • Organize • Integrate • Analyze • Synthesize • Make decisions

35. Essay Questions • The process of assigning points to each essay question is subjective and subject to human error • Essay questions fall into two categories • Restricted response • Extended response

36. Essay Questions • Restricted response • Limits the content involved as well as the response • Example 1: describe two causes of inflation. For each cause describe two actions the government may take in controlling it • The student must simply recall two causes • Evaluates understanding of inflation and demands analysis of remedies

37. Essay Questions • Example 2: analyze the two income statements attached and identify any problems the Best Company is experiencing. For each problem, explain a possible cause or causes and recommend one or more solutions • Involves analysis • Content is limited by the case, as is the response

38. Essay Questions • Extended response • Gives the student full responsibility for • Selecting relevant information • Organizing • Analyzing • And presenting the analysis

39. Essay Questions • Example: contrast Keynesian theory with supply-side theory in terms of objectives, applications, and possible effects on (a) a declining economy and (b) an inflationary economy • Forces a student to analyze and communicate effectively in writing

40. Essay Questions • Guidelines • Use essay only for those learning outcomes which cannot be measured by objective items • Develop a classification system • Summarize prominent ideas • Analyze the task and formulate logical steps for accomplishing it • Create a classification • Explain a phenomenon • Decide which data are relevant • Conclude from information given • Defend decisions or actions • Predict trends or actions based on data or information

41. Essay Questions • Design the question to elicit the behavior specified in the learning outcome • Example: if you want a student to analyze financial statements, “conclude from information given,” is appropriate • Essay question: from the financial statements provided for the Lexion Company, determine the financial condition of the company and present a complete analysis of the company’s financial strengths and weaknesses

42. Essay Questions • State each question precisely so that the individual understands what is expected • Poor: described the impact of word processing systems on office equipment • Fails to provide a clue as to what is expected (impact on numbers of people, task performed, or organization, or all?) • Better: describe the impact of word processing in the office on productivity, work measurement, workflow, and career paths

43. Essay Questions • Specify the maximum number of points to be awarded for a correct and complete answer beside each question • Aids the student in managing time • Prepare a key to use in evaluating responses at the time the questions are designed and written

44. Product and Processed Performances • Subjective judgments sometimes interfere with the evaluation process • Teachers have many opportunities to observe students affective behaviors

45. Product and Processed Performances • Teachers may structure observations using some form of assessment instrument or checklist, for example • Individual working independently—reading, performing the task, organizing for performance, or observing a peer • Individuals seeking assistance from peers • Individual giving assistance to peers • Individual working with one or more peers while interacting; observing: perhaps as part of a team; peers giving report or performing; organizing a project; or performing a task

46. Product and Processed Performances • A teacher planning to observe the student’s oral presentation might decide to observe both cognitive and affective behaviors • Cognitive • Accuracy of content • Organization • Quality of summary • Use of correct grammar

47. Product and Processed Performances • Affective • Eye contact and other nonverbal language • Attitude of the speaker when responding to questions • A rubric, rating scale, checklist, or assessment instrument can be used to reduce subjectivity • Such instruments are designed before the learning process begins and shared with the learner

48. Product and Processed Performances • Rating scales • Sets of statements describing qualities of the product or process being judged • Descriptors placed on a scale guide the evaluator in determining where on the scale the product being judged falls Rating Scale for Oral Presentations.doc

50. Product and Processed Performances • Students should be provided with copies before the learning process begins • When learners are aware of the criteria on which a evaluation will be based they can concentrate their energies on meeting them • A single performance does not necessarily represent later performances • Many performances are required