Learning Objectives and Classroom Assessment

Learning Objectives and Classroom Assessment Jeff Froyd, Texas A&M University

Workshop Presenter • Jeff Froyd, Director of Academic Development • Educational Achievement Division, College of Engineering, Texas A&M University • Project Director, Foundation Coalition

Acknowledgement • Russ Pimmel, NSF Program Officer • Former Professor of Electrical Engineering at the University of Alabama • Assembled much of the material for similar workshops

Objectives: Participants will gain experience in • Describing the rationale for preparing and using learning objectives in an individual course • Preparing specifications for high quality learning objectives • Writing learning objectives for a single course • Preparing specifications for assessment processes/tools • Generating alternative assessment processes/tools for a single course • Selecting assessment processes/tools for a single course

Agenda • Background for learning objectives • Specifications for learning objectives • Writing learning objectives • Background for classroom assessment • Specifications for assessment processes/tools • Background on alternatives for assessment processes/tools • Generating alternatives for assessment processes/tools • Selecting assessment processes/tools • Review workshop activities

Agenda • Background for learning objectives • Specifications for learning objectives • Requirements for specifications • Team exercise: develop specifications for learning objectives • Workshop exercise: improve specifications for learning objectives • Writing learning objectives • Individual exercise: write objectives for a course or a portion of a course • Team exercise: review individual objectives • Workshop exercise: develop a list of suggestions for writing learning objectives • Individual exercises: revise objectives Team exercise: review individual objectives • Workshop exercise: reflection

Agenda • Background for classroom assessment • Specifications for assessment processes/tools • Team exercise: develop a set of specifications for assessment processes/tools • Workshop exercise: improve sets of specifications for assessment processes/tools • Generating alternatives for assessment processes/tools • Team exercise: select some learning objectives and generate alternative assessment processes/tools • Background on alternatives for assessment processes/tools • Selecting assessment processes/tools • Individual exercise: select a set of learning objectives and generate alternative assessment processes/tools • Individual exercise: select one or more assessment processes/tools that you would use in your course • Team exercise: share and review choices of assessment processes/tools • Review workshop activities

Learning Objectives Background for Learning Objectives Session Objective: At the end of the session, participants will describe themselves as more confident in their ability to hold productive conversations with their colleagues regarding the place and importance of learning objectives in the teaching-learning process

Stakeholders for Learning Objectives • Who are the stakeholders in conversations about preparing and applying learning objectives? • Faculty • Students • Employers • Accreditation organizations

EC Program Outcomes (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs (d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

Bloom’s Taxonomy of Cognitive Learning • Knowledge: defines, recalls, matches, reproduces • Comprehension: explains, gives examples • Application: discovering, assessing, computing • Analysis: breaking down, organizing, inferring • Synthesis: creating, putting together • Evaluation: appraising, judging, selecting Western Michigan University, 25 October 2002, Kalamazoo, Michigan

Purpose of Learning Objectives • Communicate expectations for a course • Provide a context for what will be learned • http://wings.buffalo.edu/vpaa/ctlr/events/brazeau/index.htm

Objectives and Students • Objectives help students • Clarify their personal goals • Provide framework for measuring their success. • Reduce their anxiety • Improve their studying effectiveness • Objectives help instructors • Guide preparation of classroom material • Make homework assignments • Aid in test design • Source - http://wings.buffalo.edu/vpaa/ctlr/events/brazeau/index.htm

Strategies For Workshop Teams • Be positive, supportive, and cooperative • Limit critical or negative comments • At least 5 positive comments for every negative comment • Be brief and concise in discussions • Avoid lengthy comments, stories, or arguments • Stay focused

Team Roles • Assign team roles & follow through on responsibilities • Coordinator -- Coordinates discussion & develops consensus • Recorder -- Writes down the ideas & reports them • Gatekeeper -- Keeps the team on the subject • Timer -- Makes sure the team stays on schedule • With smaller teams – combine gatekeeper & timer

Workshop Team Roles • For first exercise • Coordinator – Individual with largest class last semester • Recorder/Reporter – Individual on left of coordinator • Gatekeeper/timer -- Individual on left of recorder • Timer -- Individual on left of gatekeeper • Roles rotate clockwise on subsequent exercise

Team Exercise • Form teams of four people • Time: 5 minutes • Develop at least four (4) advantages and four (4) disadvantages of preparing learning objectives for a course.

Advantages Communication with students expectations and what they need to learn Offers focus in preparing material to accomplish goals Evaluation tool Tie a course to program or curriculum Clarifying goals to students seeks buy-in and self-assessment ABET Assessment tool Stating goals facilitates communicates with colleagues for better coordination Written goals facilitates communications with employers and other external groups Makes course easier to transfer among faculty members because much work is already done Disadvantages Could be too narrow or focused Explicitly stating learning objectives takes (too?) much time Busy work for faculty Could turn into curriculum nightmare Poorly written objectives don’t help Could be cumbersome, ambiguous Could be more difficult to measure Team Exercise

Learning Objectives Preparing Specifications for Learning Objectives Session Objectives At the end of the session, participants will • Write specifications for learning objectives • Describe themselves as more confident in their ability to describe quality learning objectives.

Form of Learning Objectives • Write objectives as student outcome statements • Objectives should answer the questions • "What must students do to prove that they have succeeded?" • "What should students be able to do as a consequence of instruction?" • Source - http://wings.buffalo.edu/vpaa/ctlr/events/brazeau/index.htm

Elements of an Objective • Objective must contain three basic elements: • Verb describing an observable action • Conditions of this action • “When given x you will be able to..." • Level of acceptable performance • Source - http://wings.buffalo.edu/vpaa/ctlr/events/brazeau/index.htm

Verbs for Objectives Verbs for constructing concrete objectives: analyze compute classify collaborate compare appreciate contrast define demonstrate direct derive designate discuss display evaluate know identify infer integrate interpret justify list understand organize grasp report respond solicit state synthesize name explain Modified from http://www.brown.edu/Administration/Sheridan_Center/pubs/syllabus.html#coura

Learning Objectives Verbs for Categories in Bloom’s Taxonomy • Knowledge • Define, describe, list, reproduce, enumerate • Comprehension • Classify, explain, discuss, give example, summarize • Application • Determine, develop, compute, chart, utilize • Analysis • Correlate, diagram, distinguish, outline, infer • Evaluation • Compare & contrast, critique, justify, conclude • Synthesis • Adapt, combine, compare, contrast, design, generate

Complex Versus Simple Objectives • One complex objective versus several simple ones? • High-level versus low-level objectives Example • One complex objective (4 or 5 weeks of classes) “Given a verbal description of a digital module, develop an implementation using any of 7 different logic devices” • 15 to 20 simpler objectives (1 or 2 per class) “Given a verbal description, draw the truth table” “Given a truth table, obtain a minimum-cost equation” … “Draw the the NAND-gate implementation for an equation”

Comparison Of Complex & Simple Objectives • Multiple simple objectives • More manageable “chunks” for students • Explicit objective(s) for each class • Simple (more manageable) homework problems and test questions • Single complex objective • Student's attention directed to the overall process • May lead to higher level learning • Students must deal with complexity • Students must subdivide problem on their own

Advantages Of Simple Objectives • Advantages of simple objectives are more important in • Large classes rather than small classes • Introductory courses rather than advanced courses

How To Deal With “Understand” In Objectives • How do you write objectives when you want students to “understand” a complex concept, system, or process • Identify specific tasks that indicate “understanding” • Specify objectives for each task • Similar comments apply to “know”, “appreciate”, “value”

How To Deal With “Understand” In Objectives -- Example • In our computer architecture course we want students to “understand” a sample architecture made up of several modules • What would students be able to do if they “understood” • Objectives – • Students should be able to identify: All the modules and interconnecting signals Modules involved in a given system-level operation Output values for a given input values for each module Sub-module changes given a system level change …

Writing Objectives – Piecemeal Approach • Writing low-level objectives for a whole course may be overwhelming • Use a “piecemeal” approach • Write your lectures and define the homework as usual • After each class -- write down what you expect the students to be able to do • These become a list of objectives • Give them to the students before each exam • Use them to write the exam • As semester progresses -- may become comfortable writing the objectives before you prepare your lecture

Evolving Objectives • In a 3-credit semester course • Russ Pimmel (UA, now NSF) started with over 100 objectives • Four offerings later -- down to about 50 • Eliminated peripheral “stuff” that was not central • Broadening, informational, perspective material

Objectives and Homework Assignments • Homework assignments should match objectives • Students need to practice and explore the skills, knowledge, and attitudes defined in objectives • Frequently require supplementary homework problems • In some of my courses • 1/3 of homework is from textbook • Rest are supplementary problems • With well defined objectives • Writing homework problems is straightforward • Same is true for exam questions

Student’s Use Of Objectives • Survey in 400-level required course • Did you find the objectives helpful? Yes --- 52 % No ---- 48% • Did you read the objectives? Frequently ------ 22 % Occasionally -- 37 % Never ------------- 41 %

Team Exercise -- Guidelines For Learning Objectives • Task • Write 3 or 5 guidelines for good learning objectives • What are the common features? • What should objectives look like? • Think of guidelines as specifications • Methodology • Brain storm individually -- 2 minutes • Establish consensus as a team -- 5 minutes • Report team results -- 3 minutes • Revise guidelines as a team-- 2 minutes

Team Exercise • Must be testable and measurable • Achievable • Clearly and precisely articulated • Appropriate to course and audience • Relate to program objectives • Linked to course outcomes to allow for assessment • Simple, one sentence, common format • Simple better than complex • Should be specific and unambiguous • Relate to topic coverage

Learning Objectives Preparing Learning Objectives Session Objectives At the end of the session, participants will • Write learning objectives for one or more courses that they teach • Describe themselves as more confident in their ability to describe quality learning objectives.

EC Program Outcomes (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs (d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

Bloom’s Taxonomy of Cognitive Learning • Knowledge: defines, recalls, matches, reproduces • Comprehension: explains, gives examples • Application: discovering, assessing, computing • Analysis: breaking down, organizing, inferring • Synthesis: creating, putting together • Evaluation: appraising, judging, selecting Western Michigan University, 25 October 2002, Kalamazoo, Michigan

Challenges in Engineering Education • Challenges • Challenge of lifelong learning • Challenge of problem solving • Challenge of engineering design • Challenge of transfer

Lifelong Learning at Penn State Self-Directed Learning Readiness Survey (SDLRS), Guglielmino & Associates, http://www.guglielmino734.com/prod01.htm, March 2003. 27 “Although the data suggest a slight upward upward trend, the trend proved not to be statistically significant based upon an analysis of variance (ANOVA). Thus the cross-sectional study did not find evidence of an increase in readiness for self-directed learning, even for students in the later semesters who are taking elective courses and their capstone courses.” Litzinger, T., Wise, J., Lee, S., and Bjorklund, S. (2003) Assessing Readiness for Self-directed Learning, Proceedings, ASEE Annual Conference

Challenge of Problem Solving “Despite individual professors’ dedication and efforts to develop problem solving skill, “general problem solving skill” was not developed in the four years in our undergraduate program. Students graduated showing the same inability that they had when they started the program. Some could not create hypotheses; some misread problem statements. During the four-year undergraduate engineering program studied, 1974-1978, the students had worked over 3000 homework problems, they had observed about 1000 sample solutions being worked on the board by either the teacher or by peers, and they had worked many open-ended problems. In other words, they showed no improvement in problem solving skills despite the best intentions of their instructors.” Woods, D. et al (1997) “Developing Problem Solving Skills: The McMaster Problem Solving Program,” Journal of Engineering Education,

Challenge of Problem Solving • Ineffective approach #1. give the students open-ended problems to solve; This, we now see, is ineffective because the students get little feedback about the process steps, they tend to reinforce bad habits, they do not know what processes they should be using and they resort to trying to collect sample solutions and match past memorized sample solutions to new problem situations.

Challenge of Problem Solving • Ineffective approach # 2: Show them how you solve problems by working many problems on the board and handing out many sample solutions • This, we now see, is ineffective because teachers know too much. Teachers demonstrate "exercise solving". Teachers do not make mistakes; they do not struggle to figure out what the problem really is. They work forwards; not backwards from the goal. They do not demonstrate the "problem solving" process; they demonstrate the "exercise solving" process. If they did demonstrate "problem solving" with all its mistakes and trials, the students would brand the teacher as incompetent. We know; we tried!

Challenge of Problem Solving • Ineffective approach #3: Have students solve problems on the board • Different students use different approaches to solving problems; what works for one won't work for others. When we used this method as a research tool, the students reported "we learned nothing to help us solve problems by watching Jim, Sue and Brad solve those problems!"

Challenge of Problem Solving • Through four research projects we identified why and how these and other teaching methods failed to develop process skills and which methods were successful in developing the skills • Woods, D.R., J.D. Wright, T.W. Hoffman, R.K. Swartman and I.D. Doig (1975) "Teaching Problem Solving Skills," Annals of Engineering Education, 1, 1, 238-243. • Woods, D.R. et al. (1979) "Major Challenges to Teaching Problem Solving" Annals of Engineering Education, 70, No. 3 p. 277 to 284, 1979 and "56 Challenges to Teaching Problem Solving" CHEM 13 News no. 155 (1985). • Woods, D.R. (1993a) "Problem solving - where are we now?" J. College Science Teaching, 22, 312-314. • Woods, D.R. (1993b) "Problem solving - what doesn't seem to work," J. College Science Teaching, 23, 57-58. • Woods, D.R. (1993c) "New Approaches for developing problem solving skills," J. College Science Teaching, 23, 157-158.

Challenge of Engineering Design The literature is filled with positive comments from students, instructors, and industrial sponsors who have participated in capstone design courses. The vast majority of participants feel that the course benefited all involved. The nature of capstone design courses, however, often leads to a purely subjective evaluation with little or no “hard evidence” of actual benefits. Born, for example, does not attempt to prove the value of senior level design courses. He simply states that he is convinced from his experiences that such courses are valuable. Other educators have similar “feelings” as to the relative costs and benefits of capstone design courses. Dutson, A.J., Todd, R.H., Magleby, S.P., Sorensen, C.D., (1997) “A Review of Literature on Teaching Engineering Design Through Project-Oriented Capstone Courses.” Journal of Engineering Education

Challenge of Transfer Researches posed this problem to people. "Suppose you are a doctor faced with a patient who has a malignant tumor in his stomach. It is impossible to operate on the patient, but unless the tumor is destroyed the patient will die. There is a kind of ray that can be used to destroy the tumor. If the rays reach the tumor all at once at a sufficiently high intensity, the tumor will be destroyed. Unfortunately, at this intensity the healthy tissue that the rays pass through on the way to the tumor will also be destroyed. At lower intensities the rays are harmless to healthy tissue, but they will not affect the tumor either. What type of procedure might be used to destroy the tumor with the rays, and at the same time avoid destroying the health tissue?"

Challenge of Transfer Consider the following story "A small country was ruled from a strong fortress by a dictator. The fortress was situated in the middle of the country, surrounded by farms and villages. Many roads led to the fortress through the countryside. A rebel general vowed to capture the fortress. The general knew that an attack by his entire army would capture the fortress. He gathered his army at the head of one of the roads, ready to launch a full-scale direct attack. However, the general then learned that the dictator had planted mines on each of the roads. The mines were set so that small bodies of men could pass over them safely, since the dictator need to move his troops and workers to and from the fortress. However, any large force would detonate the mines. Not only would this blow up the road, but it would also destroy many neighboring villages. It therefore seemed impossible to capture the fortress. However, the general devised a simple plan. He divided his army into small groups and dispatched each group to the head of a different road. When all was ready he gave the signal and each group marched down a different road. Each group continued down it road to the fortress at the same time. In this way, the general captured the fortress and overthrew the dictator."

Challenge of Transfer • After the subjects read and summarized this story, they were asked to solve the tumor problem under the guise of a separate experiment. • Given the clear analogy, you might think that performance would be near ceiling. Surprisingly, only 30% of the subjects offered a convergence solution. • Moreover, when these same subjects were given the suggestion that they should use the General story, 80% provided a convergence solution. • This finding demonstrates that half the subjects could apply the General story to the tumor problem when they were instructed to but did not do so on their own.

Focusing Activity (8 minutes) • INDIVIDUALLY – use 3 minutes to write your description of learning, what it is, what it looks like, how you might recognize when it has occurred, etc. • AS A PAIR – use 5 minutes to discuss descriptions with someone sitting next to you. If you have additional time, develop a consensus description of learning.

Learning Objectives and Classroom Assessment