"Teaching is leading students into a situation in which they can only e scape by thinking"

"Teaching is leading students into a situation in which they can onlyescape by thinking" Spencer Benson, Ph.D., Director The Center for Teaching Excellence, University of Maryland 2008-09 Fulbright Scholar, Hong Kong University HKAC General Education Initiative

The Roles of Learning Outcomes at a Research Intensive University: Successes, Challenges, and Lessons Learned Hong Kong UniversityNov. 12, 2008

Today’s Presentation • What are learning outcomes? • Background for the work at Maryland • The Provost Commission on Learning Outcomes Assessment • Successes • Challenges • Lessons Learned

Truth in Advertising Outside “experts” seldom have the “answer” so the learning outcomes for this presentation are : • A better understanding of learning outcomes • An appreciation for what we have done at Maryland • Some ideas of how to adapt what we did to issues at HKU “where applicable”

Learning Outcomes (LO) • LO answer the following guiding questions: • What will my students know? • What will my students understand? • What will my students be able to do? • What will my students be able to appreciate? • LO are determined by the faculty

Learning Outcomes • Can be developed for and applied to: • Graduation requirements • General education • Programs/courses • Majors and minors • Academic components • Set of courses • Individual classes • Individual assignments

Characteristics of Effective Student Learning Outcomes • Are student-focused vs. content or professor-focused • Are focused on learning vs. the learning activity • Are meaningful to faculty and students • Are general enough to capture important learning • Are clear and specific enough to be measured • Are part of the course learning activities and assessments • Are readily measurable with a sample of student work • Are things that good teacher have always done Effective Learning Outcomes NEED to MAKE SENSE

Bottom-line • Learning outcomes are measurable changes in student knowledge, skills and attitudes/appreciation (KSAs) as the result of a course of study e.g. learning • They are determined by the faculty

Why Develop LO? • Required by external body • Required by internal body • To facilitate informed (evidence based) improvements • To help in envisioning and developing a course or program • To answer the question “what works”

Background I: Accreditation • Middle States Accreditation Standard 14: Assessment of Student Learning : 1. Develop clearly articulated learning outcomes: knowledge, skills, and competencies 2. Provide purposeful opportunities for students to achieve those learning outcomes; 3. Assess student achievement of those learning outcomes 4. Using the results of those assessments to improve teaching and learning and inform planning and resource allocation decisions. • 2006-07 Middle States Report • 50% of the areas of concern (failure) were in student learning issues

Background II: US National Trend • Increased Accountability • 2006 Spelling Commission Report: The Future of Higher Education • US Department of Education calling for increased accountability within higher education • Similar signals were being sent at the state level • If universities don’t not take responsibility for assessing the quality of undergraduate learning it could be imposed

65,000 USD

The University of Maryland • Large public research intensive comprehensive university • ~25000 undergraduate students • ~10,000 graduate students • 13 colleges and schools • ~400 programs of study • Ranked 18th US public universities • Ranked 9th as Kiplinger's best value • Ranked 37th in world by Jiao Tung ranking • A core mission is to improve undergraduate education • “One can not be a great university unless one has great undergraduate teaching” Provost Nariman • Recruitment and retention of the best students

LOA Timeline UMD • AY 2004-05 formation of Provost’s Commission on Learning Outcomes Assessment • AY 2005-06 - all programs submit LO, Middle State accreditation • AY 2006-07 - Start of 4-year cycle assessment schedule; assessments results • AY 2007-09 - assessment schedule; assessments results, impacts • AY 2008-09 - assessment schedule; assessments results, impacts • AY 2009-10 - assessment schedule; assessments results, impacts • AY 2010-11 – Fall 2010 all programs complete cycle, restart of cycle

The Learning Outcomes Assessment Circle Faculty Measure LO, Data Collection, Analysis 2006-2010 Faculty Develop 3-5 Learning Outcomes (LO) 2005-2006 Evidenced-base program improvements Ongoing

Key Components for Success • High level support • Provost and Deans • A clear understanding and message that this is about improving the university not clearing an administrative hurtle • An effective and persistent administrative structure • Clear guidelines and deadlines • Support • Workshops • Examples • Website • https://www.irpa.umd.edu/Assessment/LearningOutcomes/ • Faculty driven and faculty empowered

Implementation Structure The Provost’s Commission on Learning Outcomes Assessment • Chaired by the Assoc. Provost and Dean for Undergraduate Studies under the authority of the provost. • A 25-member committee of College Coordinators • Coordinators selected by college deans • Faculty working groups in colleges and departments. • A steering committee of 6 college deans. • A planning team that meets regularly (5 members). • Dean of Undergraduate Studies. • Office of Institutional Research, Planning, and Assessment. • Center for Teaching Excellence. • Faculty

Critical Component: College Coordinators Committee • Seminar-style regular meetings with: • Presentations of work that faculty were submitting to them. • Collaborative critique of that work. • Opportunities to learn from other disciplines. • Recognition for faculty/department accomplishment. • Benefits of peer review. • In April 2006 the Coordinators completed a comprehensive written review of 383 plans, discussed their findings, and forwarded them to the provost. • This group reviews the LO assessment work of the colleges • Annual report to the provost via the Dean of Undergraduate Studies • The University developed a large group of local LO experts.

Results from an Anonymous Survey of the Coordinators • What was the most important experience you had in working with your college? • The language of evaluation has changed in my college. • There is a larger sense of a shared commitment to our students. • There is a shared sense of the value of articulating the learning outcome goals. • Faculty have been very cooperative. • I have shared good ideas from other disciplines with my college. • Communication across campus with different disciplines increased my understanding.

LO Sample (Biochemistry) • Students should have mastered the critical knowledge at each level in the curriculum that is necessary to move on to the next level in the curriculum • Students should demonstrate an ability to use and apply quantitative methods, • Students at the upper level should be able tointegrate and apply basic knowledge to the evaluation of existing scientific studies and to the design of studies to test specific hypotheses in a specific field of the chemical and life sciences. • Students should effectively communicate in writing the processes of science and the results of scientific inquiry.

LO Sample (General Business) • Students will be able to demonstrate a clear understanding of important concepts in the core business disciplines and in the field of General Business. • Students will demonstrate critical reasoning and written communication skills through the individual analysis and write-up of a business case. • Students will demonstrate their oral communication skills by presenting an analysis of a business case to their class. Students will demonstrate their leadership skills by leading a class discussion on a business case. • Students will demonstrate their abilities to work effectively with other members of a team in the preparation of a group project.

LO Sample (Journalism) • Students will demonstrate written and oral communications skills through a high level of ability to report relevant news stories • Students will demonstrate an understanding and awareness of the history of journalism and its role in a democratic society. • Students will demonstrate an understanding and awareness of the ethical guidelines and practices • Students will demonstrate the ability to apply tools and technology • Students will demonstrate the ability to conduct research and evaluate information by methods appropriate to the profession. • Students will demonstrate the ability to apply basic numerical and statistical concepts appropriate for the profession.

Measurement Example (Biological Sciences) • Pretest in required courses • Measures if students learned necessary content • Faculty committee selects questions • Given via course management platform • Embedded quantitative questions scored in upper level courses • Writing assignments in upper level courses • A sample of 10% (~70 papers will be scored) using a common rubric

Measurement Example (Journalism) • Selected student writing samples from lower and upper level classes scored against established criteria rubric by faculty panel • 550 students scored from 36 course sections • > 90% of students scored 2 or better (scale of 1-3) in designated criteria • Identified that math literacy was weak based on sample • Identified that technology literacy was not an issue

Bottom-Line • Clear signal and strong support from upper administration • Faculty driven and faculty empowered • Outcomes and assessments determined by faculty • Products vetted by faculty • No one size fits all (flexibility and listening to faculty) • A clear administrative process with guidelines, goals, time-lines • Maryland is a national model for learning outcome assessment

Successes • All undergraduate and graduate programs (~400) have developed, submitted LOA, which have been faculty reviewed • A process for measurement and review is in place • There is significant cohort of LO experts across all sectors of the university • There is increased awareness and engagement of LO across the university • There is a change in culture with respect to LO and program assessment

Examples of Changes in the Campus Culture • All GE (CORE) courses are reviewed based on their alignment with the CORE learning outcomes • The First Year Book program includes learning outcomes • The Marquee Courses have a common set of learning outcomes • All new academic initiatives have embedded LO as part of the review process

CORE General Education Outcomes • After completion of CORE Program requirements students should be able to: Demonstrate understanding of major findings and ideas in a variety of disciplines beyond the major; • Demonstrate understanding of methods, skills, tools and systems used in a variety of disciplines, and historical, theoretical, scientific, technological, philosophical, and ethical bases in a variety of disciplines; • Use appropriate technologies to conduct research on and communicate about topics and questions and to access, evaluate and manage information to prepare and present their work effectively to meet academic, personal, and professional needs; • Demonstrate critical analysis of arguments and evaluation of an argument's major assertions, its background assumptions, the evidence used to support its assertions, and its explanatory utility; • Understand and articulate the importance and influence of diversity within and among cultures and societies; • Understand and apply mathematical concepts and models; and • Communicate effectively, through written and oral communication and through other forms as appropriate.

First Year Book Program • A reading program directed at incoming students • http://www.firstyearbook.umd.edu/ • Learning Outcomes • Students are able to identify the main issues in the First Year Book • Students understand the relationship between the issues from the First Year Book and their academic subjects • Students are able to articulate a well thought out opinion on the issues raised by the First Year Book

The Marquee Courses in Science and Technology • New Non-major GE Science Course Initiative • Six new courses, focused on real world problems • More the 600 students in the combined courses • Taught by nationally recognized research faculty • http://www.marqueecourses.umd.edu/index.html

Marquee Science Course Learning Outcomes • Ask good questions (sense-making questions; e.g. questions that lead to increased understanding) • Relate science to a personal situation (Science is around them in their everyday life) • Find information using various sources and evaluate the veracity of the information (e.g. information literacy) • Look at complex questions (e.g. global warming, medical technology, biodiversity) and identify the science in the question and how it impacts and is impacted by political, social, economic, and ethical dimensions • Critically evaluate science arguments (e.g. those that are made in a news article, a student presentation, on a TV show, presented to a lay person by a physician etc) • Determine what they know and what they do not know. (Learn how to learn) • Communicate effectively ( to a variety of target audiences and within team situations) – engage in conversation with staff on Capitol Hill, explain a concept to peers).

Challenges • Initial faculty push back: • “you can’t measure learning in my discipline” • “this is another administrative fad, like strategic plans it will have no affects, so ignore it” • “we are already measuring student learning, it called grades” • “I am already over worked with research and teaching I don’t have time” • “ we don’t have the resources to do this” • “it will be use to rate teaching/programs” • “ this will be used to take away program resources” • etc, etc, etc

The Graduate Programs • Initially difficult to get on board • Especially true for Ph.D. programs • The thesis is an individual piece of scholar work • Changed when faculty saw this as a potential solution to an unpleasant problem and as way to improve their programs. • Facilitated by a new dean of the graduate school (e.g. top down support is essential)

Ph.D. Candidacy Exam 1. Demonstrate knowledge of a recognized content area in the discipline by: • Drawing from the literature basic knowledge from the student’s chosen discipline category • Formulating an argument/hypotheses/research project on the basis of this knowledge • Demonstrate knowledge of scholarly materials related to the chosen area of study • Be able to articulate connections between primary and secondary sources and show awareness of major issues in the chosen area of study 2. Data: • Likert scale (1-5) evaluations conducted at the end of the exam by the student’s committee

The LOA Circle Measure 1-2 LO, Data Collection Analysis 2006-2010 3-5 Learning Outcomes 2005-2006 Evidenced-base program improvements Ongoing

Lessons Learned • Get top-down and bottom-up buy in • Establish a process with sufficient people power • Set clear standards, goals, and time lines • Empower the faculty • Needs to be faculty driven, vetted, and embraced • Use what is already present in the classes • Must be of value to the faculty • Must result in program improvement • Needs to be on-going • Needs to be flexible, no administrative straightjackets • Provide support • Be patient: change take time : nothing is a 100%

The Alternative: Herding Cats • http://clips.ksl.com/kickapps/_Cat-Herders/video/85716/5124.html

Thank You https://www.irpa.umd.edu/Assessment/LearningOutcomes/ QUESTIONS/COMMENTS?

Contact Information Spencer Benson, Ph.D. Center for Teaching Excellence University of Maryland, College Park, MD USE Email: sbenson@umd.edu http://www.cte.umd.edu "Teaching is leading students into a situation in which they can only escape by thinking" Spencer Benson, Ph.D. Fulbright Scholar University of Hong Kong Email: sbenson@hku.hk Phone 852 2299 0354 http://www.cte.umd.edu "Teaching is leading students into a situation in which they can only escape by thinking"

"Teaching is leading students into a situation in which they can only e scape by thinking"