Using PLA to Liberate Learning (PLA: participatory learning approach)

Using PLA to Liberate Learning(PLA: participatory learning approach) Michael Bieber, Jia Shen, Dezhi Wu, Vikas Achhpiliya Information Systems Department College of Computing Sciences New Jersey Institute of Technology http://web.njit.edu/~bieber November 2003 1

Outline • Motivation • PLA: Participatory Learning Approach • A bit of theory • Experimental results • Interesting issues 2

Motivation • To increase learning of course content • Learning through active engagement • involve students as active participants • with the full problem life-cycle • through peer evaluation • Minimize overhead for instructors 3

PLA Process All entries posted on-line • Each student creates 2 exam problems • Instructor edits the problems if necessary • Each student solves 2 problems • Students evaluate (grade) the solutions to the problems they authored, writing detailed justifications • Ph.D. students evaluate each problem a second time • Instructor gives a final grade • optional: Students can dispute their solution’s grade, by evaluating it themselves and writing detailed justifications • Instructor resolves the dispute 5

Instructor Control Process Student Learning Process Course Design Process Flow: Learning from doing the PLA activities Make up problems Set up on-line environment Read- other problems - other solutions - grade justifications - disputes additional learning from reading everything peers write Solve problems Exam Process Control Assign ID Edit questions Assign who answers questions Assign level-2 graders Level-1 and Level-2 graders grade solutions Determine Final Grades Resolve Disputes Dispute final grade 7

Instructor Control Process Student Learning Process Confirmation ID, understand process Course Design Make up problems Set up on-line environment Read- other problems - other solutions - grade justifications - disputes Solveproblems Exam Process Control Assign ID Edit problems Assign who solves problems Assign level-2 graders Level-1 and Level-2 graders grade solutions Determine Final Grades Resolve Disputes Dispute final grade 8

Evaluation (grading) • Evaluation includes: • Written critique or “justification” (positive or negative) • Optional: separate sub-criteria to critique • Solution result is correct and complete (40%) • Solution was well explained (30%) • Solution demonstrated class materials well (10%) • Solution cited appropriate references (20%) • Grade (optional; recommended to save instructor time) • Evaluation/grade may be disputed (optional) • Student must re-evaluate own solution when disputing 9

Instructor should provide… • Detailed instructions and timetable • Solution: what is expected • Critiquing and grading guidelines 10

Constructivism(Learning Theory) • The central idea is that human learning is constructed, that learners build new knowledge upon the foundation of previous learning{learning throughout the exam process} • Two classic categorizations • Cognitive Constructivism (Piaget’s theory) • Social Constructivism (Vygotsky’s theory) 12

Cognitive Constructivism (Piaget 1924) • Knowledge is constructed and made meaningful through individual’s interactions and analyses of the environment. --> knowledge is constructed in the mind of individual • Knowledge construction is totally student-centered. 13

Learning • Learning is a constructivist, often social activity occurring through knowledge building (Vygotsky, 1978) • Knowledge building activities include contributing to, authoring within, discussing, sharing, exploring, deploying a collective knowledge base (O’Neill & Gomez 1994; Perkins 1993). 14

Learning • People learn as they navigate to solve problems (Koschmann et al, 1996) and design representations of their understanding (Suthers 1999) • Learning requires cognitive flexibility (Spiro et al. 1991), and results from interaction with people having different experiences and perspectives (Goldman-Segall et al. 1998) 15

Expert-like Deep Learning • Categorizing knowledge and constructing relationships between concepts are likely to promote expert-like thinking about a domain (Bransford 2000). • To design appropriate problems for their peers, students must organize and synthesize their ideas and learn to recognize the important concepts in the domain. • This results in deep learning(Entwistle 2000): • seeing relationships and patterns among pieces of information, • recognizing the logic behind the organization of material • achieving a sense of understanding 16

Where is Knowledge Constructed in PLA? • In all PLA stages:constructing problems, solutions, grade justifications, dispute justifications • When reading everything their peers write • Students also are motivated to learn more when peers will read their work (McConnell, 1999). 17

Assessment & Learning • Main goals of tests: • To measure student achievement • To motivate and direct student learning • The process of taking a test and discussing its grading should be a richly rewarding learning experience (Ebel and Frisbie 1986) • Assessment should be a fundamental part of the learning process (Shepard 2000) 18

Course Information NJIT CIS677: Information System Principles • Graduate level core course (Masters/Ph.D.) • Aim: study how IS/IT can be used effectively • Both on-campus and distance-learning sections • software: Virtual Classroom/WebBoard • Traditional Exam: • Three-hour, in class, 3-4 essay questions, 6 pages of notes • Used PLA 5 times between Fall 1999 and Summer 2002 • We compared control groups without PLA and treatment groups with PLA • Also, we used with shorter essay questions in CIS 365, undergraduate course on file structures in Fall 2002, with similar survey results. 20

Enjoyability Cronbach’s Alpha=0.68 SA - strongly agree (5 points); A - agree (4); N - neutral (3); D - disagree (2); SD - strongly disagree (1); the mean is out of 5 points; S.D. - standard deviation 21

Perceived Learning Cronbach’s Alpha=0.88 22

Recommendation: Do Again! Similar results for CIS365: undergraduate file structures course using short essay questions (Fall 2002) 23

What students liked best • Active involvement in the exam process • Flexibility • Reduction in tension 25

Trade-offs • Trade-offs for students (traditional vs. PLA) • Timing: Concentrated vs. drawn-out (2.5 weeks) • Access to information: limited vs. the Internet • Experimental integrity: we couldn’t justify the process to the students fully • Trade-offs for professors • Fewer solutions to evaluate, but each is different • Timing: Concentrated vs. drawn-out process • Much more administration 26

Timing • PLA for exams took 2.5 weeks • For frequent activities PLA processes could overlap • e.g., quizzes, homeworks • Students could be creating problems for one quiz,while solving problems for the prior quiz, while evaluating solutions from the quiz before that • Benefits to overlapping PLA activities: • working with materials from several classes at the same time • could reinforce class materials • could result in synthesis (combined understanding) 27

Scope • Which activities? • so far: exams • what about: quizzes, homeworks, larger projects, in-class projects • Which problem types? • so far: short and long essay questions • what about: multiple choice, short answer, computer programs, semester projects • Sub-problems: • computer program design & implementation • semester project outline & execution 28

Scope, cont. • Course Level • Graduate, undergraduate, secondary school (high school, junior high) • Disciplines • IS/IT, business, science, engineering, humanities, medical, all of secondary school 29

Scope, cont. • Degree of Evaluation (assigning grades) • Currently: solutions • What about: • quality of problems • quality of evaluations/grades • All could be disputed • Degree of Participation • students could evaluate each • students could arbitrate disputes 30

Full Collaboration • Groups for: • Problems, solutions, evaluation, dispute arbitration • Requires group process support • Group roles: leader, scheduler, etc. • Process: work on each activity together or separately, internal review • Grading of individual group members • Process Tools: brainstorming, voting, etc. 32

What can go wrong • Students are late; students drop the course • Entries posted in wrong place • Inadequate critiques • “Good” • “I agree with the other evaluator” • and of course, technical difficulties… 33

PLA Environment Software • Guide the process • Form groups • Assign problem solvers, evaluators, dispute arbitrators • On-line templates to ensure full entries • Guide people to post entries in correct place • Incorporate group process tools • Handle problems as much as possible • Remind people who are late • Reallocate who does what • Based on a workflow management tool… 34

Anonymity/Privacy Issues • Should student entries be anonymous? • Will students reveal their IDs? • Is it fair to post critiques if not anonymous? • Is it fair to post grades if not anonymous? • Will anonymity work in small classes? 35

Issue: Perceived Fairness • Should students evaluate/grade peers? • But they must evaluate others in the workplace… • It’s the instructor’s job to evaluate and grade • PLA is a (constructivist) learning technique • Students have no training in evaluation • Evaluation is a skill that must be learnt (and taught) • Many evaluators = inconsistent quality • safeguards in the PLA process 36

Grading Issues • Disputing high grades: • Award bonus points if students dispute (and justify with a critique) grades that are too high • Encouraging honest grading: • For successful disputes, deduct points from evaluators 37

Grade Inflation • Detailed grading guidelines for sub-criteria: • great: 20 points • very good: 18 points • good: 14 points • OK: 10 points • poor: 6 points • Student does “good” on 5 problems, grade = 70 • U.S. students will protest vigorously • Evaluators will hesitate to assign “good” • Result: pressure for highly skewed grading rubrics 38

Other Cross-Cultural Issues • In some cultures: • Students are so competitive, they would only give failing grades to peers • Students would not hurt peers’ feelings, and would only give good evaluations • Some systems only have pass/fail, so numeric grades are mostly irrelevant 39

Thank you! Questions, please? PLA: Contributions • Systematic technique to increase learning • Constructivist approach, actively engaging students in the entire problem life-cycle • Minimizes overhead for students and instructors • Experimental evaluation • Supporting software • PLA liberates learning from its traditional instructor-controlled structure! 40

Using PLA to Liberate Learning (PLA: participatory learning approach)