Adaptive Learning Systems WORKSHOP Evaluating and Implementing the Right One ( for you)

1. Adaptive Learning Systems WORKSHOP Evaluating and Implementing the Right One (for you) EDUCAUSE Learning Initiative Conference2016 Dale P. Johnson Wayne Anderson

2. Your“Guide on the Side” • Cengage (Learning Objects) – psychology • CogBooks – biology and US history • Kahn Academy – remedial math • Knewton – remedial math • McGraw Hill ALEKS - college algebra • McGraw HillLearnSmart Master - remedial math • McGraw Hill LearnSmart - chemistry • Pearson with KnewtonMyMath Lab - college algebra • Pearson with Knewton Mastering - physics • SmartSparrow – Habitable Worlds custom science course Adaptive software is necessary but not sufficient to ensure student success. • Dale Johnson • Manager, Adaptive General Education Program,  Arizona State University

3. Yourneeds and interests today? • Who’s already working with adaptive learning (AL) systems? • Who’s trying to decide whether to use them? • Who’s evaluating results from their completed project? • Who’s in the wrong workshop? • Get the MOST out of this workshop: ACTIVITY: What do you want to accomplish today? What does your institution want to accomplish?

4. Workshop Learning Objectives • Understand adaptive systems & terminology • Identify challenges and opportunities with adaptive learning systems • Analyze the costs and benefits of adaptive learning systems • Create a plan for implementing and evaluating an adaptive learning system in your organization

5. Workshop Learning Objectives • Understand adaptive systems & terminology • Identify challenges and opportunities with adaptive learning systems • Analyze the costs and benefits of adaptive learning systems • Create a plan for implementing and evaluating an adaptive learning system in your organization

6. Understand adaptive systems & terminology What is the promise of adaptivity? Students ON-Track Students OFF-TRACK Who needs help? What do they need help with? What’s the best way to help?

7. Understand adaptive systems & terminology How are the AL systems different? LMS ADAPTIVE - Lesson Plan Fixed • Variable • Individual • Personalized - Presentation Group - Content Common

8. Understand adaptive systems & terminology How do the systems help students? • Respect their prior knowledge • Respond to their learning needs • Reduce gaps in their understanding 3 Rs Personalize

9. Understand adaptive systems & terminology How do the systems help faculty? • Monitor which students need assistance • Measure curriculum performance • Maximize course outcomes 3 Ms Tracking

10. Understand adaptive systems & terminology What should we expect to gain? Learning Benefits from these Pedagogical Techniques* • Formative Evaluation (d=.90) • Acceleration (.88) • Effective Feedback (.73) • Meta-cognition (.69) • Mastery Based Learning (.58) • Concept Mapping (.57) • Interactive content (.52) *Source: John Hattie’s Visible Learning (2008) 15 year, 800+ meta analysis on achievement Standard deviation is effect size where d = 1.0 (i.e. improvement of learning by at least 50%) Average effect size d = .40 • When d is > .40 there were excellent learning gains

11. Understand adaptive systems & terminology Where are we in the technology cycle? Adaptive learning systems are still experimental. WE ARE ALL HERE!

12. Understand adaptive systems & terminology What is adapting to the student? What is guiding the adaptation? • Concept • Content • Algorithm(analytics) – recommendations • Assessment – rapid remediation • Assignment – decision tree • Agency – student chooses

13. Understand adaptive systems & terminology What do we mean by Concept? • Concept (or Lesson, Skill, Topic, Unit, Module, etc.) • Vendors use different terminology • Example: College Algebra

14. Understand adaptive systems & terminology How are the Concepts organized? • Concept Chart • (or Knowledge Map, Skill Graph, etc.) Algebra Calculus Basic Math Course Configuration Concept 1 C-2 C-5 C-7 C-x C-4 C-3 C-6

15. Understand adaptive systems & terminology How do students experience Concept adaptivity? Course Configuration Student Lesson • AlgorithmAssessment • AssignmentAgency Concept Objective Assessments Objective Concept Concept Assessments Content Content Content Instructional Materials Instructional Materials Content Practice Problems Practice Problems Assessment Instruments Assessment Instruments

16. Understand adaptive systems & terminology What is an example of Concept adaptivity? Khan Academy ACTIVITY: Go to https://www.khanacademy.org/exercisedashboard

17. Understand adaptive systems & terminology How is Content adapting to the student? Content (or Resource, Material, Text, Video, Exercise, Problem, etc.) Learning Loop Concept #1 Concept X Algorithm Assessment Assignment Agency Pre-Assessment Content Content Algorithm Assessment Assignment Agency Algorithm Assessment Assignment Agency

18. Understand adaptive systems & terminology Whatis an example of Content adaptivity? • Algorithm • Assessment • Assignment • Agency CogBooks uses all four models:

19. Understand adaptive systems & terminology What does all this mean to the student? ASU MAT142 August to December 2012 Concepts completed by each student Median • Progress can depend on performance! • NOT JUST THE SCHEDULE

20. Workshop Learning Objectives • Understand adaptive systems & terminology • Identify challenges and opportunities with adaptive learning systems • Analyze the costs and benefits of adaptive learning systems • Create a plan for implementing and evaluating an adaptive learning system in your organization

21. Identify challenges and opportunities with adaptive learning systems What are the major challenges and decisions? ACTIVITY: Build your plan as we go… • Pedagogy • System • Course Development • Faculty • Facilities • Evaluation

22. Identify challenges and opportunities with adaptive learning systems How can the system support your Pedagogy? • - homework and assessment • - knowledge delivery and assessment • - knowledge, assessment and homework • Lecture • Flipped • Online • Other?

23. Identify challenges and opportunities with adaptive learning systems What other Pedagogical decisions matter? • Self-paced or Synced • Complex tracking Lesson alignment • Flexible exam dates Fixed exam dates • Independent Interactive • Competency Completion Faculty need to make these decisions first. • Student learning

24. Identify challenges and opportunities with adaptive learning systems What Pedagogy strategy did ASU decide upon? Our best results have been with “Flipped and Synced” Options we’ve tried: Online and Self-paced Flipped and Self-paced Flipped and Synced Active Learning in class Adaptive Learning before class Bloom’s Taxonomy

25. Identify challenges and opportunities with adaptive learning systems Howdoes this work in practice? ACQUIRE KNOWLEDGE Read textbook, watch video, do simulation, etc. ANALYZE Do practice problems, take quiz before class ADAPTIVE SYSTEM ACTIVE CLASS ASSIMILATE Write essay, solve problems, take quiz, etc. APPLY Discuss an applied problem (case study) with classmates.

26. Identify challenges and opportunities with adaptive learning systems Whatdoes a course look like? Biology and Society Evolution Physiology Energy Photosynthesis Genetics and DNA Scientific Method Biodiversity Body Systems Lab Lab Lab Lab Lab Lab Exam Exam Exam ALS is only part of the education ecosystem.

27. Identify challenges and opportunities with adaptive learning systems Whatare the major System components? Student UX Instructor UX Lesson Reporting Reporting Grouping Adaptive Assessments Resources Activities Etc. Information Architecture Analytics Content Curation Course Creation Add Associate Add Sequence

28. Identify challenges and opportunities with adaptive learning systems What are some key System selection issues? • Construct or Configure • Time consuming Quick • Flexible Constrained • Costly Cheap • Riskier Safer Consider the commitment before committing. • Course Creation

29. Identify challenges and opportunities with adaptive learning systems How are different Systems implemented? Configure Course Creation – Construct Knewton Knowledge Graph C2 C5 C8 Concept 1 C4 Cx C3 C7 C6 The market is moving toward modularity.

30. Identify challenges and opportunities with adaptive learning systems How does the System handle different subjects? Adaptive logic may not be the same for all subjects. • Course Creation – Subject Matter • Math Model • Pretest, practice, posttest • Predicated on 12 years of prior math education • Other Disciplines • More traditional presentation • Learn, assess, remediate

31. Identify challenges and opportunities with adaptive learning systems How will you manage content in the System? Will the vendor allow outside content. • Content Curation • Vendor Resources • Prebuilt courses • Library of materials • Other Resources • Faculty produced content • Open Educational Resources (OER)

32. Identify challenges and opportunities with adaptive learning systems How will you manage Course Development? Faculty Instructional Designers Technologists Vendor Personnel Instructional Designers Video Producers Video Producers Video Producers Instructional Designers Video Producers Instructional Designers Others? Others? Others? Vendor Personnel Others? Vendor Personnel Vendor Personnel Graphic Designers Graphic Designers Graphic Designers Graphic Designers Faculty Faculty Faculty Librarians Librarians Librarians Librarians Technologists Technologists Technologists Adaptive tech is a team sport, so plan accordingly.

33. Identify challenges and opportunities with adaptive learning systems What will help Faculty succeed with AL? • Strong provost and departmental support (and $) • Faculty leadership for each course • Instructor peer mentoring for training • “Guide on the Side” is not for everyone • Be patient, it’s a learning process The technology success depends on the teacher!

34. Identify challenges and opportunities with adaptive learning systems What types of classroom Facilities will you need? • How many rooms? • How big do they need to be? • How long will they take to build? • How much will they cost? • Who is going to pay for them? • How will you support them? Think the project all the way through.

35. Identify challenges and opportunities with adaptive learning systems What factors could you Evaluate? • Macro indicators • Persistence (lower student withdraw rate) • Performance (higher student pass rate) • Satisfaction • Student • Instructor • Administrator • Financials • Money saved or spent • Micro indicators • Assessment results (lesson or exam level)

36. Identify challenges and opportunities with adaptive learning systems How does ASU Evaluate our GenEd efforts? • MISSION:Enable Student Success • OBJECTIVES: • Improve critical thinking & problem solving • Increase student subject mastery • Increase student retention • Improve instructor insight Achieve 80% graduation rate • Do Applied Concept Exercise every class • Help 90% of students get C or better • Re-enroll 90% of students next semester Identify struggling students by week 2

37. Workshop Learning Objectives • Understand adaptive systems & terminology • Identify challenges and opportunities with adaptive learning systems • Analyze the benefits and costs of adaptive learning systems • Create a plan for implementing and evaluating an adaptive learning system in your organization

38. Benefits and costs What are the expected benefits (REVIEW)? Learning Benefits from these Pedagogical Techniques* • Formative Evaluation (d=.90) • Acceleration (.88) • Effective Feedback (.73) • Meta-cognition (.69) • Mastery Based Learning (.58) • Concept Mapping (.57) • Interactive content (.52) • When d is > .40 there were excellent learning gains *Source: John Hattie’s Visible Learning (2008)

39. Benefits and costs How has it worked at ASU? • Biology 100 – The Living World (300 non-majors) • Persistence – withdraws fell to 1.5 % • Performance – C or better increased to 94% • Satisfaction Student course evaluations improved Professor said, “I’m never lecturing again.” • Financials We saved money from not having repeat students. We spent money to put more instructors in the classroom. • Biology 100 – The Living World (300 non-majors) • Persistence • Performance • Satisfaction • Financials

40. Benefits and costs What do the students think? • Rio Salado College student survey results… • 98% of the students liked the adaptive course • 96% felt the system adapted to their needs • 89% stated the adaptive course was more helpful than traditional textbooks **N = 138 students enrolled in ENG 081 and 091 (69% response rate)

41. Benefits and costs What is this going to cost? • Implementation • Construct • Configure • Operation • Implementation • Construct ~ $50,000 • Faculty time ~ 9 months • Staff time ~ 12 months • Systems integration ~ 1 month • Content development and licensing • Configure ~ $5,000 • Faculty time ~ 1 month • Staff time ~ 3 months • Operation Student license $35 - $100 Faculty training – every semester

42. Workshop Learning Objectives • Understand adaptive systems & terminology • Identify challenges and opportunities with adaptive learning systems • Analyze the costs and benefits of adaptive learning systems • Create a plan for implementing and evaluating an adaptive learning system in your organization

43. Create a Plan What is your plan for using adaptive systems? Get the MOST out of this workshop: Mission Objectives Strategy Tactics

44. Create a Plan What is your program Mission? • ASU has 8 university “Design Aspirations” • 1. Leverage Our Place • 2. Transform Society • 3. Value Entrepreneurship • 4. Conduct Use-Inspired Research • 5. Enable Student Success6. Fuse Intellectual Disciplines7. Be Socially Embedded8. Engage Globally Try to connect your efforts to a higher goal. • ASU has 8 university “Design Aspirations” • 1. Leverage Our Place • 2. Transform Society • 3. Value Entrepreneurship • 4. Conduct Use-Inspired Research • 5. Enable Student Success6. Fuse Intellectual Disciplines7. Be Socially Embedded8. Engage Globally

45. Create a Plan WhatObjectives do you want to achieve? • Do Applied Concept Exercise every class • Help 90% of students get C or better • Re-enroll 90% of students next semester Identify struggling students by week 2 Measure your objectives as clearly as possible. • Improve critical thinking & problem solving • Increase student subject mastery • Increase student retention • Improve instructor insight

46. Create a Plan What Strategy will work best for you? “Flipped and Synced” is working for us. Biology and Society Evolution Physiology Energy Photosynthesis Genetics and DNA Scientific Method Biodiversity Body Systems Lab Lab Lab Lab Lab Lab Exam Exam Exam Tailor your strategy to your situation.

47. Create a Plan What Tactics will you use to succeed? • Course • System • Course Development • Faculty • Facilities • Evaluation Try to keep your options open to experiment! • Course – Math or Other? • System – Construct or Configure? • Course Development – Staff and Resources? • Faculty – Selection and Support? • Facilities – New ones needed? • Evaluation – Measurable metrics?

48. Thank You for participating! Dale Johnson Dale.Johnson@asu.edu 480-884-1927