Design Treatment Presentation Fall 2005 T/TAC Online Project VA Assessments December 15, 2005

1. Design Treatment PresentationFall 2005 T/TAC Online Project VA AssessmentsDecember 15, 2005 Matt Humphrey Aala’a Mashaal Pamela Sharpe Tim Smith Maria Washington Graduate School of Education George Mason University

2. Our Faculty and Facilitators • Dr. Nada Dabbagh • Dr. Kevin Clark • Dr. Mike Behrmann • Gary Dickelman • Jocelyn DelSignore • Anya Evmenova

3. Clients & Information Sources

4. Part One Part Two Part Three Part Four Part Five Immersion Team Process Instructional Design Process Background & Introduction Mission and Goal of Project Analyses Needs Analysis Performance Analysis Learners Analysis Task Analysis Findings and Results Design Approach Development Next Steps Questions and Answers Presentation Overview

5. Immersion Team Process • The Learning Process • Teamwork • Portfolios and Reflections • Challenges • Benefits

6. The Instructional Design Process • Who • Learner • What • Learner needs • Instructional Strategies • How • Evaluation

7. Instructional Design ProcessIntegrated Learning Design Framework

8. Develop Implement Evaluate Analysis Design Instructional Design ProcessThe ADDIE Model

9. Project Background • No Child Left Behind Act (NCLB) • Individuals with Disabilities Education Improvement Act of 2004 (IDEA) • Federal Mandates and VA Assessments • Standard of Learning (SOL) Assessment • Alternative SOL (VGLA and VSEP) • Alternate SOL (VAAP)

10. Training/Technical Assistance Centers (T/TAC) • T/TAC Onlineis a web-based community linking people and resources to help children and youth with disabilities http://ttaconline.org

11. Project Problem StatementVA Assessments • Transition to Cognitive Standards Based Assessments • Communication Skills • State Mandated Assessments • New Process for Teachers

12. Analysis Develop Implement Evaluate Design Analysis Needs Analysis Performance Analysis Learner Analysis Task Analysis

13. Needs AnalysisOverview The process of determining, analyzing, and prioritizing needs and, in turn, identifying and implementing solution strategies to resolve high-priority needs.

14. Needs AnalysisPurpose • Determine the best solution for T/TAC Online • Actuals and Optimals • Generate data to support in making a recommendation • Provide specific recommendations • Decide priorities for action

15. Needs AnalysisSteps Taken • Determine Current Status • Performance Analysis • Analysis of Results • Teacher Expectations • Identify Discrepancies • Determine the Goals • Set Priorities for Action

16. Performance Analysis Overview • Used to determine what needs to be done in order to accomplish the client’s stated goals and objectives • Focuses on determining the elements that support the project • Investigates the current situation (actuals) and desired performance (optimals)

17. Performance Analysis Results Initial Actuals • Teachers were not using standardized procedures for assessments • Teachers did not have sufficient access to resources on the VA DOE website • Teachers were not targeting the Aligned Standards of Learning

18. Performance Analysis Results Initial Optimals • Teachers will use standardized procedures for assessment • Teachers will have easy access to resources • Teachers will target the ASOLs for assessment

19. Learner Analysis A learner analysis is performed to determine who the users will be, what skills and knowledge are needed by the learners, the environment in which they work, and their motivations and goals for performing.

20. Learner Analysis Why Create Personas? • Personas identify the user motivations, expectations and goals responsible for driving behavior. • Although personas are fictitious, they are based on knowledge of real users. Some form of user research is conducted before they are written to ensure they represent end users rather than the opinion of the person writing the personas. • There are many varying personas that represent our users.

21. Learner AnalysisHow We Developed Our Personas Using the data collected from the surveys and interviews with SMEs, teachers, and administrators we found that our users were: • Novice Special Education Teacher • Intermediate Special Education Teacher • Expert Special Education Teacher • Reluctant Special Education Teacher • T/TAC Online Administrator • Parent/Guardian

22. Needs AnalysisData Collection • Reviewed the VAAP Manual • Attend conferences to observe teacher’s reactions • Administer surveys • Visit schools to interact with special education teachers • Interview: • Dr. Behrmann • Mary Wilds • John Eisenberg • Jocelyn DelSignore • Anya Evmenova

23. Needs AnalysisSurvey • Gathered information in the following topic areas: • Demographics • Technology skills • Attitudes toward the VAAP • Attitudes toward new assessments

24. Needs AnalysisFindings – New VAAP Figure 1 • 85.6% of the teachers indicated that they agree with the assessments. • 67% of teachers surveyed indicated that they spent a lot of time assessing the students. • 61% of teachers surveyed agreed that they have five or more tools for assessments Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

25. 77% 74% 72% 75% 79% 62% 50% Needs AnalysisFindings - Assessment Figure 2 Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

26. Needs Analysis Findings - Age Ranges Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

27. Needs AnalysisFindings - Years of Teaching Experience Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

28. Needs AnalysisFindings - Licensure Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

29. Needs Analysis Current Situation/Actuals • Special Education teachers have not been doing assessments for their students in a standardized format • Special Education teachers’ collaboration outside the classroom is limited • The assessment documents are difficult to locate on the Virginia Department of Education website • The challenge of assessing special education students creates a barrier • Special education teachers of students with severe disabilities have not been required to conduct SOL based assessments • Teachers spend an inordinate amount of time developing portfolios which are not aligned with SOL expectations

30. Needs Analysis Desired Performance/Optimals • For students designated to be assessed under the VAAP, teachers must begin aligning lesson plans to new aligned standards of learning (ASOL) guidelines, which include cognitive skills. • Skills will be measured in the four subject areas: Reading, Math, History & Social Science and Science. • Teachers must develop lesson plans that align to SOL guidelines and will receive a rating of three 3 or above on a scale of 1-4. • Differentiate the SOL enhanced lesson plans that are currently available on the SOL Enhanced area of T/TAC Online for their students assessed under VAAP guidelines.

31. Needs Analysis Desired Performance/Optimals • It is important to note that if a student does not perform well in the four subject areas they may still garner a passing score by performing well in the following communication areas: • Student performance • Context • Settings • Socialization • Support

32. Needs AnalysisDiscrepancies in Teacher Performance • Teachers do not have experience in aligning their lesson plans to the new standards. • Some of the teachers do not have experience assessing using the former VAAP standards. • The greatest discrepancy lies in the issue that the cognitive skill assessment process is new to all teachers in Virginia. • Teachers have limited resources to help them with the assessment process. • Teachers have limited time outside the classroom to dedicate to this process.

33. Needs Analysis Desired Performance/Optimals • Use computer: • Download blank forms • Internet access • Be able to use word-processing software • Be familiar with state and district-wide assessment • Have knowledge of learner’s skill level, needs, and characteristics • Have knowledge/access to proven research based instruction techniques • Be able to develop lesson plans that fit the student’s current stage of learning for a particular skill.

34. Needs Analysis Desired Performance/Optimals • The solution system should: • Assist teachers with assessing their students using the different assessment procedures outlined in the VAAP, VGLA, and VSEP manuals • Allow teachers to access the documents in one location or a “one stop shop” • Facilitate the sharing of lesson plans with other teachers also involved in the process of creating lesson plans • Facilitate the submission of standardized assessment portfolios with streamlined evidences thereby reducing the amount of time teachers spend compiling evidence of student performance • The documents should be searchable for easy access and downloading

35. Needs AnalysisPriorities for Action • Analyze lesson plan process • Determine potential solution based on the needs • Create prototype of the solution system

36. Task AnalysisOverview • The process of analyzing and articulating the kind of learning that you expect the learners to know how to perform • A blueprint of the instructional sequence and it assures the instructional designer that there are no gaps in the instruction

37. Task AnalysisLesson Plan Process • Instruction is focused on the process of creating lesson plans • Support the teachers in aligning them to the new standards • Task analyze the lesson plan process • Understand the process used by a special education teacher • Support those processes in the solution system

38. Task AnalysisInformation Processing Analysis (IPA) • Used most often for procedural and cognitive tasks • First step in decomposing or breaking down a goal into its constituent parts • Identifies what instruction is needed to attain the goal

39. Task AnalysisIPA/Lesson Plan Process • Creating a lesson plan is a procedural task • Break down each step of the lesson plan creation process into its constituent parts • Identify what we need to provide to the teachers to support them throughout the process • Supports will be utilized in the solution system

40. Task AnalysisWhat the Teacher Needs To Do Complete prerequisite tasks • Review present level of performance, assessment data, and IEP team recommendations • Review student profile document • Identify communication skills • Identify the student’s entry points • Plot entry level points and communication skills

41. Task AnalysisWhat the Teacher Needs To Do Complete the lesson plan process • Identify a main theme or event of the lesson plan • Record the date of creation • Choose an ASOL subject matter • Organizing topic, standard used, support in activity context • Problem-solving cognitive task 4. Choose a communication skill that can be incorporated into the lesson • Can be used multiple times throughout the lesson • Problem-solving cognitive task

42. Task AnalysisWhat the Teacher Needs To Do 5. Identify the student tasks and activities • Main tasks, sub tasks, prerequisites, instructional strategies • Rule-using cognitive task 6. Identify the materials needed • Rule-using cognitive task 7. Identify the assistive technology needs • Ensure that all students have appropriate assistive technology throughout the lesson • Rule-using cognitive task 8. Identify the evidence collections materials • Problem-solving cognitive task

43. Task AnalysisChallenges • Is this the only lesson plan process? • Identifying types of cognitive tasks • Identifying appropriate supports for each task • Identifying instructional strategies • What does the final result of each task look like?

44. Task AnalysisLesson Plan Document

45. Solution System Based upon the results of the needs and task analysis phases, our goals for the solution system are to: • Provide scaffolding for teachers to develop lesson plans aligned with the new SOLs. • Produce a searchable VAAP manual. • Provide additional resources for teachers to utilize throughout the assessment process.

46. Develop Implement Evaluate Analysis Analysis  Design Design Performance Analysis Needs Analysis Personas Task Analysis Design Approach Storyboards

47. Design ApproachEPSS Characteristics Electronic Performance Support System (EPSS) • Computer software program that improves performance by: • Reducing complexity • Providing the performance information to perform a task • Providing a decision support system that enables the user to identify appropriate actions

48. Design ApproachEPSS Characteristics • Used on the job • Controlled by the user • Reduce the need for prior training in order to accomplish the task • Supports new constraints on an existing process

49. Design ApproachRationale • Supporting an existing process • Teachers have created lesson plans • New constraints on that process • New standard format for lesson plans is a system requirement • Teacher experience levels vary • Just-in-time support in the classroom

50. Design  Development Analysis Implement Evaluate Design Develop Flowcharts Schemas Prototype