Security Strategies for Every Stage of the Testing Process

1. Security Strategies for Every Stage of the Testing Process DeDeHedlund, Creighton University Jeff Place, Questionmark Evangelist NCTA – Minneapolis, MN Friday Aug. 3, 2012, 11:30-12:30 pm

2. Goals of this Session There are no shortage of test security challenges At each stage of the assessment process (Authoring, Scheduling, Delivery, and Reporting) solutions exist to meet security challenges Let’s Identify the threats to watch out for Show you today how certain product functionality can meet your test security needs by walking through a typical examination process and some real life examples

3. Threats Impersonation Content Theft Cheating

4. Issues Fraud Triangle Threats • Rationalization • Opportunity • Motivation • Impersonation • Content Theft • Cheating

5. Low/High to High/High Stakes Tests What’s at Stake? Higher Stakes Life and Limb Promotion & Jobs & Legal Concern Educational Exams Tests Elearning & Surveys

6. Monitoring Tests Securely ID Fraud Protect Content Minimizes Cheating √ √ √ √ √ √ √ √ √ √ √ √ √ X √ √ X X

7. Providers ID Fraud Protect Content Minimizes Cheating √ √ √ √ √ √ √ √ √ √ √ √ √ X √ √ X X

8. Assessment Management Process

9. Authoring: Item Banking and Security • Security impacts • Validity of assessment results • Fairness to participants • One approach: security rights that are role-based • Set access permissions by role or profile • Limit an author’s or reviewer’s access to the various functions • Limit access to topic (item) folders • Limit what the author may do in those folders

10. Scheduling • Create participants and schedule assessment delivery • Manage participants individually or in hierarchical groups • Schedule access to assessments: • Limit dates • Limit attempts • Require proctor/invigilator • Deliver to certified test center • Online or offline delivery • Email Broadcast:Notifications to Participants

11. Blended Delivery Levels of Monitoring Stakes of Assessment

12. Secure Browser Regular Browser Questionmark Secure • Stops people from printing questions • Stops people from typing in their own URL • Always display in full screen and it’s not possible to maximize or minimize • Stops people starting a new task • Does not provide menu options or icons • Disables control keys and task switching • Disables right-click menu options • Disables the ability to start new applications • Prevents going backwards to a previous page • Stops people exiting in a high stakes, proctored, environment • Hides the HTML source • Provides an API to control certain functions of a browser from the server • Server can command Questionmark Secure to display a toolbar

13. Browser Check Ensures Test Takersuse a compatible browser

14. Delivery Security • Perception can be configured to use SSL to ensure communication between participants and the server are secure • Data in item repositories are encrypted (QML is encrypted) providing added item bank security • All participant scoring is conducted on the Perception server (rather than on client/participant PCs) • Scoring algorithms are not sent in any way to participant PCs

15. Delivery Security Features

16. Limiting Item Exposure • Question by Question (QxQ) delivery options limit the amount of content shown at one time • Randomization of questions and choices to prevent cheating

17. Reporting • Score List Report • IP addresses (as expected?) • Time taken (too quick, too long?) • Easy flagging of participants who get perfect scores for further investigation • Coaching Report • Drill down on an individual participant, comparison to benchmark, use in appeal process • Item Analysis Report • Determine psychometric performance and examine item difficulty drift over time • Test Analysis Report • Assessment defensibility review and test difficulty drift

18. Case studies Case studies from Creighton University • What their test security needs were • How our software addressed those needs • Demo of solutions

19. Creighton University School of Pharmacy & Health Professions • Private Jesuit University located in Omaha, NE • 2001 started the first and only accredited truly distance based full time Doctor of Pharmacy program • Issues to resolve and security requirements • How to securely deliver and administer exams to students located throughout the country • Paper and pencil were too expensive and it’s difficult to distribute and collect approximately 5600 exams per semester • Control who sees what test and when • Lock down the computer to eliminate printing, instant messaging, browsing the internet during the tests • Ability for students to “mark” an identified area of a graphic

20. Creighton University School of Pharmacy & Health Professions Solutions offered to meet requirements: • Provided online testing solution and eliminated almost all paper and pencil tests to reduce costs • 22 Question types including Hotspot allowing online exams to replicate past paper exam formats with the additional ability to provide instant feedback. • Ability to randomly deliver different variations of an exam • Manage large item banks of questions including security for collaborative authoring among faculty • Lock down browser • Scheduling (right test to right student) Demos • Hotspot question type & feedback text/images • Password system & variations of an exam • Item Banking and security for collaborative authoring • Questionmark Secure • Scheduling

21. Demonstration

22. Variations of an Exam

23. Item Banking

24. Creighton University School of Pharmacy & Health Professions Additional security measures • Established guidelines for choosing new proctor sites • Formalized the process of communicating what is expected from a proctor site • Proctor Certification forms are required from every student who completes an exam • Exam specific instructions are emailed to the proctors

25. Remote Video Monitoring Custom Hardware Laptops and PCs

26. Candidate (Participant) Logs in As Software Secure and Questionmark integration is for higher stakes exams a Participant will be referred to as a Candidate

27. Schedule Appointment Exam Candidate selects exam and “Schedules Appointment” Exams (4) Surveys (1)

28. Schedule Day and Time

29. Review Schedule If the exam is to be schedule in the next 3 days or if the participant might be expected to pay some of the cost of the exam then there are more steps

30. Payment Process

31. Candidate Takes Exam When It Was Scheduled

32. Candidate (Participant) Logs in

33. Selects Exam to Take Exams (4) Surveys (1)

34. ProctorU / Software Secure • Camera and Microphone Access • Identity Checks • Captures Photo, Proctor Checks against files • Proctor checks has Participant scan the room

35. Security Checks Complete Candidate Takes Exam • After Authentication Candidate takes Exam • Video and screens are monitored during exam

36. Potential Applications for Mobile Delivery

37. Mobile Test Centers • Use notebooks/3G laptops/iPads to create mobile test centers • Enables greater flexibility in where you deliver assessments • Assessments can be conducted in ‘on location’ environments Slide 10

38. “Securing” an iPad… • An app should prevent candidates from easily getting to other URLs • But – access to “home button” allows task-switching • Hardware solutions are available to enable use of iPad in a “kiosk” mode Slide 11

39. Considerations: • The higher the stakes, the higher the propensity to cheat –still require human monitoring! • Bandwidth – Be sure to test the wifi and/or 3G signal strength prior to event Slide 12

40. Cheat-resistant questions and assessments

41. Considerations for cheat-resistant questions and assessments Randomization Refresh items regularly Build large item banks Track item parameters Tests that are unique for each test taker make it hard to share answers Adaptive tests provide different questions based on responses to previous questions

42. Improve Your Multiple Choice Questions Answer choices should be roughly the same length and kept as short as possible. Provide a minimum of three answer choices and a maximum of five. Four is considered optimal. Keep your writing clear and concise – you’re testing knowledge, not reading comprehension. Make sure that you’re putting the correct answer in the first two positions as often as the last two positions.

43. Avoid Previously Delivered Questions

44. Extended Matching Question Type • The number of answer options depends on the logical number of realistic options for the test taker.) • The same answer choice could be correct for more than one question in the set • Some answer choices may not be the correct answer for any of the questions • So it is difficult the answer this type of question correctly by chance. • A well-written lead-in question is so specific that students understand what kind of response is expected, without needing to look at the answer options.

45. Scenario Based Questions

47. Scan a QR Code with your mobile to start an assessment!

48. Closing and questions • Thank you for your time and attention! • Questions/Comments? • Visit our booth to talk to us more • We have a whitepaper on this topic (Delivering Computerized Assessments Safely and Securely) • Visit our website to download: http://www.questionmark.com/us/whitepapers/index.aspx

49. Thank you for attending! DeDeHedlund dhedlund@creighton.edu Jeff Place jeff@questionmark.com