User Studies II

1. User Studies II With your instructor, Jeremy Hyland

2. Plan for Today • Discuss the reading: • Why Johnny Can’t Encrypt • Johnny 2: Judgment Day • Do a little testing of our own…

3. Why Johnny Can’t Encrypt • Who’s Johnny and why can’t he encrypt? Posner says What’s Johnny trying to hide?

4. Why Johnny Can’t Encrypt • Whitten and Tygar, 1999 • http://www.usenix.org/publications/library/proceedings/sec99/full_papers/whitten/whitten_html/index.html • A Usability Evaluation of PGP 5.0

5. Why Johnny Can’t Encrypt • “Security mechanisms are only effective when used correctly” So: If Usable then else

6. Why Johnny Can’t Encrypt • Defining Usable Security Software • Whitten and Tygar: • Security software is usable if the people who are expected to use it: • are reliably made aware of the security tasks they need to perform. • are able to figure out how to successfully perform those tasks • don't make dangerous errors • are sufficiently comfortable with the interface to continue using it.

7. Why Johnny Can’t Encrypt • Why is usable security hard? McNealy says You have no usable security, get over it.

8. Why Johnny Can’t Encrypt • Why is usable security hard? • Five reasons: • 1. The unmotivated users • “Security is usually a secondary goal” • 2. Policy Abstraction • Programmers understand the representation but normal users have no background knowledge.

9. Why Johnny Can’t Encrypt • Why is usable security hard? • Five reasons: • 3. The lack of feedback • We can’t predict every situation. • 4. The proverbial “barn door” • Need to focus on error prevention. • 5. The weakest link • Attacker only needs to find one vulnerability

10. Why Johnny Can’t Encrypt • Usability Evaluation • PGP 5.0 • Pretty Good Privacy • Software for encrypting and signing data • Plug-in provides “easy” use with email clients • Modern GUI, well designed by most standards

11. Why Johnny Can’t Encrypt • Usability Evaluation • Whitten and Tygar focus their evaluation on a question based off their definition of usable secure software: • If an average user of email feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all? Loaded question?

12. Why Johnny Can’t Encrypt • Usability Evaluation • Cognitive walk through • Mentally step through the software as if we were a new user. Attempt to identify the usability pitfalls. • Focus on interface learnablity.

13. Why Johnny Can’t Encrypt • Usability Evaluation • Cognitive walk through results: • Visual metaphors • Public vs. Private keys • Signatures and verification

14. Why Johnny Can’t Encrypt • Usability Evaluation • Cognitive walk through results: • Different key types • Compatibility increases complexity • Keys listed as users

15. Why Johnny Can’t Encrypt Keys listed as users

16. Why Johnny Can’t Encrypt • Usability Evaluation • Cognitive walk through results: • Key server • Hidden? • What is it doing? • Revocation not automatic Would that help?

17. Why Johnny Can’t Encrypt • Usability Evaluation • Cognitive walk through results: • Key management policy • Unneeded confusion • What’s the difference between trust and validity?

18. Why Johnny Can’t Encrypt • Usability Evaluation • Cognitive walk through results: • Irreversible actions • Need to prevent costly errors • Consistency • “Encoding”?!? • Too much information • More unneeded confusion • Show the basic information, make more advanced information available only when needed.

19. Why Johnny Can’t Encrypt • Usability Evaluation • User Test • PGP 5.0 with Eudora • 12 participants all with at least some college and none with advanced knowledge of encryption • Participants were given a scenario with tasks to complete within 90 min • Tasks built on each other • Participants could ask some questions through email

20. Why Johnny Can’t Encrypt • Usability Evaluation • User Test Results: • 3 users accidentally sent the message in clear text • 7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problem • Only 2 users were able to decrypt without problems • Only 1 user figured out how to deal with RSA keys correctly. • A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted emails. • One user was not able to encrypt at all

21. Why Johnny Can’t Encrypt • Conclusion • If an average user of email feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all? • Nope • Is this a failure in the design of the PGP 5.0 interface or is it a function of the problem of traditional usable design vs. design for usable secure systems? • Security as the primary function vs. a secondary function

22. Johnny 2 • Garfinkel and Miller, 2005 • http://www.simson.net/clips/academic/2005.SOUPS.johnny2.pdf • Follow-up to Why “Johnny Can’t encrypt” • Test of new encryption technology • Key Continuity Management • S/MIME certificates • Better interface • Simple buttons

23. Johnny 2 • Garfinkel and Miller: • Johnny couldn’t encrypt because of the key architecture behind PGP. • “….the fundamental usability barriers that Whitten identified could be overcome by replacing the underlying third-party certification model with Key Continuity Management.”

24. Johnny 2 • User Test • Tried to stay as close to the Johnny experiment as practical • Same methods of user solicitation/selection • Same basic scenario • Similar user tasks • Added attackers

25. Johnny 2 • User Test • Attacks: • new key attack • new identity attack • unsigned message attack • How well does the interface enable users to respond to these attacks?

26. Johnny 2 • User Test • Test application: CoPilot • “Wizard of Oz” prototype • S/MIME certificate handling: • First time = Yellow • Trusted certificate = Green • Changed certificate = Red • Unsigned message = White • Unsigned message from a sender that normal sends signed messages = Gray • Better tools allow for a more automated and scientific test

27. Johnny 2 • User Test • 43 test subjects • Three groups: • No KCM • Color • Color+Briefing

28. Johnny 2 • User Test • Results: • Users generally understood the basics • Little understanding of signature integrity guarantees • Verifying attack message authenticity was difficult for most users • No group resisted attacks 100% of the time • Color and Color+Briefing resisted new key attack and the unsigned message attack better then No KCM • The interface did not help against new identity attacks

29. Johnny 2 • User Test: Conclusions • A few surface interface issues • Do not trust button • Misconceptions about the security of sealed messages • Generally, the new interface simplifies email encryption • Still problems with determining certificate trust, however some of these problems may be unavoidable.

30. So Now What? • Now its time to do your own test!

31. User Test • 3 groups: • Cell Phone • CD player • Calculator • Take a few minutes to create a simple user test • One member of each group switches to be a tester

32. User Test • Guidance: • Decide whose going to do what! • Create a Use Case Scenario • Define user tasks for completion of the scenario • Set up metrics for results evaluation • What qualifies as success vs. failure?

33. User Test • Results!?