Usability Study Symplicity / JagJobs

1. Usability StudySymplicity / JagJobs By Carl Klutzke, Phani Nallam, Sneh Khatri, and Tiffany Tseng

2. ABOUT : SYMPLICITY / JAGJOBS JAGJOBS Symplicity is a SaaS-based student employment solution, which is branded for IUPUI student use as JagJobs.org.

3. ABOUT : JAGJOBS HAS AN IDENTITY CRISIS

4. USABILITY PROBLEMS: FROM CLIENTS Usability: Have Complains After Use Awareness: JagJobs = On-campus jobs JagJobs = Internship and Full-Time Utility: Postings are not up-to-date

5. RESEARCH QUESTIONS : FOR THIS STUDY How do IUPUI students find out about off-campus and professional jobs that are being posted for them to fill? How usable is the sign up process for the IUPUI student job board? How usable is the process of finding a job on the IUPUI student job board?

6. ABOUT : SYMPLICITY / JAGJOBS

7. ABOUT : SYMPLICITY / JAGJOBS

8. ABOUT : SYMPLICITY / JAGJOBS

9. ABOUT : SYMPLICITY / JAGJOBS

10. ABOUT : SYMPLICITY / JAGJOBS

11. ABOUT : SYMPLICITY / JAGJOBS

12. ABOUT : SYMPLICITY / JAGJOBS

13. METHODOLOGY : RECRUITMENT IUPUI Student Where would you go to find job listings specifically for IUPUI students? “jagjobs.org” or “employment.iupui.edu” Any Other NO RECRUITED Have you searched for a job on that site since the end of the Spring 2018 semester? YES SORRY

14. METHODOLOGY : TEST SESSIONS TIMED TEST TASKS POST-TASK RETROSPECTIVE SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY SEMI-STRUCTURED INTERVIEW (job search experiences)

15. METHODOLOGY : TEST SESSIONS TASK 1 : Find the IUPUI student job board. Use the internet to find a IUPUI student job board that includes off-campus and professional jobs. This task is complete when the job board is visible in your web browser. When you feel you have completed this task, please say so. TIMED TEST TASKS POST-TASK RETROSPECTIVE TASK 2 : Sign into the student job board that you have found. Please sign in to the student job board and find a page that will allow you to search for jobs. This task is complete when the job search page is visible in your web browser. When you feel you have completed this task, please say so. SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY TASK 3 : Look for an off-campus job. Please search for an off-campus job that meets one the following criteria: a permanent degreed position OR an internship This task is complete when your browser displays the qualifications for such a job. When you feel you have completed this task, please say so. SEMI-STRUCTURED INTERVIEW (job search experiences)

16. METHODOLOGY : TEST SESSIONS TIMED TEST TASKS POST-TASK RETROSPECTIVE SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY Please tell us what you were thinking as you were completing this task. What was going through your head? SEMI-STRUCTURED INTERVIEW (job search experiences)

17. METHODOLOGY : TEST SESSIONS TIMED TEST TASKS POST-TASK RETROSPECTIVE SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY SEMI-STRUCTURED INTERVIEW (job search experiences)

18. METHODOLOGY : TEST SESSIONS TIMED TEST TASKS POST-TASK RETROSPECTIVE SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY SEMI-STRUCTURED INTERVIEW (job search experiences)

19. METHODOLOGY : TEST SESSIONS Post-Study Questionnaire: • Current Employment status? • Where would you go to find off campus/professional job listings specifically for IUPUI students? (your previous experience, before this interview) • What device do you use for the search? • If you have used JagJobs/ IUPUI employment website, when did you first came across it? • Before starting the college • During the college, specify the semester_____________ • How did you know about it? Tell us your source • What are the most popular sources for job search among students? (according to you) TIMED TEST TASKS POST-TASK RETROSPECTIVE SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY SEMI-STRUCTURED INTERVIEW (job search experiences)

20. METHODOLOGY : TEST SESSIONS TIMED TEST TASKS Finally, I’d like to discuss your experiences and strategies for finding jobs as an IUPUI student. • How do you currently search for/find information about employment opportunities? • What resources do you use? • Who do you involve in the process? • Where do you look for information? • What information do you look for? • What kind of jobs do you look for? • (eg: on-campus, full-time, co-op, internship....) • What IUPUI resources if any do you use for job searching? • What challenges do you have, if any, when seeking information about job prospects? • What resources or techniques or strategies do you find helpful for seeking information about the same (what has worked well for you in the past)? POST-TASK RETROSPECTIVE SYSTEMS USABILITY SCALE BACKGROUND AND DEMOGRAPHIC SURVEY SEMI-STRUCTURED INTERVIEW (job search experiences)

21. METHODOLOGY : PARTICIPANTS RECRUITED

22. METHODOLOGY : DATA ANALYSIS Qualitative Data for Insights: Task observation notes Post-task retrospectives Questionnaire responses Interview notes Heuristic Analysis

23. METHODOLOGY : DATA ANALYSIS Quantitative Data: Task Success/Failure Time on Task SUS

24. RESULTS: TEST SESSIONS TASK 1 : Find the IUPUI student job board. Use the internet to find a IUPUI student job board that includes off-campus and professional jobs. This task is complete when the job board is visible in your web browser. When you feel you have completed this task, please say so. COMPLETION 6/8 SUCCESSFUL UNSUCCESSFUL : IU SOUTHEAST AVG. TIME 77 SECONDS SEARCH 4 GOOGLE / “IUPUI JOB BOARD” SEARCH 3 ONE.IU / “JOB BOARD” SEARCH 1 ONE.IU / “JAGJOBS”

25. RESULTS: TEST SESSIONS TASK 1 : Find the IUPUI student job board.

26. RESULTS: TEST SESSIONS TASK 2 : Sign into the student job board that you have found. Please sign in to the student job board and find a page that will allow you to search for jobs. This task is complete when the job search page is visible in your web browser. When you feel you have completed this task, please say so. COMPLETION 3/8 SUCCESSFUL AVG. TIME 55 SECONDS “Somewhat inconclusive, because we did not clearly and consistently direct the participants where the task was to begin or end.”

27. RESULTS: TEST SESSIONS TASK 3 : Look for an off-campus job. Please search for an off-campus job that meets one the following criteria: a permanent degreed position OR an internship This task is complete when your browser displays the qualifications for such a job. When you feel you have completed this task, please say so. COMPLETION 4/8 SUCCESSFUL Through JagJobs COMPLETION 4/8 SUCCESSFUL Through External Job search link AVG. TIME 231 SECONDS SEARCH 4 EXTENDED JOB SEARCH SEARCH 4 JAGJOBS

28. RESULTS: TEST SESSIONS TASK 3 : Look for an off-campus job.

29. RESULTS: QUALITATIVE DATA “Looking at this platform for the first time” “Unsure if the particular platform is for on-campus jobs or off-campus jobs” “It’s my first time to do this. I don’t even know there is a job board. I think I probably can find anything on One.IU.” Before I finish my profile there is nothing here. But After I have done, something come out. Why can you just appear everything?!(laught) I was looking for some filter. I didn’t see it, but it is here. It didn’t appear before. I don’t know why. Maybe it has some bug. Other than that it is pretty easy.

30. RESULTS: SYSTEM USABILITY SCALE AVG. SUS SCORE 57

31. RESULTS: HEURISTIC ANALYSIS (just a few of many) PROVIDE FILTERS IN SEARCH The default job search page allows searching only by keyword and location. It doesn’t provide any features to specify job criteria such as on/off campus, internship, full/part time, etc. TITLE CRISIS The website title, as displayed on the browser tab, does not indicate to the user that they are on the JagJobs site. Instead it merely displays the name of the current page on the site: “Home” or “Academic”. NEW LINKS !!! The “extended job search” feature opens a new window to a completely different job search website (http://network.experience.com/), without notifying the user that they are leaving the JagJobs site. SAVE THE SEARCH When your search results include an interesting job listing, you cannot open that job listing in a new tab. If you look at the details of a such a listing, there is no way to return to your previous search to look at other interesting listings, and your search parameters are not preserved: you must start your search over again. CHECKLIST When filling out career interests, the system presents a long multiselect list of options, with a last option of “Not Sure Yet”. When the user clicks “Not Sure Yet” it removes all of the previous selections, undoing the user’s work.

32. FINDINGS: How do IUPUI students find out about off-campus and professional jobs that are being posted for them to fill? LinkedIn JagJobs redirects to a site with poor usability Buried Link Unfamiliar How usable is the sign up process for the IUPUI student job board? The site tries to force the user to fill out a complete profile--a time consuming process--before letting them use the its search features. How usable is the process of finding a job on the IUPUI student job board? Can't re-use saved searches No filter options No employer information Uses different website for job search.

33. RECOMMENDATIONS: PROMOTE JAGJOBS = ONCAMPUS + OFFCAMPUS + INTERNSHIP + CO OP + FULL TIME DIRECTLY JAGJOBS.ORG = WEBSITE PROFILE FILLING GIVING INFORMATION FOR SEARCH = OPTIONAL

34. RECOMMENDATIONS: REMOVE : EXTENDED JOB SEARCH ADD : EMPLOYER INFORMATION FILTERS E.G. FULL/PART TIME, ON/OFF CAMPUS, APPLICATION DEADLINE TEACH WORKSHOPS TO TEACH STUDENTS HOW TO USE JAGJOBS

35. CONCLUSIONS: If students are aware of JagJobs.org, they believe it is only for part-time student jobs. Access to JagJobs.org is forced through an Office of Student Employment web page that has poor usability. JagJobs attempts to force new users to fill out a time-consuming profile before allowing them to use its search feature. JagJobs’ basic search features lack sophistication and could do much to be more useful to the user. JagJobs prominently provides an “Extended Job Search” that is actually a link to a completely different site.

36. CONCLUSIONS: We believe our findings support our clients’ concern that IUPUI is not currently getting good value for its investment in Symplicity. We have recommended some actions that IUPUI can take to make the site better known, easier to access, and more useful. However, we also recommend that Symplicity be compared with other similar services to see if they provide better value.

37. THANK YOU