Quality Assurance and Game Testing

1. Quality Assurance and Game Testing

2. Brief intro • Post. doc., Center for Computer Games Research, IT University of Copenhagen • Don’t pay much, but opportunity for ”blue sky research”  • General expertise with the user experience • Game testing • Game design • Etc. etc. etc.

3. Overview • Primer on empirical research methods • Game Testing 101: General principles • Game testing during the production cycle • With introduction to several key methods Focus: Theory, Practice & Tools

4. A primer onEmpirical Research Methods

5. Overview • Scientific theory • Empirical research approaches • Empirical game studies

6. Scientific theory? What has that got to do with QA? • All technology testing is based on empirical research and evaluaiton methods • To understand what games testing really is, you must understand empirical research approaches • If not: Blind use methods you do not understand

7. Science • Science: Any systematic knowledge or practice. • Science generally refers to a way of acquiring knowledge through the scientific method, as well as the organized body of knowledge gained through such research. • Adheres to positivist philosophy: Only authentic knowledge is scientific knowledge • Science = Logic + Observation

8. Science • Three types of science: • Natural science: The study of natural phenomena • Social science: The study of human behavior and societies • Formal science: Mathematics – uses a priori rather than empirical methods, includes statistics and logic • Two first are empirical sciences, third a mixture, however all feed into each other • A priori = deductive knowledge (independent of experience) • A posteriori = Inductive knowledge (dependent on experience)

9. Science • Experimental science: Another term for empirical sciences • Applied science: Application of scientific research to specific human needs – such as game testing! • The two are often combined

10. Empirical Sciences • Empirical sciences • Knowledge obtained from observable phenomena • Reproduceable: Phenomena must be reproduceable under experimental conditions by other scientits, in order to validated. • Careful, objective and systematic study of an area of knowledge • Must follow the scientific method

11. The scientific method • The scientific method • A body of techniques for investigating phenomena, acquiring knowledge • Collection of data through observation and experimentation, and the formulation and testing of hypotheses • Evidence must be observable, empirical and measureable, subject to principles of reasoning

12. The scientific method • Empirical research must follow: • Define the question • Gather information and resources (observe) • Form hypothesis • Perform experiment and collect data • Analyze data • Interpret data and draw conclusions that serve as a starting point for new hypothesis • Redo entire cycle if necessary • Publish results • Retest (frequently done by other scientists) Alternative: Explorative approach – similar requirements on objectivity and reasoning, but forgoes hypothesis forming.

13. In comparison:The game testing method • Example • Are there any bugs with this feature of game X? • Get the game, set up a lab and assimilate knowledge from other test cases • Hypothesis: There probably are some bugs in our game .... • Run tests and collect test data • Analyze data • Interpret test data and draw conclusions: We found X number of bugs – do we have reason to believe the bugs all been found? • Redo entire cycle if necessary • Publish results to bug database and get designers to fix them • Retest to see if bugs have been fixed • Game testing should always follow the scientific method!

14. Hypothesis (problem, case...) • A hypothesis defines an expected relationship between variables, which can be empirically tested. • For example: • Eliminating the minimap in StarCraft will increase player engagement • Making the bazooka do more damage will balance the weapons in this game • There are no bugs in this level

15. Quantitative vs. Qualitative • Empirical research methods come in two forms: • Quantitative methods: Collect numericaldata, strictlyobjective, analyzed using statistical methods • Qualitative methods: Collect data in the form of text, images, sounds etc. • Drawn from observations, interviews, documentary evidence etc., analyzed using qualitative data analysis methods (e.g. content coding) • Data and analysis can be subjective: Relies on researcher experience

16. Selecting methods • Qualitative: • More appropriate in early stages of research (exploratory research) and for theory building • Qualitative methods applies well in real world setting, but lack validity and control • Problem with subjective interpretation of the data • Examples • Case study: Observations carried out in a real world setting • Action research: Applying a research idea in practice, evaluate results, modify idea (cross btw. experiment and case study)

17. Selecting methods • Quantitative: • Appropriate when theory is well developed. • Theory testing and refinement • Examples: • Experiment: Apply treatment, measure results: This is the only method that can demonstrate causal relationship between variables. Associated with the scientific method • Survey: Asking rated questions in an interview • HIstorical data: Patterns in investments • Most quality research include both types of methods

18. Selecting methods • Method selection is critical to success of any project • Selection must be driven by state of knowledge

19. Data analysis • Determining whether a hypothesis/theory is supported • – easy with bugs, hard with game balancing • Quantitative data analysis: Use of statistical methods to identify patterns and relationships in the data • Qualitative data analysis: More subjective, relies on the researcher’s knowledge to identify patterns, extract themes and make generalizations

20. Data analysis • Data is objective – otherwise it is information • Processed (refined) information is termed knowledge • Generally: Data Information Knowledge • Foundational principle for all IT industries

21. Knowledge acquisition • QA is a knowledge acquisition process • Summarized: • QA is the empirical process of acquiring data, refining the data into information, and converting it to knowledge that can be implemented by company stakeholders (design, marketing etc.)

22. Getting data takes ¤%#& time ... A reason why companies hire lots of testers during crunch time ...

24. Game Testing 101:General Principles

25. Overview • QA in game the games industry • Components of game testing • General purposes of game testing • Testing phases: Intro

26. Why game testing? • Purpose of game testing: • To see how specific components of, or the entirety of, a game is played by people • The litmus test that allows developers to evaluate the state of the game and the quality of the gaming experience

27. QA position in game companies The Company Sort of the company ...

28. QA: The Stepchild of Games • QA is not a part of the main company by necessity: Keeping QA separate eliminates bias • QA is viewed as a necessary evil – low pay and crappy conditions are common • QA informs what is wrong in games under development (causing frustration) • Many forget QA can also tell what is good (causing happiness)

29. QA in game development • General software industry: QA takes 8-12% total resource • Games industry: less than 1% .... • General software industry: QA throughout production • Games industry: QA often delegated to secondary position in production pipeline

30. QA in game development • Result: Digital games has horrible quality compared to e.g. desktop applications

31. QA in game development • Non-technical game testing falls within HCI • HCI: Human-Computer Interaction • Mixture of computer science, psychology etc. • Many different types of measures – quantitative and qualitative • 20+ years of use in the software industry

32. Component of a game test • Purpose: Technical, content, functional • Phase: Positioning in the development cycle • Testing method: e.g. usability, bug hunting • Game feature: The element being tested

33. Purpose of testing • Technical • Issues relating to the game engine itself and hardware • Well-established methods common to software development • Functional • Bug hunting, stability, integrity of game assests, gameplay, localization issues, controls, interface • Content • Presentation, graphics, level design, game story, userexperience

34. Testing phases • Game production cycle has 3 general steps: • Pre-production • Production • Post-production • Most game companies follow agile development • Sprints and Scrums • Rapid iterations of game elements • Requires QA to follow same iterative nature

35. Game testing during the production cycle

36. Overview • Pre-production • Focus group tests • Benchmarking • Production • Metrics • Bug hunting • Playtesting • Usability testing • Game test labs • Post-production • Post-mortems & managing communities

37. Pre-production • Important phase, but often overlooked • Testing of design and concepts: Story, character, world, artwork • Two typical methods: • Focus groups • Benchmarking

38. Focus group testing • Popular method, but problematic • Good use can lead to valuable insights, bad use to disaster • Good for generating ideas, player impressions, norms/values of the audience • Bad for providing concrete feedback to specific issues

39. Focus group testing • Intensive design: Few units but lots of variables • Central weakness: Non-representativeness of the group participants • Analytical selection: Group participants should display the characteristics required to illustrate the case at hand

40. Focus group testing • Size: from 3-12 • Less ruins interaction, more makes them impossible to manage • Testers: build a good tester database • Screen people before adding them • Cover target audience and outside it

41. Focus group testing • Types of participants • Internal: From the company • Literature advises against using people we know in focus groups • But some internal testers can be treated as expert testers • External: From outside the company • Fans and non-fans: The problem with bias

42. Focus group testing Practical considerations for running focus groups • Homogenous or inhomogenous structure? • Should participants know each other? • Less likely to speak freely if they do • Easier to get people to talk if they do

43. Focus group testing • Group size • Small groups: Good for digging deep into associations of players • Low degree of moderation, loose structure • Large groups: Good for gathering many different perspectives • High degree of moderation, tight structure

44. Focus group testing • Running a focus group • Prepare in advance: interview guide, purpose • Decide loose or tight structure • Loose structure harder to compare across groups • Tight structure less chance of new knowledge • Visual aids should be ready

45. Focus group testing • The moderator • Monitors and moderates the focus group • Incredibly important: must be a good listener and highly attentive to the participants and the social interaction • Usually teamed with an observer

46. Focus group testing • A note of warning: • Focus groups are often run by marketing, game testing by QA – this is BAD! • Consumer testing (on e.g. the box art) should be run by marketing, but NOT game test focus groups

47. Benchmarking • Little used in the industry, early-phase • A form of requirements analysis • Methodical evaluation of competing games, recording what works and what does not • Provides the minimum benchmark the new game must meet

48. Production • Vast majority of testing during this phase • Early testing of game controls and specific game elements • Later testing of alpha builds, mechanics, story etc.

49. Production • Iterative test pattern following agile development • E.g. on a bi-weekly basis • Defining tests needed • Run tests on newest builds • Collate and analyze data • Deliver reports • Log results in test database

50. Metrics • Numerical data drawn from client installs or servers • Tracking what players do when they play e.g. • Who shoots whom where and when? (heatmaps) • Which areas of the map to players explore? • Is the balancing between weapons working? • Immensely useful in games!