Measuring User Satisfaction in Virtual Environment

1. Measuring User Satisfactionin Virtual Environment Maciej A. Orzechowski Design System and Urban Planning Group @ TU/e Workshop Mass Customisation 26.06.2003

2. Plan • Introduction • VR System (brief description) • Belief Networks (introduction) • Results of the experiment (Benchmark) • MuseV3 in action – Live Demo

3. General Idea ofMeasuring User’s Preferences The Virtual Environment (VE) is used to present an architectural design to a user. The user is asked to modify that design according to his/her needs and desires. Behind that visual system there is a statistical model to estimate and predict respondent’s preferences based on applied modifications.

4. MuseV – VR System • MuseV3 – a virtual reality (VR) application with functionality of a simple CAD system for non-designers. • Two categories of modifications: • Structural modifications (change of layout) • Textural modifications (change of visual impression)

5. Change of internal and external layout Direct impact on overall costs Structural Modifications The most important from the point of view of estimation of user’s preferences. Expressed in simple and direct commands: create/resize/divide space; insert openings

6. No influence on costs Textural Modifications Secondary modifications (visual impact), mainly used to check proportions, dimensions (inserting furniture) and to decorate (applying finishes). Not included in the preference model

7. MuseV3 in Desktop CAVE

9. Belief Network • Searching for new, flexible method to access user’s preferences. • Criteria: • Interaction with the model during the time of preferences estimation • Possibility to find weak points (where the knowledge about preferences is the worst) • Improve data collection by direct feedback • Incremental learning

10. Short explanation of BN • What it is? • Belief network (BN) also known as a Bayesian network or probabilistic causal network • BN captures believed relations (which may be uncertain, stochastic, or imprecise) between a set of variables which are relevant to some problem (e.g. coefficients and choices). How does it work? After the belief network is constructed, it may be applied to a particular case. For each variable you know the value of, you enter that value into its node as a finding (also known as “evidence”). Then Netica does probabilistic inference to find beliefs for all the other variables. Incremental learning. After the beliefs are found (post priori) MuseV updates the network, so they become a’ priori for the next respondent.

11. Step 1 Step 5 Step 15 Step 64 Step 0

12. BN - Model In our proposal the network (model) is learning  while a user is modifying a design! To improve the quality of collected data and the knowledge about design attributes, the system, (based on beliefs), can post a question to user.

13. Experiment & Results

14. Experiment Types There are in total four experiment types (FMVR, OEVR, MECA, VECA). Two in each of two groups (VR and CA). Each respondent had to complete two random tasks (one from each group), however each combination of tasks should be presented approximately equal number of times.

15. Experiment Types – cont.

16. Results

17. Respondents The truth about the respondents: We sent 1,600 letters in total !!!! The preparations to send those letters took 2,5 days for two people (Vincent and Maciek) Within 2 weeks we received 96 positive conformations. At the end of the experiment we end up with solid number of 64 respondents that have completed the both appointed to them tasks! 5 of the 64 respondents would not buy the house that they have designed. 4 respondents did not completed second task (as the design was not relevant to them) 2 respondents did not started the experiment for the same reason!

18. The most preferred system

19. The difficultness and pleasure of the tasks.

21. External validity Real Life Data – Overall (CA, BN) External validity Real Life Data – BN (FMVR, OEVR)

22. External validity Real Data – BNbased on probability distribution of each option The table illustrates ratio (percentage) of choosing certain design option. In case of real life - based on numbers of subjects buying certain option. Ri = Ni / N, where Ri – ratio for option i, Ni – number of subjects choosing option i, N – all subjects In case of BN based on beliefs read from the network.

23. Summary • The majority of the respondents prefer the VR environment to the traditional. • Respondents highly valued the freedom in modifying the architectural design. • Due to learning and understanding the software - VR is slightly difficult. • The traditional method was find as the most difficult (due to problems related with imagining the description of the house)

24. Summary • Direct observation of respondent's engagement (created designs and the time spent on the process) into the VR - indicates that people prefer to work with 3D models rather then with textual description. • The possibility of experiencing with the not existing house reinsure users' decision, raise questions and provokes discussions. • The numerical analyses showed that working with virtual reality helps respondents to understand the design and improve their decision consistency.

25. MuseV3 in Action !DEMO