Introduction to User Studies

1. Introduction to User Studies Yale Braunstein November 1999

2. Three Types of Studies • Traditional studies of users and usage (often of libraries) • Studies that include Internet uses and similar measures • Studies of Digital Libraries and their users (“digital communities,” just starting)

3. Traditional Studies

4. Basic Problems • Is there a theoretical basis to the analysis? • Are the results reproducible? • What can be changed based on the findings? • How do we know changes will help? Possible solution:- Develop theoretical approach early

5. Typical Steps in Social Science Research • State general nature of problem, question, relationship • Review previous studies • Create and state “stylized facts” • Develop testable hypotheses • Discover or develop data to be used to test hypotheses • Perform appropriate (statistical) tests • Describe results • Draw conclusions

6. Traditional Bibliometric Model of Scholarly Productivity Socio-economic factors Institutionalfactors Funding Research output

7. Revised Bibliometric Model of Scholarly Productivity Socio-economic factors Institutional factors Funding Research output Computer skills/literacy Computer use / Internet use

8. Revised Bibliometric Model of Scholarly Productivity Socio-economic factors Institutional factors Funding Research output Computer skills/literacy Computer use / Internet use Major issues: - What sample/universe to use? - How to measure inputs? - How to measure outputs? - Functional form? (linear?, additive?)

9. List of Selected Processes

12. Our main hypothesis in this research is that scholars' Internet-use data add explanatory power to models of scholarly productivity. The formal null and alternate hypotheses are: • H0: Internet use data does not add explanatory power to the Traditional Publication Model. • H1: Internet use data adds explanatory power to the Traditional Publication Model. • We used multiple regressions to estimate the four models listed in Figure 5. Since we are interested in whether a set of one or more Internet-use variables “improves” the explanatory power of the traditional model, the appropriate measure is the F-statistic from a comparison of the restricted (traditional) model with the unrestricted (new) model.

17. The exact equation, as estimated in Equation 1, is: PUBAV = 17.1 – 0.313 AGE + 0.197 PHD-AGE + 0.0860 RL – 0.0794 PIM1 + 19.6 (Login-FTP principal component) + 2.99 (Kermit principal component).

19. Digital Libraries and Collaborative Knowledge Construction Based on a presentation by Nancy Van House September 15, 1999

20. Digital Libraries • Support knowledge work • Knowledge work occurs in material and social matrices • DLs mutually constituted with work, practices, tools, artifacts, knowledge, knowledge communities

21. Evaluating DLs • Direct effects: how well they meet users’ needs & expectations • Higher-order effects: Understanding how DLs support and potentially change • Work • Its practices and artifacts • People/communities who do the work • Institutions that support it

22. Possible Theoretical Bases for DL Evaluation • Situated action • work and learning • Science studies • knowledge creation • knowledge communities • Actor-Network Theory • knowledge creation and communities • stabilization of socio-technical systems

23. Characteristics of Knowledge Work • Situated in place and time • Distributed x people, places, time • Social - collectively decide what is known, how we determine what is known, how we do the work

24. The UC Berkeley Digital Library Project • Supporting environmental planning • Innovative tools • Large, diverse testbed

25. This Research • To better understand distributed, collaborative cognitive work, the role of information and information practices and artifacts, and the potential effects of digital information • To investigated the practices of environmental planning, particularly information use and production

26. This Study • Not about the UC Berkeley DL but about potential of DLs in general, and current issues around sharing of digital data. • Method: Interviews

27. Knowledge Communities • Two engaged in work related to environmental planning • Water planning • Botanical • Commonalities • Multiple scientific (mostly) disciplines • Various organizational settings • Large datasets of observations and summaries and analyses • collected by many, over time, differing methods • Potentially major economic impact • Often highly political

28. Findings • Contributions • Use • Cooperation in creating and operating DL

29. Findings: Barriers to Contribution • Misuse of data • People who ‘don’t understand’ • “Using our data against us’ • ‘Productizing’ • Burden • Mismatched incentives • Making invisible work visible

30. Findings: Use • User must be willing to trust both content and functionality of DL • Assessing trustability of contents • Assessing sources • Ability and training • Interests • DL’s filtering policies and processes • DL’s functionality • E.g., search tools, GIS overlays

31. Findings: Cooperation in Creating and Operating DL • Both a research project and active testbed • Groups to interest: computer scientists and applications area specialists • Cutting edge vs reliable, incremental improvement over past • Differences in decision-making styles: hierarchical, planned vs distributed, emergent • Articulation work

32. Possible Theoretical Bases for DL Evaluation • Situated action • work and learning • Science studies • knowledge creation • knowledge communities • Actor-Network Theory • knowledge creation and communities • stabilization of sociotechnical systems

33. Two Broad Approaches to Social Theory • Reductionist: economics, behavioral psychology, structural-functional sociology • Irreductionist: ethnomethodology (Garfinkel), practice/structuration theory (Giddens, Bourdieu, Lave)

34. Irreductionist Approaches • Social order/institutions not pre-given but continually recreated through the concrete, day-to-day activities Interest in processes by which people construct meaning and re/produce social order • Interest in processes by which this done • Emphasis on practices and artifacts which shape understanding and carry it across time and space

35. Situated Action, Situated Learning • Knowledge not inert substance transferred from teacher to student • Knowledge and learning on-going construction of meaning through action and interaction • Knowing not activity of single mind but complex social and material phenomenon, “nexus of relations between the mind and work and the work in which it works” (Lave, 1988).

36. Communities of Practice: the Social Matrix of Knowledge Work • People learn & work within groups who share understanding, practices, technology, artifacts, and language, e.g., professions, workgroups, and disciplines. • A person typically belongs to multiple communities of practice • Important task of communities of practice: • deciding what is known; • the processes and principles by which knowledge claims are evaluated; • who is entitled to participate in the discussion; • whom to believe.

37. Information Artifacts • Include texts, images, maps, databases, thesauri, classification systems… • Not simply reflections of knowledge but instrumental in creating it, coordinating work across space and time • Tightly bound up with practices and communities

38. Science and Technology Studies • Science Studies • practices of scientific work • how the participants come to an agreement about what counts as fact or discovery • what inferences are made from facts • what is regarded as rational or proper conduct • how credibility of claims is assessed • Science and Technology Studies (STS) • Increased emphasis on technology, esp’ly large sociotechnical systems • Science as prototype of rational knowing > applicable to knowledge work generally

39. Approaches to Research • Sci & tech knowledge neither purely natural nor social • Attention to processes of sci knowledge production • Methodological orientation • “Follow the actor” • Attention to activities • Not deciding ahead of time where to draw boundaries • Funding as part of scientific activity • Inscriptions play key role in sci knowledge production • E.g., labs as producing papers • Carrying work across space and time

40. Actor-Network Theory (Bruno Latour, Michel Callon, John Law) • Not a theory but a “method of looking at the actors’ own world building activities” • Doesn’t take the stability of sociotechnical systems as given, but asks how they are created and maintained. • “Organization is an achievement, a process...Its components -- hierarchies, organizational arrangements, power relations, flows of information -- are uncertain consequences of the ordering of heterogeneous materials.. ANT is thus a theory of agency, of knowledge, and of machines.” (Law in systems science (Emphasis added)).

41. Key Concepts of ANT • Action-at-a-distance • The actor-network • Translation • Black-boxing • Intermediaries

42. Key Concepts of ANT • Action-at-a-distance is the problem of coordinating and sustaining collective activities over space and time). • The actor-network “is most simply defined as any collection of human, non-human, and hybrid human/non-human actors who jointly participate in some organized (and identifiable) collective activity in some fashion for some period of time.” • Perhaps the most radical contribution of ANT is the inclusion of the non-human in its “heterogeneous networks.”

43. Key Concepts, cont. • Translation is a key process by which actor-networks are created and stabilized, however temporarily. Actors’ disparate interests get translated into a set of interests that coincide in the network. • Black-boxing is a process of closing questions and debates. • An intermediary is an actor (of any type) that stands between two others and translates between them in such a way that their interaction can be more effectively articulated. • Inscriptions, Latour’s “immutable, combinable mobiles:”

44. Approaches to Researchof Situated Action, Science Studies, ANT • Contention that sci & tech knowledge neither purely nature nor purely social • Methodological orientation: • “Follow the actor” (ethnography) • Inscriptions

45. Applied to our DL • Applications area: environmental planning • The DL as a large sociotechnical systems • Method

46. Implications: Joining Findings & TheoryI. DLs and Knowledge Productions • DLs support knowledge work, which is situated, distributed, and social • DL ‘evaluated’ according to its relationship with knowledge work and its social and material matrix

47. Users’ and Providers’ Concerns – the DL and Knowledge Production • Fear of misuse of info, inability to evaluate credibility = crossing boundaries of communities and assemblages within which/for which created • Practices of (1) work and (2) assessing/demo’ing credibility undermined by • Ready crossing of sociotechnical boundaries • Fluid recombination of inscriptions • Opening of black boxes, e.g. document, methods of work • Closing of others, e.g. analytical and search tools

48. Implications: Joining Findings & TheoryII. The DL as a Sociotechnical System • Successful DL enrolls dynamic network of users, sources (human and nonhuman), builders, and technology • Requires on-going enrollment and multiple translations • Stabilization not equilibrium but on-going achievement

49. Implications for DL Design • Articulate with existing assemblage of practices, artifacts, participants • Attention to existing and emergent practices of trust and credibility • Design specific to communities and tasks • Tension of specificity and generality • Include high-order user involvement • Design to be emergent and dynamic – on-going enrollment and co-constitution

50. Implications for DL Evaluation • Not just how well it meets users’ needs but how co-constituted with assemblages of work • Target to specific user communities and tasks • Consider not just service delivery but stabilization of DL as a heterogeneous network