1 / 51

Group-Level Measurement

Group-Level Measurement. Katherine Klein University of Pennsylvania kleink@wharton.upenn.edu CARMA Presentation February 2007. Why Group-Level Measurement?. Burgeoning of multilevel theory and research in last 25 years Great progress in conceptualizing and measuring group-level constructs

Télécharger la présentation

Group-Level Measurement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.


Presentation Transcript

  1. Group-Level Measurement Katherine Klein University of Pennsylvania kleink@wharton.upenn.edu CARMA Presentation February 2007

  2. Why Group-Level Measurement? • Burgeoning of multilevel theory and research in last 25 years • Great progress in conceptualizing and measuring group-level constructs • Especially shared constructs • Continuing challenges and opportunities • Especially regarding configural constructs

  3. A Few Terms and Assumptions • I’ll refer to groupsbut much or all of what I say will apply as well to organizations, departments, stores, etc. • I’ll focus on the creation and use of original survey measures to assess group constructs. • I’ll address statistical issues in passing only. • But see past CARMA presenters including James LeBreton, Gilad Chen, Paul Bliese, Dan Brass, Steve Borgatti, and others

  4. Roadmap • Fundamentals: Theory First • Construct Types: Global, Shared, and Configural Constructs • Practicalities and Technicalities • Survey Wording • Sampling • Qualitative Groundwork • Single-source Bias • Justifying Aggregation • Opportunities and Challenges • The Configuration of Diversity • Social Network Analysis

  5. Fundamentals: Theory First • Constructs are our building blocks in developing and in testing theory. • High quality measures are construct valid. • The development of construct valid measures thus begins with careful construct definition. • Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000): • Global, shared, or configural.

  6. Global Constructs • Relatively objective, easily observable, descriptive group characteristics. • Originate and are manifest at the group level. • Examples: • Group function, size, or location. • No meaningful within-group variability. • Measurement is generally straightforward.

  7. Shared Constructs • Group characteristics that are common to group members • Originate in group members’ attitudes, perceptions, cognitions, or behaviors • Which converge as a function of attraction, selection, socialization, leadership, shared experience, and interaction. • Within-group variability predicted to be low. • Examples: • Group climate, norms, leader style. • Measurement challenges are well understood.

  8. Configural Group-Level Constructs • Group characteristics that describe the array, pattern, dispersion, or variability within a group. • Originate in group member characteristics (e.g., demographics, behaviors, personality, attitudes) • But no assumption or prediction of convergence. • Examples: • Rates, diversity, fault-lines, social networks, team mental models, team star or weakest member. • Measurement challenges are less well understood.

  9. A Related Framework: Chan’s (1988) Composition Typology • Shared Constructs • Direct consensus models (e.g., group norms) • Referent shift models (e.g., team efficacy) • Configural Constructs • Dispersion model (e.g., climate strength) • Additive models (e.g., mean group member IQ) • Multilevel, Homologous Models • Process model (e.g., efficacy-performance relationship)

  10. Construct Definition Complexities:An Example: Shared Leadership • Shared leadership • “A dynamic, interactive influence process among individuals in work groups in which the objective is to lead one another to the achievement of group goals… [It] involves peer, or lateral, influence and at other times involves upward or downward hierarchical influence” • Conger & Pearce, 2003, p. 286 • Is this a shared construct, or a configural construct, or … ?

  11. Construct Definition Complexities:An Example: Shared Leadership • Well, how would you measure it? • Shared team leadership as a shared construct • “Team members share in the leadership of this team.” • “Many team members provide guidance and direction for other team members.” • Shared team leadership as a configural construct (network density): • “To what extent do you consider _____ an informal leader of the team?”

  12. Construct Definition Complexities:An Example: Shared Leadership • Calling it a “referent shift” construct is not the answer. • Referent shift is a measurement strategy, not a construct type • Shifting the referent in an unthinking manner can be quite problematic: • The members of my team… • “Express confidence that we will achieve our goals” • “Will recommend that I am compensated more if I perform well” • “Are friendly and approachable” • “Rule with an iron hand”

  13. A Quick Recap • Theory first: Define and explain the nature of your group-level constructs. • Is it a clearly objective description of the group? • If yes, a global construct. • Do you expect within-group agreement? • If yes, a shared construct. • Does it describe the group in terms of the pattern or array of group members on a common attribute? • If yes, a configural construct.

  14. Now What? • Having defined your constructs, the goal is to create measures that: • Are construct valid • Show homogeneity within (shared constructs) • Show variability between (all group-level constructs) • Practicalities and technicalities • Survey wording • Sampling • Qualitative groundwork • Minimizing single-source bias • Testing for aggregation

  15. Survey Wording:Global Constructs • Draw attention to objective descriptions of each group. • Gather data from experts and observers (SMEs) who can provide valid information about the groups in question. • No need to gather data from individual respondents within groups • Use language that fits your sample.

  16. Survey Wording: Shared Constructs • Draw attention to shared group characteristics • Use a group referent rather than individual referent to enhance: • Within group agreement • Between group variability • Predictive validity • Gather data from individual respondents so within-group agreement can be assessed. • Actual consensus methods (discussion prior to group survey completion) work well but are labor-intensive.

  17. Survey Wording:Configural Constructs • Draw attention to individual group member characteristics by using an individual referent. • Gather data from experts and observers (SMEs) who can provide valid information regarding individual group members, or gather data from individual respondents within groups. • The challenge is perhaps less in the survey wording than in operationalizing the array or pattern of interest.

  18. Sampling • Substantial between-group variability is essential. Seek samples in which groups vary considerably on the constructs of interest • Whether they are global, shared, or configural. • Statistical power reflects both: • Group sample size (n of groups) • Within-group sample size • When group size is large (number of respondents per group), measures of shared constructs are more reliable. • More research needed on power in multilevel analyses.

  19. Qualitative Groundwork • The survey wording and sampling guidelines seem fairly obvious and easy, but … • Check your assumptions in the field prior to survey data collection. • Are you measuring the right “groups”? • Example: Grocery stores or departments? • Is there meaningful between-group variability? • Example: Fast food chain • Are you measuring the right variables, and not too many of them? • Beware the blob.

  20. Single-Source Bias • Group-level correlations between measures of shared group constructs may be disturbingly high. • Examples: • Transformational and transactional leadership • Task, emotional, and procedural conflict • Aggregation does not “average away” response biases. • Rather, group members may share response biases • Halo, logical consistency, social desirability • Response bias may be particularly influential when respondents must make subtle distinctions among constructs.

  21. Single-Source Bias:Beating the Blob • Survey measures • Choose and measure truly distinct constructs • Use different survey response formats • Survey design • Keep survey items measuring distinct constructs separate. • Help respondents recognize the distinction between leadership types, or conflict types, for example.

  22. Single-Source Bias:Beating the Blob • Survey analysis • Randomly split the within-group sample of respondents during data analysis. • All receive the same survey, but half provide IV and the other half provide the DV for analyses • Survey administration • Randomly split the within-group sample of respondents during data administration. • Respondents receive distinctive surveys. Half receive the IV survey and the other half receive the DV survey.

  23. A Quick Recap • Having • Defined our constructs • Written our survey items • Conducted qualitative groundwork • Sampled appropriately • Taken steps to reduce single source bias • We’re almost ready for hypothesis testing • But first: We need to justify aggregation

  24. Justifying Aggregation • Why is this essential? • In the case of shared constructs, our very construct definitions rest on assumptions regarding within- and between-group variability. • If our assumptions are wrong, our construct “theories,” our measures, and/or our sample are flawed and so are our conclusions. • So, test both: • Within group agreement • The construct is supposed to be shared, but is it really? • Between group variability (reliability) • Groups are expected to differ significantly, but do they really?

  25. Justifying Aggregation: rwg(j) • Developed by James. Demaree, & Wolf (1984) • Assesses agreement in one group at a time. • Compares actual to expected variance. • Answers the question: • How much do members of each group agree in their responses to this item (or this scale)? • Highly negatively correlated with the within group standard deviation • Valid values range from 0 to 1 • Rule of thumb: rwg(j) of .70 or higher is acceptable

  26. Justifying Aggregation: rwg • Common to report average or median rwg(j) for each group for each variable: • If rwg(j) is below .70 for one or more groups, check: • Does the group have low rwg(j) values on several variables? • Do many groups have low rwg(j) values on this variable? • Remember: rwg(j) indicates within-group agreement, not between-group variability. • Beware: When variance in a group exceeds expected variance, out of range rwg(j) result.

  27. Justifying Aggregation: h2 • Assesses between-group variance relative to total variance, across the entire sample. • Based on a one-way ANOVA • Answers the question: • To what extent is variability in the measure predictable from group membership? • The F-test provides a test of significance • The larger the sample of individuals, the more likely eta2 is to be significant. • Beware: h2 may be inflated when group sizes are small (under 25 individuals per group) • But, this is an easy way to begin tests of aggregation

  28. Justifying Aggregation: ICC(1) • Assesses between-group variance relative to total variance • Based on a one-way ANOVA • Answers the question: • To what extent is variability in the measure predictable from group membership? • The F-test provides a test of significance • Based on h2 but controls for the number of predictors relative to the total sample size, so ICC(1) is not biased by group size.

  29. Justifying Aggregation: ICC(2) • Assesses the reliability of the group means (i.e., between-group variance) in a sample, based on ICC (1) and group size. • Answers the question: • How reliable are between-group differences on the measure? • Reflects ICC(1) and within-group sample size • Example: If ICC(1) = .20 and: • Mean group size is 5, expected ICC(2) = .56 • Mean group size is 20, expected ICC(2) = .71

  30. Justifying Aggregation: An Example

  31. A Quick Recap • The hope is that we have successfully: • Defined our constructs. • Written our survey items. • Conducted qualitative groundwork. • Collected data from a large sample of groups. • Taken steps to reduce single source bias. • Justified aggregation. • And moved on to test our hypotheses. • So, what remains?

  32. Opportunities and Challenges:The Configuration of Diversity • Configural constructs describe the array, pattern, dispersion, or variability within a group. • The easy example is diversity • Demographic diversity • Climate strength • But even the easy example isn’t so easy: What is the definition of diversity? And how should it be measured?

  33. The Configuration of Diversity • A starting definition of diversity: • The distribution of differences among the members of a group with respect to an attribute, X, such as age, ethnicity, conscientiousness, positive affect or pay. • Okay, but what’s maximum diversity? • Which team has maximum age diversity? • 20, 20, 20, 70, 70, 70 • 20, 30, 40, 50, 60, 70 • 20, 20, 20, 20, 20, 70 • 20, 70, 70, 70, 70, 70

  34. The Configuration of Diversity • Diversity isn’t one thing. • It’s three things: Separation, Variety, or Disparity • The three types differ in: • Meaning or substance • Pattern or shape • Likely consequences • Appropriate operationalization • Blurring across these distinctions leads to fuzzy theory, misguided operationalizations, and potentially invalid research conclusions

  35. The Configuration of DiversityExample: Three Research Teams • Team S • Members differ in their view of qualitative research. • Half of the team members respect it, half don’t. • Team V • Members differ in their discipline. • 1 psychologist, 1 sociologist, 1 anthropologist, etc. • Team D • Members differ in their rank • 1 senior professor, others are incoming graduate students.

  36. Diversity as Separation • Differences in group members’ position, attitude, or opinion along a continuum • Min: Every member has the same opinion • Max: Two polarized extreme factions • Theory: Similarity-attraction • Operationalization: Standard deviation

  37. Diversity as Variety • Differences in kind or category • Min: Every member is the same type • Max: Each group member is a different type • Theory: Requisite variety, cognitive resource heterogeneity • Operationalization: Blau’s index of categorical differences

  38. Diversity as Disparity • Differences in concentration or proportion of valued assets or resources • Min: Every member has an equal portion of the resource • Max: One member is “rich” and all others are “impoverished” • Note: Disparity is asymmetric • Theory: Inequality, relative deprivation, tournament compensation • Operationalization: Coefficient of variation (SD/Mean)

  39. The Configuration of Diversity:A Recap • Theory first • Separation is about position, attitude, or opinion • At maximum: Polarized factions • Variety is about knowledge or information. • At maximum: One of a kind • Disparity is about resources or power. • At maximum: One towers over others • Operationalize accordingly • The coefficient of variation is not a default or catch-all

  40. Opportunities and Challenges: Social Network Analysis • Multilevel analysis and social network analysis have developed along separate paths. • Rich opportunities for cross-fertilization. • Social network analysis provides a means to conceptualize and operationalize configural constructs. • Illuminating the pattern or array of interpersonal ties within a group

  41. Opportunities and Challenges: Social Network Analysis • Many of our shared constructs appear to rest on tacit, often fuzzy, assumptions about interpersonal ties with groups. • Examples: Cohesion, communication, coordination, knowledge sharing, shared leadership, conflict • But we know little about the configuration of interpersonal ties – the structures – that underlie our shared constructs and measures.

  42. An Example: Social Network Analysis and Shared Team Conflict • When teams report high task or emotional conflict, what is the structure of interpersonal ties within the team? • As a starting point: • How dense are positive (advice) ties? • How dense are negative (difficulty) ties?

  43. An Example: Social Network Analysis and Shared Team Conflict • Task and emotional conflict: The blob • r = .83 • Advice density and negative tie density: More weakly correlated • r = -.36 • Task conflict (mean task and emotional conflict), advice density, and negative tie density • Team Conflict and Advice Density: r = -.47 • Team Conflict and Difficulty Density r = .40

  44. Negative Tiesin a Low Conflict Team

  45. Negative Ties in a High Conflict Team

  46. Advice Ties in a High Conflict Team

  47. Advice Ties in a Low Conflict Team

  48. Social Network Analysis:A Recap • Social network analysis illuminates the configuration of interpersonal ties in groups. • What network structures underlie our shared constructs and measures? • Do network measures provide incremental validity? • Not just density, but centralization, cliques, and more. • What explains between-group differences in network structures?

  49. In Conclusion • Theory first. Define your constructs. • Are they global, shared, or configural? • Measure constructs and collect data with care • Match item wording to the construct • Conduct qualitative groundwork • Sample appropriately • Take steps to reduce single source bias • Test for aggregation • Studying configural constructs remains a challenge and an opportunity • Conceptualizing and measuring diversity • Integrating social network analysis within our arsenal

  50. Some Helpful References • Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass. • Borgatti, S. P. (2003). The network paradigm in organizational research: A review and typology. Journal of Management, 29, 991-1013. • Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246. • Harrison, D. A. & Klein, K. J. (2007). What’s the difference? Diversity as separation, variety, or disparity in organizations. Academy of Management Review. • Harrison, D. A. & McLaughlin, M. E. (1996). Structural properties and psychometric qualities of organizational self-reports: Field tests of connections predicted by cognitive theory. Journal of Management, 22, 313-338. • James, Demaree, & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.

More Related