Defining and Measuring Variables

Defining and Measuring Variables Chapter 3 Dusana Rybarova Psyc 290B May 17 2006

Outline: • An overview of measurement • Constructs and operational definitions • Validity and reliability of measurement • Scales of measurement • Modalities of Measurement • Other Aspects of Measurement

1. An overview of measurement • two aspects of measurement are particularly important in planning a research study or reading a research report: • often there is not a one-to-one relationship between the variable measured and the measurement obtained (knowledge, performance and exam grade) • there are usually several different options for measuring any particular variable (types of exams and questions on exams) • Direct measurement (height, weight) vs indirect measurement (motivation, knowledge, memory, marital satisfaction)

2. Constructs and operational definitions • Theories summarize our observations, explain mechanisms underlying a particular behavior and make predictions about the behavior. • many research variables, particularly variables of interest to behavioral scientists, are hypothetical attributes or mechanisms explaining and predicting some behavior in a theory are called constructs external stimulus  construct  behavior factor reward  motivation  performance

constructs can not be directly observed or measured • however, researchers can measure external, observable events as an indirect method of measuring the construct itself • operational definition • is a procedure for measuring and defining a construct, indirect method of measuring something that can not be measured directly • an operational definition specifies a measurement procedure for measuring an external, observable behavior and uses the resulting measurements as a definition and a measurement of the hypothetical construct • e.g. IQ test is an operational definition for the construct intelligence •  - provide and example of a theoretical construct and its operational definition • You don’t always have to come up with your own operational definition of the construct, you can use some conventional measurement procedure from previous studies

3. Validity and reliability of measurement • How do you decide which method of measurement (operational definition of a construct) is the best? • there are two general criteria for evaluating the quality of any measurement procedure • validity • reliability

Validity of measurement • Validity of measurement • concerns the “truth” of the measurement • it is the degree to which the measurement process measures the variable it claims to measure • Is the IQ score truly measuring intelligence? What about size of the brain and bumps on the scull?

Different kinds of validity • face validity • the simplest and least scientific definition of validity • it is demonstrated when a measure superficially appears to measure what it claims to measure • Based on subjective judgment and difficult to quantify • e.g. intelligence and reasoning questions on the IQ test • Problem - participants can use the face validity to change their answers • concurrent validity (criterion validity) • is demonstrated when scores obtained from a new measure are directly related to scores obtained from a more established measure of the same variable • e.g. new IQ test correlates with an older IQ test

Different kinds of validity (cont.) • Different kinds of validity • predictive validity • when scores obtained from a measure accurately predict behavior according to a theory • e.g. high scores on need for achievement test predict competitive behavior in children (ring toss game) • construct validity • is demonstrated when scores obtained from a measure are directly related to the variable itself • Reflects how close the measure relates to the construct (height and weight example) • in one sense, construct validity is achieved by repeatedly demonstrating every other type of validity

Different kinds of validity (cont.) • Different kinds of validity • convergent validity • is demonstrated by a strong relationship between the scores obtained from two different methods of measuring the same construct • e.g. an experimenter observing aggressive behavior in children correlated with teacher’s ratings of their behavior • divergent validity • is demonstrated by using two different methods to measure two different constructs • convergent validity must be shown for each of the two constructs and little or no relationship exists between the scores obtained from the two different constructs when they are measured by the same method • e.g. aggressive behavior and general activity level in children

Convergent validity, divergent validity and construct validity • By demonstrating strong convergent validity for two different constructs and then showing divergent validity between the two constructs, you obtain strong construct validity of the two constructs High convergent validity Related scores High Diver gent Vali dity Unrelated scores High Diver gent Vali dity Unrelated scores High convergent validity Related scores

Reliability of measurement • Reliability of measurement • a measurement procedure is said to be reliable if repeated measurements of the same individual under the same conditions produce identical (or nearly identical) values • reliability is the stability or the consistency of measurement measured score = true score + error IQ score = true IQ score + mood, fatigue etc.

Reliability and error of measurement • Inconsistency (lack of reliability) of measurement comes from error • The higher the error the more unreliable the measurement • Sources of error • observer error • the individual who makes the measurements can introduce simple human error into the measurement process • environmental changes • small changes in the environment from one measurement to another (e.g. time of the day, distraction in the room, lighting) • participant changes • participants change between measurements (mood, hunger, motivation)

Types and measures of reliability • successive measurements • Obtaining scores from two successive measurements and calculating a correlation between them • the same group, the same measurement at two different times • test-retest reliability • simultaneous measurements • obtained by direct observation of behaviors (two or more separate observers at the same time), consistency across raters • inter-rater reliability • internal consistency • degree of consistency of scores from separate items on a test or questionnaire consisting of multiple items • you want all the items or groups of items tapping the same processes • researchers commonly split the set of items in half, compute a separate score of each half, and then evaluate the degree of agreement between the two scores • split-half reliability

The relationship between reliability and validity • they are partially related and partially independent • reliability is a prerequisite for validity (measurement procedure can not be valid unless it is reliable – e.g. IQ, huge variance of repeated measurements is impossible if we are truly measuring intelligence) • it is not necessary for a measurement to be valid for it to be reliable (e.g. height as a measure of intelligence)

4. Scales of measurement • Scales define the type categories we use in measurement and the selection of a scale has direct impact on our ability to describe relationships between variables • the nominal scale • simply represents qualitative difference in the variable measured • can onlytell us thata difference exists without the possibility telling the direction or magnitude of the difference • e.g. majors in college, race, gender, occupation • the ordinal scale • the categories that make up an ordinal scale form an ordered sequence • can tell usthe direction of the difference but not the magnitude • e.g. coffee cup sizes, socioeconomic class, T-shirt sizes, food preferences

Scales of measurement (cont.) • the interval scale • categories on an interval scale are organized sequentially, and all categories are the same size • we can determine the direction and the magnitude of a difference • May have an arbitrary zero (convenient point of reference) • e.g. temperature in Farenheit, time in seconds • the ratio scale • consists of equal, ordered categories anchored by a zero point that is not arbitrary but meaningful (representing absence of a variable • allows us to determine the direction, the magnitude, and the ratio of the difference • e.g. reaction time, number of errors on a test

5. Modalities of measurement • One can measure a construct by selecting a measure from three main categories • There are three basic modalities of measurement: • self-report • physiological measurement • behavioral measurement • behavioral observation • content analysis and archival research

Self-report measures • you ask a participant to describe his behavior, to express his opinion or characterize his experience in an interview or by using a questionnaire with ratings • Positive aspects • Only the individual has direct access to information about his state of mind • More direct measure • Negative aspects • Participants may distort the responses to create a better self-image or to please the experimenter • The response can also be influenced by wording of the questions and other aspects of the situation

Physiological measures • Physiological manifestations of the underlying construct • e.g. EEG, EKG, galvanic skin response, perspiration, PET, fMRI • advantages • provides accurate, reliable, and well-defined measurements that are not dependent on subjective interpretation • disadvantages • equipment is usually expensive or unavailable • Presence of monitoring devices may create unnatural situation • question: Are these procedures a valid measure of the construct (e.g. increase in heart rate to fear, arousal)

Behavioral measures • behaviors that can be observed and measured (e.g. reaction time, reading speed, focus of attention, disruptive behavior, number of words recalled on a memory test) • How to select the right behavioral measure? • Depends on the purpose of the study • In clinical setting the same disorder can reveal itself through different symptoms • In studying memory we want to have the same measure for all subjects to be able to compare them • Beware of situational changes in behavior (e.g. disruptive behavior in school vs when observed) and different behavioral indicators of a construct

6. Other aspects of measurement • multiple measures • sometimes you can use two (or more) different procedures to measure the same variable (e.g. heart rate and questionnaire as a measure of fear) • problems (the two variables may not behave in the same way) • e.g. a specific therapy for treating fear may have large effect on behavior but no effect on heart rate • the lack of agreement between two measures is called desynchrony • One measure can be more sensitive than other • Different measures may indicate different dimensions of the variable and change at different times during the treatment

Sensitivity and range effects • are the measures sensitive enough to respond to the type and magnitude of the changes that are expected? (e.g. seconds vs. milliseconds, difficult or easy exams) • range effects • a ceiling effect (the clustering of scores at the high end of a measurement scale, allowing little or no possibility of increases in value, e.g. test that is too easy) • a floor effect (the clustering of scores at the low end of a measurement scale, allowing little or no possibility of decreases in value, e.g. test that is too difficult) • Range effects are usually a consequence of using a measure that is inappropriate for a particular group (e.g. 4-grade test for college students)

Participant reactivity and experimenter bias • participant reactivity is the way how participant reacts to the experimental situation (e.g. overly cooperative, overly defensive, or hostile) • To avoid these problems one can try to disguise the true purpose of the experiment or observe individuals without their awareness (beware ethical issues) • experimenter bias is the way experimenter influences results (e.g. by being warm and friendly with one group of participants vs. cold and stern with other group) • to avoid participant reactivity and experimenter bias we use: • standardized procedures (e.g. instructions recorded on a tape) • a research study is single blind if the researcher does not know the predicted outcome • a research study is double blind if both the researcher and the participants are unaware of the predicted outcome

Participant reactivity and experimenter bias • to avoid participant reactivity and experimenter bias we use ‘blind’ experiments • a research study is single blind if the researcher does not know the predicted outcome • a research study is double blind if both the researcher and the participants are unaware of the predicted outcome

Defining and Measuring Variables