Quantitative Methods Topic 4 Sampling

Quantitative MethodsTopic 4Sampling

Outline • Populations and samples • Sampling Frames • Representativeness • Probability and non-probability samples • Example sampling methods

Reading on Sampling • IIEP Module 3 • Kenneth N Ross • TIMSS 2003 technical report – sampling chapter • Pierre Foy and Marc Joncas

Populations and samples Population eg all students at a school Sample - small N selected to represent population

Why sampling • Study of part rather than the whole population • Advantages • Reduced cost • Generalisations about • Estimates of characteristics

Population and Units of Analysis • Defining the population • Without definition, we don’t have a context for the results • In an educational survey, the population will be defined by the units of analysis which may be: • the student (eg studies of attainments) • the teachers (eg studies of teaching practice) • the school (eg studies in school environment) • Each unit of analysis may require a different sampling strategy.

Populations • Desired: population for which the results are ideally required • Defined: population which is actually studied, • Excluded: The elements that are excluded from the desired target population in order to form the defined target population

Sampling Frames • A listing of the elements in a population. • E.g., School’s enrolment records can readily provide a sampling frame for the population of students: all students are listed, and each student listed only once.

Representativeness • A sample is considered as representative if certain percentage of frequency distributions of some characteristics within the sample data are similar to those within the whole population • The population characteristics selected for comparisons are called “marker variables”: • In education, common marker variables are sex, age, SES, school types, location, school size, ethnicity.

Sample types • Probability: • Each member of the defined target population has a known and non zero chance of being selected into the sample • Estimating the values of population parameters from sample parameters • Testing statistical hypothesis about population from samples • Non- Probability: • It is not possible to determine whether a non-probability sample is likely to provide very accurate or inaccurate estimates of population parameters.

Types of non-probability samples • Judgement sampling • Base on researchers’ judgment • Convenience sampling • Subjects or elements of a sample were selected base on their accessibility to the researcher • Quota sampling • Number of elements (subjects) are drawn from various target population strata in proportion to the size of these strata • Little or no control over the procedures used to select elements within these strata • There is no way of checking the accuracy of estimates

Types of probability samples • Simple random sampling • Systematic sampling • Stratified sampling • Multistage Cluster Sampling

Simple Random Sampling • There is a single sampling frame or list of names • A sample is selected from the list in a single operation • e.g. list of students in a faculty used to select a sample for course evaluation

Golden Rule of Simple Random Sampling • Each member of the population shall have an equal chance of selection.

Class activity 1: Using SPSS to take a random sample • The data file VNsample.sav contains ID of all students of a district. Draw a simple random of 10% of the population as follows: • Click DATA -> SELECT CASES -> RANDOM SAMPLE -> SAMPLE-APPROXIMATELY 10% CONTINUE • Click on ‘copy selected cases to a new data set’. In the box type the new data set name. Click OK.

Class activity 1 • Examine the new data set to see how many students were randomly selected. • Calculate frequencies of girls and boys. Compare with the main sample. • Calculate mean mathematics achievement (variable pma500). Compare with the results of the main sample. • Repeat the selection three times to draw different samples and check how results vary.

Systematic Random Sampling Example • To draw a systematic random sample of size 16 from our list of Metropolitan schools (160 schools), ordered by school number, we would • Calculate the sampling interval (160/16 = 10) • Draw a random number between 1 and 10 (say it is 7) • The sample will then consist of the following 16 schools from the list: the 7th, the (7 + 10 = 17)th, the (7 + 2´10 = 27)th and so on to the (7 + 15´10 = 157th) • Note that the number of different samples that can be drawn by systematic sampling is typically quite small (10 in this example)

Systematic Random Sampling(Random Start Sampling Interval) • Work out sampling interval. • Select a random start. • Every qth element in the register is selected from the random start • May be more efficient than Simple Random Sampling, e.g. when there is a systematic relation between the population order and the response variable(s) (i.e. give estimates of greater precision than a SRS of the same size) • May result in a biased sample if there is pattern in the list.

The list of schools that we have been working with is largely, but not completely, arranged in alphabetical order of the school name.It is unlikely that, here, the order would be related to the N of students studying Psychology. Hence, it is unlikely that the precision of the sample would be superior to that obtained from a simple random sampleThe list could, however, be sorted by a variable that we might expect was related to the N of Psychology students (e.g. school or Year 12 cohort size)In this case, we would expect the precision of estimates from the sample to improve for any characteristic related to N of Psychology students.

Class Activity 2: Draw RSSI sample • Open METROSCHOOLS.SAV in SPSS. • DATA EDITOR WINDOW -> DATA-SORT CASES; put yr12en into the box sort by then OK. • In the DATA EDITOR window examine the variable yr12en across a few cases: comment. • Draw a RSSI 10% sample. • does the sample represent the full range of school sizes (year 12 enrolment) ? Why?

Class Activity 2: Dangers of RSSI • A. British electoral registers are lists of street addresses in street number order. Even numbers are on one side of the street, odd numbers on the other. • With RSSI, and an even sampling interval (eg 20, 22 or 24) how many sides of any street will you sample? • B. You are using RSSI to draw a sample from a list of club members, in alpha order with many married couples listed in male female order… any problems?

Stratified Sampling (1) • The target population is divided into non-overlapping sub-populations called strata • Sampling is performed independently within strata

Types of stratified sampling • Proportionate • the within-stratum sample size is calculated such that it is proportional to the size of the sub-population • Disproportionate • Uses different sampling fractions within the various strata • Is used in order to ensure that the accuracy of sample estimates obtained for stratum parameters is sufficiently high to be able to make meaningful comparisons between strata

Analysis of Disproportionate Stratified Sample • Weighting is required to analyze the full sample. • Weighting is not required to analyze strata separately • Post-stratification can be used to weight a sample to know population characteristics after selection and/or after data collected. EG Post stratify by age or ethnic background

Stratified Sampling (2) • Provides increased accuracy in sample estimates without leading to increases in costs • Can guarantee representation of small sub-populations in the sample • Many population frames are readily divided into sub-populations - e.g. into States and Systems (government, Catholic, private) in national education surveys; into States and residential location (rural/urban) in health or employment surveys • In some studies stratification is used for reasons other than obtaining gains in accuracy • Strata may be formed in order to employ different sample design within strata. • Subpopulations defined by the strata are designated as separate domains of study.

Multi-Stage Cluster Sampling • Used where there are naturally formed groups of population elements (e.g. schools, households, community health centres etc.) and, frequently, • Used when a full population frame is not available (e.g. all students in all Government schools in Australia, all patients seen by medical staff in all community health centres in Victoria) • In face to face interview studies: When the sample is geographically dispersed and the costs of travel would otherwise be prohibitive. • Enables the researcher to gather data from within the sampled clusters only, and thus lowers the cost of a survey.

An Example of Cluster Sampling • Primary Sampling Stage: Select a number of schools in Victoria (it can be done using simple random sampling or stratified sampling of Government, Roman Catholic, and ‘Private’ schools in Victoria • Stage 2: A sample of Year 12 students is then drawn from the enrolment records of each of the sampled schools • Note that a full list of all Year 12 students in all Victorian schools is not needed. All that is required is a list of students for each of the sampled schools

At least TWO stages in MS sampling • Stage One: select primary sampling units, eg schools, electorates, local authority areas, community health centres • Stage Two: select secondary sampling units: eg pupils within schools, patients within community health centres • Note that each stage is a separate sampling operation, and these operations need not be uniform: • may use stratification • may use RSSI sampling

Accuracy of estimates from samples • The degree of accuracy of sample estimates may be judged by the difference between the sample estimates and the value of the population parameters • In most situations, the value of population parameters are unknown. • It is possible to estimate the probable accuracy of the obtained sample through a knowledge of the behaviour of estimates derived from all possible samples.

A simple example • Students are given two packs of cards, combined into one deck. They are asked to guess the proportion of cards that are red. • Students then draw a sample of 10 cards, and use it to make the estimate. This is repeated several times to draw several samples of 10. • Similarly, students draw repeated samples of 50. • Results are graphed to form two sampling distributions: one for samples of 10, the other for samples of 50. (example on next slide). • Which sample size: 10 or 50, will give the most accurate result?

Class Activity 3 : Sampling Task Use the EXCEL procedure as for Week 5 for simulating drawing of cards.

Sampling Distribution: estimates of % cards red from several samples 10 25 Number in each sample: Number of samples drawn 50% 47.6% Estimate from first sample Average for all samples: 10 9 8 7 6 5 4 3 2 1 0 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percent of red cards in sample

Sampling Distribution: estimates of % redcards from several samples 50 25 Number in each sample: Number of samples drawn 50% 52..8 Estimate from first sample Average for all samples: 10 9 8 7 6 5 4 3 2 1 0 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percent of red cards in sample

Difference in the distributions • samples of 10 • have a higher Standard Deviation (SD) • have a more dispersed distribution • the estimates from individual samples vary greatly • samples of 50 • have a lower SD • have a less dispersed distribution • the estimates from individual samples vary less

Factors affecting accuracies of estimates from samples • Sample size, as seen from the above example. • Other factors: • Sampling design • Stratified and Systematic sampling may increase accuracy. • Cluster sampling may reduce accuracy.

Clustering sampling in education • Schools are selected first. • Then, students are selected from the selected schools. • If there is a large “intraclass correlation”, precision of estimates will be reduced.

Intra-class correlation • In the context of schools/students: • The degree to which students are similar within schools. • Large intra-class correlation: • Schools are highly tracked. High ability students are in the same schools. Low ability students are in other schools. • Low intra-class correlation: • The range of abilities of students is about the same in all schools.

Effect of cluster sampling • If intra-class correlation is high, then we need to select more schools to get the variations of student abilities. • In the extreme case, if all students within each school have the same ability, then sampling all students from one school is equivalent to sampling just one student. • Our estimate from the sample will be quite imprecise as compared with the population parameter. (“loss of sampling efficiency”)

(Sampling) Design Effect • Defined as the loss of efficiency from sampling • If n1 students are required to achieve the same precision as for n2 students from a simple random sample, then the design effect is n1/n2.

Table of intra-class correlation - 1 • Table 1.1 in Reading ‘IIEPModule3.pdf’ • For example, if we sample 20 students from each school (cluster size of 20), and the intra-class correlation is around 0.2, then the design effect is 4.8 • This means that, if cluster sampling is used, we need a sample size 4.8 times larger than the sample size for a simple random sample. • In Australia, the intra-class correlation is around 0.2, or a little higher.

Table of design effects • See document ‘DesignEffectPISA2003.xls’ (data from PISA 2003 technical report)

Computation of sampling error • More complicated when multi-stage cluster sampling is used. • Can be estimated once the intra-class correlation is known (say, from previous studies)

Quantitative Methods Topic 4 Sampling