RRFC Portal - Sept 2006 Newsletter 2

Sept 2006 Newsletter 2

A Quick Guide to Sampling

By: Susan Wagner, Data Driven Enterprises

Certain State Performance Plan (SPP) indicators allow a State to sample population members (e.g., parents or students). Sampling can provide accurate State-level results as long as the sample is representative of the population and, if a response rate of less than 100 percent is received, as long as the responses received are representative of the population. This article explores two issues: (A) How to select a representative sample and (B) How to determine if your responses are representative of the population. The context of this article will be Indicator #8 (Parent Survey). However, the same principles apply to other sampling indicators.

A. How to select a Representative Sample

In determining who will get a parent survey, the State must ask itself two general questions:

Do you want to survey a sample or a census of districts each year?

If you are going to survey a census of districts (i.e., all districts), then go to question 2.

If you select a sample of districts:
- The sample must be representative of the State in terms of:
  - Geographic area
  - Small, medium, large districts
  - Districts’ rates for race/ethnicity, type of disability, free/reduced lunch, special education.
- Every district with 50,000 or more students must be sampled every year.
- Every district needs to be sampled at least once over the life of the State Performance Plan (SPP).
Benefits of Sampling

Did you know that your results will be more accurate (in terms of describing the attitudes of the entire population) if you send your survey to a representative sample and get a fairly high response rate than if you send your survey to the entire population and get a fairly low response rate—even if the sampling method results in fewer returned surveys than the census method?

For example, survey results will most likely be more accurate if the results are based on 1,000 returned surveys from a representative sample of 2,000 parents than on 2,000 returned surveys from the entire population of 15,000 parents.
Do you want to survey a sample or a census of parents within each selected district each year?

Whether you select a sample or census of parents depends on how accurate you want your results to be, the resources you have to survey a census, and the importance of giving all parents an opportunity to respond to the survey. Psychometrically speaking, it’s generally better to survey a sample of parents than a census of parents because you can do various follow-up activities to get a high response rate whereas if you survey all parents, you may not have the resources (money, time, and people) to do all the follow-up activities with all the parents. However, giving every parent an opportunity to complete the survey may be very important to the State, district, and/or school in which case, surveying all parents within the selected districts is the better method.

If you select a sample of parents (either from a sample of districts or from all districts within the State), you need to decide what the sample size will be. Confidence levels, confidence intervals, and response rates dictate sample size.

Do you want to be 99%, 95%, or 90% certain of your results? (Confidence Level)

A typical confidence level is .95. A confidence level of .95 means that you are 95% sure that the “true” population results fall within your confidence interval.
Do you want an error rate of 1%, 2%, 3%, 4%, 5%, or higher? (Confidence Interval)

The error rate indicates the confidence interval (i.e., the range within which the true score falls). For example, if your error rate is 4 percent, then the confidence interval is plus/minus 4 points; this means that if 52 percent of the survey respondents selected an answer, then between 48 percent and 56 percent would have selected this answer had you asked the question of the entire population. If your confidence level is .95, then you’re 95% certain that the true score lies between 48-56 percent.
What percentage of the sample do you expect to complete and return a survey? (Response Rate)

If you expect a 25 percent response rate, then you need to sample four times as many people to maintain the desired error rate.
Example:
- You have a population of 15,000 parents of children with disabilities.
- You want a confidence level of .95.
- You want an error rate of 5 percent.
- You expect a response rate of 25 percent.
In this scenario, you would need 375 returned surveys. To get 375 returned surveys, you would need to send the survey to 1,500 parents (375/.25).

Table 1 illustrates the error rates for a population size of 15,000 (e.g., if you have 15,000 parents of students with disabilities in your State, this table is appropriate). Go to http://www.custominsight.com/articles/random-sample-calculator.asp to calculate the error rate given a certain population size.

Table 1: Maximum Error Rates for a Population Size of 15,000

Confidence Level

Number of Completed Surveys

100

200

300

400

500

600

700

800

900

1000

90%

8.2%

5.8%

4.7%

4.1%

3.6%

3.3%

3.0%

2.8%

2.7%

2.5%

95%

9.8%

6.9%

5.6%

4.8%

4.3%

3.9%

3.6%

3.4%

3.2%

3.0%

99%

12.8%

9.0%

7.4%

6.4%

5.7%

5.2%

4.8%

4.4%

4.2%

3.9%

Please note that the error rate will be larger for the subgroups than the overall sample. In other words, if you have 700 returned surveys, the error rate will be 3.6 percent. However, if you produce results by district and one district has 100 returned surveys, the error rate for this district will be 9.8 percent; if another district has 50 returned surveys, the error rate for this district will be 13.8 percent.

Do any groups of parents need to be over-sampled?

If a subgroup is particularly small or if you expect a lower response rate from one subgroup compared to other subgroups, you may need to oversample it. For example, if you are surveying 20 percent of parents in your State, and one district has only 15 parents of students with disabilities, you might want to survey 100 percent of parents in this district. Or if parents of District #12 respond at a rate half of what parents in other districts respond, then you might want to survey 40 percent of the parents in District #12 and 20 percent of parents in the other districts.

How should you actually select your sample?

The sample should be selected using a random stratified procedure. A random stratified sampling procedure is one in which the researcher separates the members of the population into exclusive groups (e.g., males and females) called strata. Random samples are then taken from each strata; this guarantees that the strata sample sizes are proportionate to the strata population sizes (e.g., if 30 percent of the parent population are from District A, then 30 percent of the parent sample will be from District A). For the parent survey, you might want to stratify by district, school, student grade, student race/ethnicity, and student primary disability.

Example: Before selecting a sample of parents from the list of all parents of students with disabilities, the population is sorted based on district and school (e.g., parents in District A—School A, parents in District A—School B, parents in District B—School A, etc.) and within each stratum the nth person is selected.

How to Determine if Your Responses are Representative of the Population

Whenever you solicit responses, either through a written questionnaire or a phone interview or some other data collection method, you want a high response rate. If you get less than a 100 percent response rate there’s a chance the responses you received are not representative of the population and/or are biased. The lower your response rate, the greater the likelihood that you have a non-representative and biased response. This is why a high response rate is so important. A response rate of at least 50 percent is desirable. If you get a response rate less than 50 percent you should check for nonrepresentativeness and nonresponse bias.

How do you check for nonrepresentativeness?

A “representative” response rate is when the respondents are similar to the sample (or population) on various demographic characteristics. For example, 15 percent of the respondents are from Region A; 15 percent of the sample is from Region A. The demographic variables that are asked on the survey (e.g., disability, race/ethnicity, district, grade, etc.) are those that will be used to check for representativeness.

If a sample is not representative on a given characteristic, there are three options:

Ignore it (this is okay if you think the characteristic is not related to parents’ attitudes).
Weight the responses to make it representative. This is conditioned on receiving a minimum number of respondents with a given characteristic. If you receive two surveys from parents at School A, and these two surveys are supposed to represent the 50 School A parents in the sample, well, that’s too few!
Collect additional surveys from underrepresented groups to make the response rate representative. In the example in the preceding paragraph, you could contact School A parents and try to get more returned surveys from them.

Please note that if you find that certain groups are underrepresented, you should examine the data collection process to determine how it can be changed to prevent unrepresentativeness in the future.

How do you check for nonresponse bias?

Generally, it is assumed that the respondents’ attitudes reflect the attitudes of the entire sample (or population). In other words, if you have a 25 percent response rate, we assume that the 25 percent who responded have similar attitudes and experiences to the 75 percent who did not respond. However, this may or may not be the case.

Nonresponse bias can be examined in two ways:

By comparing later returned surveys to earlier-returned surveys:
- If respondents who return their surveys late in the data collection process express different attitudes from respondents who return their surveys early in the data collection process, then it’s possible those who don’t return their surveys have different attitudes.

By contacting a sample of the nonrespondents
- If the nonrespondents who are contacted express different attitudes than the respondents, then nonresponse bias exists.
- Nonrespondents may be contacted via telephone interviews, in-person interviews, or follow-up written questionnaires.

If you determine that nonresponse bias is present, then you should collect a sample of surveys from the nonrespondents and then weight your overall survey results so that the responses of the contacted nonrespondents get the appropriate amount of weight (depending on response rate and their demographic characteristics).

Please note that “nonresponse bias” is a different question than the representativeness of the respondents. It is possible to have a “representative” group of respondents in terms of race/ethnicity, child disability, region of the State, etc. and have nonresponse bias in terms of attitudes. Nonresponse bias means that those who didn’t respond have different attitudes than those who responded; it does not necessarily mean that they have different demographic characteristics.

Want more information about sampling?

Read the book: Sampling of Populations: Methods and Applications (1999) by Paul S. Levy, Stanley Lemeshow

Last Updated ( Sunday, 24 September 2006 )

[ Back ]