NOW ONLINE: THE GUIDE TO EXAM 1. CLICK HERE.
 
 
COMING UP IN THE FUTURE:
ASSIGNMENT 3 SPECIFICATIONS
EXAMPLES OF ASSIGNMENT 3 PROJECTS

 

METHODS READINGS AND ASSIGNMENTS
ASSIGNMENT 2 FEEDBACK AVAILABLE
GUIDE 5:
A SURVEY RESEARCH TIMETABLE

OVERVIEW

 
 
NOTE: Although surveys of adults are classified as "exempt" under federal human subjects regulations, your organization's Institutional Review Board must still quickly check your study design. Most surveys need a few short phrases at the beginning to assure respondents that their participation is voluntary, that they only answer the questions that they want, and (if circumstances warrant, such as a classroom or business) that there is no penalty if respondents withdraw from a study. For more information on Human Subjects check HERE.

 
GUIDE 1: INTRODUCTION
GUIDE 2: VARIABLES AND HYPOTHESES
GUIDE 3: RELIABILITY, VALIDITY, CAUSALITY, AND EXPERIMENTS
GUIDE 4: EXPERIMENTS & QUASI-EXPERIMENTS
GUIDE 5: A SURVEY RESEARCH PRIMER
GUIDE 6: FOCUS GROUP BASICS
GUIDE 7: LESS STRUCTURED METHODS
GUIDE 8: ARCHIVES AND DATABASES

EDF 5481 METHODS OF EDUCATIONAL RESEARCH
INSTRUCTOR: DR. SUSAN CAROL LOSH                        FALL 2002


INTRODUCTION
SAMPLING
ADMINISTRATION
QUESTIONNAIRES 
INTERVIEWING

In survey research, the investigator asks verbal questions and either an interviewer or the respondent records the answers.

If you do these two steps, you are conducting a survey, even if you call it a "standardized test" or a "personality profile" or some other high-falutin' name, or if you embed a questionnaire inside your experiment. Public opinion surveys or polls are simply one special kind of survey, a survey that seeks to generalize its results to a well-defined general population.This means that no matter what you call your research, if you ask questions and record the answers, you must consider the same rules for good question construction, and your data are subject to the same methodological problems as most other surveys are.

Be aware that you are "trying to hit a moving target" in a survey. People change their knowledge, belief, and attitude responses, in part because of the reactivity that occurs as a result of being studied. In terms of reactivity, surveys typically rank below experiments, but above field research such as ethnographies or more secondary collections, such as content analysis. And, of course, people change their beliefs and attitudes as they learn new things, obtain new information, or as a result of persuasion campaigns.

Everyone must be concerned about sampling issues. I elaborate upon them here because excellent sampling is a hallmark of general public opinion surveys. However, sampling directly addresses external validity, and thus is a feature of any kind of study design. Who wants to spend months doing a study, only to conclude that you can generalize to a particular group of students taking an online course and no one else? Most of us are more ambitious than that.

Because you typically deal with "organismic" or naturalistic variables, you have less internal validity in a survey than an experiment but possibly as much as a quasi-experiment. Statistical control, in which you measure as many intervening and confounding variables as possible, then statistically control these, typically substitutes for experimental control. Public opinion surveys generally define their populations very specifically and take careful samples from their populations, thus usually have very good, sometimes excellent, external validity, although internal validity can be difficult to establish.

Review the rules on causal inference and establishing independent or dependent variables in nonexperimental research HERE. Think about these rules when you consider variables to include as independent variables particularly. Be sure that you have included plausible independent variables.

Of course, you could embed an experiment within a survey, and randomly assign who gets which questionnaire version.

Consider what demographics "stand for" in survey research. Variables such as "age," "gender" or "ethnicity" are useful because they condition life experiences. By the same token, these are sometimes confounded variables that are "proxies" for sets of experiences. Older people have different options, have had different life experiences, and are treated differently than younger people. People with different educational levels have had differential access to knowledge, and college students have experienced a whole different culture than those who never went to college. Thus, think about what each demographic stands for and see if you can directly include those variables in your study if they are relevant. (For example, if you believe that a key with "age" is experience handling money, such as wages, checking accounts, or credit, be sure that you include questions related to wages, checking accounts and credit, in addition to "years of age.")

FOR EXAMPLE: Check out the study I co-authored with my students in the September 2003 Skeptical Inquirer (reprinted several places online). We found that most of the variation due to "level of education" was due to something else: either correlates of education level (e.g., age or gender) or specific products of education (e.g., the total number of science classes.)

A second study I published in 2006 compared age and generational effects, which are totally confounded in a cross-sectional survey at one point in time. Virtually all the knowledge effects typically assumed to be due to age are due to generation (birth cohort) instead. This study, which examines science knowledge in the American adult general public was published in Proceedings of The Ninth International Conference on Public Communication of Science and Technology (PCST-9): 836-845, but it can be accessed online HERE. In yet another study, I found that generation influences access to and use of information technology but age generally does not.

There are several steps to conducting a survey. I give the most elaboration to those steps related to collecting data and the study design, leaving data management and statistics to your other methods courses.
 

STEP ONE: DECIDING ON A RESEARCH PROBLEM

REVIEW THIS SITE: ASSIGNMENT 1: CONSTRUCTING A RESEARCH PROBLEM


STEP TWO: SAMPLING

 
CHECK FOR DIFFERENT TYPES OF SAMPLES AT THE SAMPLING WEB SITE, CLICK HERE!

1. Define your population. Make your definition clear enough so that there is absolutely no question about who is or is not in the population.

It's precise, and everyone can agree on who is or is not in the sample. 2. Decide whether your unit or element is a person, or some entity larger than a person, such as a class, a school, or a school district.

3. Locate or construct the most thorough list of your population that you possibly can. This will become your "sampling frame." If a complete list is not possible, see if you can create sampling stages, and then completely enumerate all elements at each stage.

4. At about the same time, you must decide on the type of administration for your survey: self-administered (mail, group, and, increasingly, Internet) or interviewer-administered. Some types of samples, such as simple random samples, can be done with straightforward logistics by telephone (Random Digit Dial or RDD) but are often difficult to do in other circumstances (such as area in-person surveys). The increased use of cell phones rather than landlines, however, has complicated RDD. Except for single institutions (a business or a university, for example), it is often difficult to get a good list of email addresses. Remember that virtually all WEB surveys are self-administered. If your population is not very literate or relatively accustomed to being online, an Internet survey is a very bad idea.

5. Make your final decision on the type of sample to draw. Try for a probability sample. Accept that under certain circumstances this may not be possible. Decide whether in fact you are taking the entirety of a small population instead (a census).

6. Draw the sample. If appropriate, send out an introductory letter. Include inducements (shiny new quarter, Barnes and Noble gift certificate!) if feasible.
 

SAMPLING COUNTS IN EXPERIMENTS TOO! EVEN IF YOU HAVE HIGH INTERNAL VALIDITY, YOU CAN'T LEGITIMATELY GENERALIZE IF YOUR SAMPLE IS POOR (LOW EXTERNAL VALIDITY).
 

STEP THREE: ADMINISTRATION ISSUES

 
IN-PERSON, TELEPHONE, MAIL OR GROUP ADMINISTRATION?
SOME CONSIDERATIONS
.
MAIL
GROUP-ADMINISTERED
TELEPHONE
INTERVIEWS
IN-PERSON
INTERVIEWS
Inexpensive cost is over-rated. Postage, stationary, repeat mail-outs, inducements, raise the cost. Cost is low but external validity may be low too. Current predominant mode for administering general public surveys. Cheaper than in-person surveys. Cell phones create complications such as reimbursing respondents for the minutes used in the survey. Became prohibitively expensive due to more single-person households and employed wives leading to far more "not-at-homes."
Response rates better if respondents motivated about topic and endorsement cover letter is present. REMEMBER! Classes are clusters  that underestimate population variability (unless corrected). One way to compensate is to increase n. Callbacks typically needed. Overnight polls have low response rates. Callbacks needed; cluster samples cut costs THEN sample size is increased to compensate for cluster design effect on variance!
Once repeat mail-outs are included, can take several weeks (minimum) Can be speedy but what about absences? Could they introduce biases? Generally fastest method, even including callbacks. Takes the longest to gather the same sample size, especially with callbacks.
Is your population literate? Can they easily read and respond? Is your population literate? Can they easily read and respond? CATI (Computer Assisted Telephone Interviewing) makes field work faster, easier, more accurate, and skips the data entry stage. Visual aids possible: pictures, show-cards with responses printed on them for respondents to hold. Laptops increase aid with the interview here.
May make it easier to answer sensitive questions. May make it easier to answer sensitive questions. You can use more open-ended questions with interviewers. You can use more open-ended questions with interviewers.
Respondents can look up records. Adminstrator can clarify problems, help with survey directions. Proliferation of machines (answering, fax, etc.) increases contact problems. Respondents LIKE in-person surveys the best.
Average response rate for one-wave with a general population (e.g., all FSU students) runs 20 percent. Awful! Makes a larger case base possible; this helps with later analyses. Increased use of mobile phones as only phone complicates sampling. Interview can be longer than telephone or self-administered survey.

Internet surveys are proliferating. Currently, well over 65 percent of the United State population is "online" from home. This is NOT randomly distributed. The "digital divide" still exists. Single women, minorities, the less-educated all have lower online access. Wealthy agencies, such as Stanford, can afford to give respondents selected in other ways their own Internet connections and laptops so that the whole sample can respond. Web surveys clearly work better with highly literate and motivated respondents. Thus, they become similar to other self-administered questionnaires, such as mail surveys.

Build in safeguards so that respondents can only answer the survey once.

Also remember that YOU should choose the respondents. If you "advertise" and take whoever answers, there's not a whole lot of difference between your survey and dialing in to one of Entertainment Tonight's 900 numbers. Both are self-selected samples.

Presently, WEB surveys appear to work best with subsamples of the population who are already online, who are interested the topic, and with whom the researchers generally stay in touch (e.g., a class, a club).
 


STEP FOUR: QUESTIONNAIRE CONSTRUCTION
 
WE TEND TO ONLY THINK OF FORMAL INTERVIEW SITUATIONS AS "SURVEYS." BUT DOCTORS OR THERAPISTS WHO ASK PATIENTS OR CLIENTS QUESTIONS ALSO SHOULD USE CAREFUL QUESTION CONSTRUCTION AND INTERVIEWING TECHNIQUES. AND "STANDARDIZED TESTS" ARE JUST AS SUSCEPTIBLE TO QUESTION FORMAT RESPONSE SET PROBLEMS AS SURVEYS AND NEED TO FOLLOW RULES FOR GOOD QUESTION CONSTRUCTION.
 
FOR SOME RULES OF GOOD QUESTION CONSTRUCTION, CLICK HERE!
FOR A QUESTIONNAIRE EXAMPLE, CLICK HERE! 

IF YOU NEED ADOBE ACROBAT TO READ THIS FILE, CLICK HERE 
TO INSTALL THE READER ON YOUR HOME COMPUTER.
 
Doing a telephone survey? It's tough to keep respondents on the phone for more than 20 minutes. You will have more luck with an in-person survey with long questionnaires.

Cell phones are proliferating and young, mobile adults are more likely to be "cell phone only." Until very recently, researchers could not [knowingly] random dial cell phone exchanges because many plans charge subscribers for in-coming minutes and it is currently illegal to charge people to take part in a survey. Several studies are underway to see how to ameliorate these problems and we are getting more data (and surveys) from cell phone users. If the individual is on land-line, the researcher faces a thicket of answering machines, caller id, voice mail and other privacy oriented options.

People dislike writing by hand. This includes people with doctorates. If you have many open-ended questions, use an interviewer. Otherwise your sample response rate will be low and your open-ended item response even lower. Concentrate on closed questions for self-administered surveys when possible.

If you are using a self-administered survey, keep page clutter to a minimum. Use a lot of white space. Don't use complicated charts for people to complete.

MAKE SURE TO PILOT TEST YOUR QUESTIONNAIRE ALOUD!

Do this even if the questionnaire will be self-administered.

Pilot test your questionnaire on respondents who are comparable to your selected population on key issues such as education, gender, and ethnicity.

Revise the questionnaire. If needed, do a second pilot test. This is no place to cut corners!

I am out of the University of Michigan Survey Research Center training. Our questionnaires tend to have  a more casual, "chatty" feel to them which we believe encourages responses and relaxes respondents.

The very worst example of the University of Michigan style that I have ever heard--which was only partially presented as a joke--went as follows:

"A lot of people think about suicide. How about you? How many times have YOU thought about killing yourself?"

OUCH. The idea of introducing the "A lot of people..." is so that the respondent will not feel deviant and to legitimate both the question and the respondent's answer. This is supposed to help with questions in sensitive areas, such as drinking, illegal drug use, sexuality, birth control, and failing to keep to your diet.

The toughest question I ever asked on a survey--with the highest nonresponse: "Without shoes, what is your weight in pounds?" Over 10 percent of women would not answer. We made this the last question in the survey, naturally.

And remember, little kids (e.g., 5th grade on down) are not very good at writing, and that includes self-administered surveys...
 
 

STEP FIVE: INTERVIEWER SELECTION AND TRAINING
(IF APPROPRIATE)

 
When was the last time someone asked you what you really thought, cared about what you said, and LISTENED to the answer?
There are several advantages to using interviewers. Interviewers motivate respondents simply by being interested in what respondents have to say. Interviewers can help clarify directions (but NEVER change question wording), can report questions that give problems to the study directors, can probe for more detail.

However, as you might guess, a survey that uses interviewers is more expensive. You must also budget time to train interviewers, and interviewers should be periodically monitored to ensure that they are doing a good job.

The General Social Survey conducts in-person (usually in household) interviews with at least 3000 people every two years. Each survey costs about $3,000,000.

Nearly all professional telephone surveys these days are done via CATI: Computer Assisted Telephone Interviewing. CATI has many improvements over the old paper and pencil questionnaires:

Consider using CATI if your population allows it. If you are doing a WEB survey, consider the advantages in CATI programs, such as building in skip patterns, boxes for short essay responses, a provision to change answers, and moving completed questionnaires immediately to a data file.

Newer hardware/software packages include voice recognition software, although the current quality varies enormously across companies.

You want interviewers who are:

Beware of the interviewer who: Give all interviewers a copy of the questionnaire to study and rehearse in advance until they are thoroughly familiar with the contents.

Have a solid interviewer manual--also read in advance.

Have interviewers practice in teams and practice role-playing respondents.

Have interviewers practice saying in to the mirror (for in-person interviews:) "I'd like to come in and talk with you about this" OR (for telephone interviews:) "The interview will only take about 10 minutes; here's the first question."

MONITOR ALL INTERVIEWS! Check call on in-person interviews. Spot monitor on telephone interviews.
 

Since we are a methods course, not a measurement or statistics course, the detail will stop here. However, here are some further steps typical of surveys (and many other research methods:)

THESE STEPS ARE FOR LATER COURSES:


Not necessary for most CATI programs. Researcher receives diskette or CD with either ASCII data or SPSS format.

If data must be manually entered, select a random sample of cases and re-enter to validate. If large numbers of errors (over 5 percent) are found, consider checking entire database.
 
METHODS READINGS AND ASSIGNMENTS
OVERVIEW

Susan Carol Losh
Updated October 3 2006
Updated January 14 2009
This page was built with Netscape Composer