OVERVIEW

COURSE PAPER TOPIC STATEMENT DUE SEPTEMBER 16 WILL YOU GATHER DATA THIS SEMESTER? IF YES, APPLY TO FSU'S HUMAN SUBJECTS COMMITTEE. CLICK HERE .

SYLLABUS

Most social psychologists regardless of disciplinary department home are empiricists, that is, we test out our conceptually-derived hypotheses with systematically gathered data. If the data contradict or disconfirm our hypotheses, then typically we revise and retest the theory. On the other hand, social psychologists are eclectic. Most of us use a variety of methods, such as experiments or surveys or focus groups, depending on the topic at hand. While it is true that psychologists are relatively more likely to use experimental methods and sociologists are relatively more likely to use surveys, ethnographies, or observational methods, all of us tend to switch methods depending on the kind of research topic.

Let me give some personal examples.

• When I wanted to study group processes in religious congregations, I used participant observation. I coordinated over 30 observers in 38 churches and synagogues, who attended weekly services, socials, and other congregational events. We gathered data on interaction patterns, the formation of smaller groups within each congregation, how often individuals attended, and we monitored invitations to lunch, conregational parties, and other occasions. I used participant observation because I wanted to see how each group interacted in a natural setting.

• When I study science attitudes, generally I work with public opinion survey research data. While there are a variety of ways to study how someone feels (including unobtrusive observational methods), probably the most direct method is to ask them! Survey questionnaires allow the researcher to gather considerable amounts of information in a relatively short time. Further, I am dealing with individuals, not groups, which makes it appropriate to use an individual questionnaire.

• Researchers have studied children's drawings of scientists for 50 years. One assumption, particularly for young children, is that these drawings function as projective tests. Thus, a Black girl who draws a scientist as a White male is assumed to be saying "being a scientist is not for me." Other scholars have gone further and stated these drawings reflect science self-efficacy. But what if young children, still developing fine motor control, draw all occupational incumbants the same? Using a combination of experimental and "artistic" methods, my students and I content analysed over 600 children's drawings. we found  that even first graders do distinguish among occupations (but almost everyone drew scientists as the least attractive).

• On the other hand, people often cannot tell you what reasoning processes  they use or about factors such as cognitive overload because these processes may be preconscious (there is interesting work about training people to reason aloud, but that is for another day) or poorly understood by the individual. In such instances, we can use experiments, vary the conditions, and assess how reasoning was affected.

Causality is critical to the research enterprise. Much of the research process centers around designating the "true causal" or “independent variables.” What we initially may consider to be “true causal” variables may, instead, turn out to be artifacts of the research process (e.g., evaluation apprehension in response to being an experimental subject), the particular group that we studied (college undergraduates at research universities), or thinking that we had measured variable one while we had, in fact, studied variable two. A great deal of behavioral science consists of ruling out alternative causes or explanations.

Establishing causality is by no means agreed-upon even among  scientists, let alone most members of a particular culture. If you subscribe, at least in part to the "rational laws" approach, you also probably accept controlled experiments, logic, "reasonable arguments,” and statistical control as suggestive of "proof." Science is one form of knowing and one generic way of gathering evidence that either disconfirms or suggests causality.

According to science rules, definitive proof via empirical testing does not exist. Science uses the term "proof" (or, rather, "disproof") differently from the way attorneys or journalists do. Our measurements could be later shown to be contaminated. A correlation could have many causes, only some of which have been identified. Later work can show earlier causes to be spurious, that is, both cause and effect depend on some prior causal (often extraneous) variable. Statistics are NEVER EVER considered to "prove" anything although statistical results CAN disconfirm or "support". It sounds overly tentative but in fact is simply the cautious voice of experience.
 

In order to make any causal assessments in your research situation, you must have reliable measures, i.e., (in part) stable measures. If the random error variation in your measurements is so large that there is almost no stability in your measures, you can't explain anything. Picture an intelligence test where an individual's scores ranged from moronic to genius level over a short period of time. No one would place any faith in the results of such a "test" because the person's scores were so unstable or unreliable.

Reliability is a necessary but not sufficient condition to make statements about validity. Reliable measures could be  biased and hence "untrue" measures of a phenomenon) or confounded with other factors such as acquiescence response set. Picture a scale that always weighs five pounds too light. The results could be reliable, but they would be inaccurate or biased. Or, picture an intelligence test on which women or people of color always score lower (even if this doesn't occur on other tests). Again, the measure may be reliable but biased.

Internal Validity

Internal validity addresses the "true" causes of the outcomes that you observed in your study. Strong internal validity means that you not only have reliable measures of your independent and dependent variables BUT a strong justification that causally links your independent variables to your dependent variables. At the same time, you can rule out extraneous variables, or alternative, often unanticipated, causes for your dependent variables.  Thus strong internal validity refers to the unambiguous assignment of effects to causes.  Internal validity is about causal control.

Laboratory "true experiments" have the potential to make very strong causal control statements. Random assignment of subjects to treatment groups (see below) rules out many threats to internal validity. Further, the lab is a controlled setting, very often the experimenter's "stage." If the researcher is careful, nothing will be in the laboratory setting that the researcher did not place there. When we leave the lab to do studies in natural settings, we can still do random assignment of subjects to treatments, but we lose control over potential causal variables in the study setting (dogs bark, telephones ring, the experimental confederate just got run over walking against the "don't walk" sign on Stadium Drive.)

External Validity

External validity addresses the ability to generalize your study to other people and other situations. To have strong external validity (ideally), you need a probability sample of subjects or respondents drawn using "chance methods" from a clearly defined population (all registered students at Florida State University, for example). Ideally, you will have a good sample of groups (e.g., classes at all ability levels). You will have a sample of measurements  and situations (you study who follows the confederate who violates the "don't walk" sign on Woodward Avenue at different times of day, different days, and different locations on campus.)  When you have strong external validity, you can generalize to other people and situations with greater confidence.
 

Each major method that social psychologists use has its advantages and disadvantages. A method may work better to study some areas than others. For example, it is critical to assess whether you are studying individuals or groups. A method that can work well to study individuals, such as a survey questionnaire, can work poorly to study groups.

EXPERIMENTS

• stress CAUSAL CONTROL (internal validity) both in the study setting (think stage play) and in manipulating independent variables (causes).

• A "true experiment" randomly assigns subjects to treatment conditions, that is, the treatment an individual receives is determined by a coin flip, a random number table, or some other similar probability or chance device. Which treatment someone receives is basically determined by a process akin to a lottery.

• Random assignment or "randomization" controls differences in life experiences, etc. In randomized groups, ON THE AVERAGE, there will be about the same percentage of women, the same percentage of highly test anxious people, the same percentage of good athletes, the same percentage of "high risk" students, etc.

Lab participants also typically have too similar backgrounds (e.g., college students or military recruits are about the same age and the same educational level) to allow high external validity.

Lab experiments are reactive (obtrusive).  Generally, participants are acutely aware that their behavior is under scrutiny. They may react with anxiety (evaluation apprehension), with the desire to "fake" good (social desirability) and appear smarter, more attractive, or more tolerant than they typically do in their everyday behavior.

It's a good idea to use a manipulation check to be sure that on some level the subjects noticed the manipulation. If you manipulate a variable and the different treatments never register, the treatment probably won't make a difference. For example, if you use a "buddy system" in a stop-smoking group, we would expect the subjects to rate the group as friendlier than subjects who don't receive the "buddy system" treatment.

Field experiments

• occur in natural settings but still preserve random assignment of participants to treatments thus, they correct some of the problems with laboratory experiments.

• have less control over the study setting than laboratory experiments.

• are typically short in duration, making it difficult to research complex, in-depth events with this method. It will be impossible to ascertain changes over time.

• since you usually do not even have the names of experimental participants in field experiments, any kind of followup becomes impossible. This is critical if the phenomenon you are studying has "sleeper effects," i.e., the effects of a treatment take time to develop and may not be apparent until days, or even weeks, after the initial treatment period.

• In these methods, data are collected in natural settings.

• They have high mundane reality and greater INSIGHT, especially participant observation, which usually studies groups in-depth.

• In field observations, participant observation, or natural experiments, we  observe and infer rather than manipulate causal factors so these methods have lower causal control.

• Participant observers join the studied group and share many group activities. Maintaining empathy and objectivity simultaneously is difficult in this case. Participant observation is typically one facet of conducting an ethnography, or a description of group culture.

• In natural experiments, you have virtually no causal control over the independent variables. A natural experimental study takes advantage of a naturally occurring event and may, at best, have limited access to data before the event occurred. For example, in the late-1990s, Texas graduate and professional schools became forbidden by state law to consider ethnicity in their admissions. Within one year after passing this law, Hispanic and Black applications to the University of Texas Law School dropped by 25 percent. The guess is that changes in the law caused changes in the ethnic composition of applicants--but without a control group, this conclusion is a tenuous one. Other factors could have been responsible for this drop. Or, suppose you wanted to see how people work together during an emergency. Hurricane Katrina roars through your community and off you go to study cooperation. However, hurricanes can change course (as Katrina did; at one point it appeared headed straight for Tallahassee) and strike in unexpected places and you may have trouble even travelling to the site. And, of course, you can't change when the hurricane grew or where it hit.

• In field observations, observations using systematic categories for group or individual behavior are made in a natural setting. There is not the same kind of intense involvement as there is in participant observation and, in fact, observers may not even be noticed by research participants or respondents. For example, someone who codes seatbelt wearing behavior may not be noticed by passing motorists.

Public opinion SURVEYS typically take REPRESENTATIVE SAMPLES thus have relatively high external validity. In surveys, questions are asked and the answers are recorded, sometimes by interviewers and sometimes using paper-and-pencil questionnaires.

• In most professional surveys, the results may be generalized to a known population with known levels of error.

• Establishing internal validity is only fair and often depends on question wording. You have a better chance with demographics that were present from birth or early on (e.g. gender or age) than with other cognitive measures. Response rate is important here too, especially if there is any indication that respondents differ from non-respondents.

• Mundane reality is medium. Survey questions often discuss a respondent's daily life BUT the interview itself can be  reactive and can evoke  demand effects (e.g.,social desirability or evaluation apprehension).

• Further, answers may be retrospective, relying on the participant's memory--which may be inaccurate (the Nixon-McGovern 1972 election voter recall is a great example!)

• Did you know? Standardized tests are a form of survey research and are subject to most of the same problems that surveys have such as question format bias.

It is incorrect to assert, as many research methods textbooks do, that internal validity or causal control can only be accomplished through experiments. Browse through the American Educational Research Association's excellent book about establishing causality in observational and survey designs of various kinds HERE. (You actually can download the entire volume in pdf through this site.)

Researchers may use content analysis to study media or other documents. For example, you might study gender stereotypes in sports magazines or the frequency of unmarried couples cohabiting on television situation comedies. Causal control is low but mundane reality can be high. External validity can be high with careful sampling. Content analysis has been cautiously combined with surveys to show how watching television can influence adult attitudes and behaviors (such as "The Mean World Syndrome").

Simulations occur in constructed settings. Participants may role play or act "as if" they held certain roles such as a parent, a prison guard or head of state. A classic example is the Haney, Banks and Zimbardo prison study, which was both a simulation and an experiment; you can access a slide show of this study at Philip Zimbardo's web site HERE.

• Since subjects may have limited exposure to such roles, their behavior will have low mundane reality and may be artificial.

• When participants are randomly assigned to the roles they play (as they were in the Haney, Banks and Zimbardo study), the simulation is experimental and thus has high causal control.

• Simulations can be dramatic and INSIGHTFUL but the "in-role" behavior may reflect the fantasies that participants have about the role or TV portrayals if the roles are unfamiliar ones.This may have happened in the Haney, et al., prison study; the Stanford University students may have applied stereotypes about prisoners and prison guards that they learned from movies or TV.

• Conversely, simulations can be very revealing if the situations are ones the subjects have encountered first hand with relative frequency. Family role-playing is one example, wherein the subjects project their family experiences and expectations into the simulation situation.

The chart below synopsizes relative advantages and disadvantages of each method. Notice that a strong point for one method, such as strong internal validity for laboratory experiments, is often balanced by a weak point, such as the limited external validity in most laboratory experiments because of the reliance on "captive" participants such as undergraduate college students or military recruits. This means that the wise researcher will use a variety of methods. When different methods to study the same phenomenon produce comparable results, we call this triangulation. You also should carefully consider the evidence in the studies that you read about, both in your assigned readings and also when you review articles and other readings for your course project:

• Would you feel comfortable generalizing the data to different groups or individuals? Would the same results be expected for women and men? Youth and adults? Different ethnic groups? Did the researchers even consider different populations?

• How tight was causal control? Were control groups truly comparable to the experimental groups? How conscious were subjects that their behavior was being monitored?

• Was there any followup on subjects or respondents? Were the changes in behavior that were observed stable?

• Was the unit of study a group or an individual? Were the measures appropriate? Generally, surveys do not do a good job of describing group behavior. It is difficult to infer individual feelings from simply watching interaction patterns.

THE UNIT:  Since we study individuals, but we also study groups and organizations, it is imperative to make your measures consistent with the unit you have chosen.

THE TIME FACTOR: Because some results fade quickly (e.g., your specific attitudes about almost anything political) and others are "sleepers" or increase over time, you should assess whether followup over time was needed--and whether it was done.

REACTIVITY: A classic social psychology study found that simply viewing oneself in a mirror changed behavior. Picture, then, the effects of being in a research study! Was there any control for reactivity? Sometimes that does mean we use deception in experiments. After all, if you told people "this experiment is to assess your attractiveness," we can be pretty sure that people would immediately "up their image."

NOTE: The chart below refers to the "usual" or "typical" professional  study. Any specific study, of course, may be "better" or "worse".

 

ALPHABETIC ABBREVIATION  IN CHART BELOW	TYPE OF METHOD
A	Laboratory Experiments
B	Field Experiments
C	Participant Observation (often ethnographic)
D	Field Observations
E	Surveys
F	Secondary Analysis
G	Simulations
H	"Natural" Experiments

METHOD	A	B	C	D	E	F	G	H
Causal Control  (Internal Validity)	Excellent	Fair to  Very Good	Fair to Good	Fair	Fair to Good	Fair	Can be  Good	Poor
Control over Other Factors	Good	Fair	Fair	Poor	Fair	Fair	Can be  Good	Poor
Represents Population  (External Validity)	Often Poor	Fair to Good	Fair to Good	Fair	Very Good	Can be Good	Poor	Varies
Time Follow-up	Often Poor	Very Poor	Good	Fair	Can be Good	Can be Good	Poor	Can be  Good
Mundane Reality	Often Low	High	High	High	Fair to High	Varies	Low	High
Reactivity	High	Low	Middle	Middle	MAY be high	Low	High	Low
Unit of Study	Person or Group	Person or Group	Group	Person or Group	Person	**	Person or  Group	Often Group

**Varies with the original method of collecting data which is specific to each research study. For example, the unit may be a person, a group, a cultural reproduction such as a television show or magazine, or a created cohort.
 

OVERVIEW

SYLLABUS

This page was built with Netscape Composer.
Susan Carol Losh August 23 2009

New Orleans September 2005