METHODS READINGS AND ASSIGNMENTS

GUIDE 1: INTRODUCTION GUIDE 2: VARIABLES AND HYPOTHESES GUIDE 3: RELIABILITY, VALIDITY, CAUSALITY, AND EXPERIMENTS GUIDE 4: EXPERIMENTS & QUASI-EXPERIMENTS GUIDE 5: A SURVEY RESEARCH PRIMER GUIDE 6: FOCUS GROUP BASICS GUIDE 7: LESS STRUCTURED METHODS GUIDE 8: ARCHIVES AND DATABASES

OVERVIEW

This is the first of several guides that will be published on the Internet this semester. Be sure to come back because I will add links to new sites over the course of the next few months, making it easy for you to cross reference topics. All guides will be linked to the Overview and Readings WEB sites and placed in the Blackboard system.

WHAT METHODS DO

In Methods of Educational Research, we study several "quantitative" and "qualitative" methods--or, in my preferred terminology, more or less structured research designs for collecting data. We will deal with empirical research, i.e., tangible data which is assessed using the evidence of our senses. While these kinds of data collection methods are not the only way in which we "know" things, they have particular utility for testing research hunches and hypotheses in a variety of fields.

Research Methods are most often used for two major purposes:

(1) To establish "facts" or recurring regularities in the environment. Examples of facts include:

• The incidence of violence on high school campuses.
• The percentage of adult Koreans with access to the Internet at least once a week.
• How many American adults over 18 engage in physical exercise.

• Students who engage in explosive violence on high school campus have been bullied at that school.
• Science or technology professionals have greater access to the Internet at work than other workers do.
• Men more often engage in physical exercise than women.

As soon as we try to establish causal precedence, things become even more difficult. For every pair of factors that we see locked in a causal relationship:

• We could mistake the direction of causality.  For example, recent work on parents who physically discipline (spank!) their children found that nearly 19 out of every 20 parents use some form of physical discipline. The rare children whose parents never spanked them were found to have exemplary behavior. The assumption was that parental discipline patterns influenced children's behavior. BUT, isn't it possible that "exemplary children" never even tempted their parents to use physical discipline in the first place? That is, the true causal variable here was the behavior of CHILDREN, rather than parents.

• Any apparent causal relationship occurs because a third factor caused both the orginal "cause" and also the "effect." In other words, the relationship is "spurious," and not a "true" or "real" causal relationship. For example, several decades ago, researchers found that American high school students who smoked cigarettes had lower grades. Their conclusion was that something about smoking caused lower grades. Leaving aside the reversed causal possibility (your grades were so awful, you began smoking to relieve the stress), later scholars found that the "true cause" was parental social class. High school students who came from poorer backgrounds were both more likely to smoke cigarettes and also had lower grades. Once parental background was controlled, student cigarette smoking no longer predicted grade point average. Spurious relationships appear in experimental studies too; for example, your results may be due to anxiety aroused by being in a testing situation or an artifact of a particular treatment manipulation.

• A very recent example of misapplied causal inference is that of Hormone Replacement Therapy (HRT) in postmenopausal women. Early studies reported that women taking estrogen/progesterone hormone supplements had lower rates of heart attacks and lower odds of osteoporosis than women who did not take these hormones. The data appeared so impressive that many doctors did not wait for more conclusive experimental results in their recommendations, so that by early 2002, over SIXTEEN MILLION U.S. WOMEN were on HRT. However, in the early 2000s, a massive experimental study was begun. Half the U.S. women received HRT and the other half received a sugar pill placebo. The women were followed longitudinally. To the researchers' shock, the experimental data indicated that women on HRT, in fact, had HIGHER rates of heart attacks and strokes. Although the incidence was still low, the data were convincing enough the experiment was immediately terminated and millions of postmenopausal women are now uncertain of what medication course to follow.

How could this happen? Women who took very good care of themselves: (A) were more likely to see their doctors and thus receive HRT in the observational studies and (B) women who take good care of themselves have a lower incidence of heart attacks in general. The TRUE causal factor, apparently, is the level of responsibility that individual women take for their physical well-being. Although the data are still far from all in, it appears that this is one case where incorrect causal inferences in observational data were literally lethal.
 
• Your results were caused by alternative causal variables, leaving your original causal explanation suspect. For example, I recently found that the level of basic science knowledge in American adults was somewhat higher among men than among women. People who have read this material conclude that women are just less knowledgeable about science. HOWEVER, I later discovered that much of the difference occurred not only because women gave more incorrect answers than men, but also because women gave more "I don't know" responses than men did. Issues such as self-efficacy become more important in giving "I don't know" responses than incorrect ones.

STAGES OF METHODOLOGIES

In this course, we will study several different types of research designs. However, all these designs also share some common similarities and a well-planned sequence of activities. I will mention some basic ones now, and we will study these issues in more depth over the semester:

• Being able to develop a research problem

• Deciding on the unit of analysis (individual? group? organizations?) and taking measures that are consistent with that unit of analysis

• Deciding how to sample one's chosen units

• Deciding how to measure one's concepts via choice of method (experiment? survey questionnaire, including "tests"? archival search? etc.)

• Once a method has been chosen, deciding on actual measures and procedures, such as questionnaire items or experimental treatments

• Pilot testing one's measures and double-checking the results

• Moving "into the field" to collect data

• Making contact with subjects and respondents, including Institutional Review Boards (Human Subjects Committees) and any organizational representatives.
• Training field staff
• Supervising or conducting data collection
• Reducing the collected data to manageable size by selecting coding categories and coding the data

• Analyzing the data

• Reporting the results

Most of us, when we begin to write up professional research, like to start writing our papers like storytellers. We discuss an interesting recent research finding. We describe a compelling social problem. Very often, the "meat" of our study does not even emerge until the fifth typewritten page. Besides making it very difficult for your reader, who must scrutinize your vivid prose for several pages to learn what it is that you will even study and the topic of your research, this written procrastination serves as a signal that you are not really sure what your research is about!

When I work with students on research projects, I am adamant that somewhere on the first page of writing, a student must tell me:

What the project is about.  Anxiety and testing results? Hormone fluxuations and sports participation? Motivation tools and sports team performance?

Why this project is important. Why it is a subject worthy of study.  Will it cure a social problem? Will it diagnose a learning disability? Will it help individuals achieve a higher performance? Will it extend scholarship in the discipline?

What specifically will be done in this study.  An examination of how gender and educational type and level influence science knowledge in survey data? An experiment with social identity threat and pain tolerance? An observational study of group dynamics on football teams?

This combination of elements constitutes your research problem statement: the general area of your research, why this research area is important, and what specifically you will study.

Your research problem statement will also address:

  Your key conceptual variables and definitions of these variables.

Postulated causal relationships among these variables (or, conceptual hypotheses).

Writing a research problem statement will be THE MOST DIFFICULT ASSIGNMENT you will have all semester, and you will rewrite it a few times over the next several weeks.

HOW TO GET STARTED

If you are having trouble conceptualizing a research problem, you are not alone. This is typically the most difficult stage of conducting research. Further, in less structured research, you may be constantly revising the research problem as you gather data, and you may do so in any kind of research if you encounter surprising and unanticipated results. Nevertheless, here are several "tried and true" ways to begin.

CONCEPTUAL AND DEDUCTIVE APPROACH. You are thoroughly familiar with the literature in your area (say, self-regulated learning) and you are aware of gaps where theory has not yet been tested, or where theoretical predictions contradict one another, or you derive your research problem from some basic theoretical assumptions. For example, perhaps you compare the reading assessment scores of elementary school children taught via "whole language learning" versus "phonetics".

CURIOSITY.   Intrigued by regularly occuring "facts," you wish to know more about why and how those factsw occur. You may be dissatisfied with previous explanations. For example, why does educational level affect basic science knowledge? Is it the type of college major? Stimulating an interest in science? "Weeding out" the less intelligent? Holding a scientific or technical job?

You may encounter a suprising, unanticipated "serendipitous" finding that begs for an explanation. Your guesses about why this anomalous result occurred become the basis of defining your research problem. Example, several decades ago, researchers on achievement motivation discarded women subjects because their results did not "fit" the researchers' paradigm. Encountering this unexplained quirk in a footnote in my textbook, I have been examining issues in gender ever since.

IT'S THE MONEY, HONEY. Your major professor or your client defines the research problem and you conduct the study. In my experience, working for a client can be the most difficult way to begin because the client often has a very fuzzy idea at best of what they want to know or do. You often end up defining, or at the least, clarifying and refining the research problem for the client. Alternatively you are looking for grant support and write a proposal conforming to the grant parameter descriptions.

STILL STUCK? CHECK THIS SITE OUT!

KNOW YOUR TOPIC

There is no substitute for knowing your topic well. Most methods textbooks have excellent chapters that describe literature searches. Online search engines and journal or abstract services cut the time involved tremendously and alert you to new sources of information. Check out the links to various organizations (many of them sponsor journals) in the RESOURCES section of our Blackboard course.

Collect as many relevant study designs as you can. I have a file cabinet filled with survey research questionnaires on all kinds of different topics.

Talk with your clients, speak with members of your proposed participant pool. Find out which aspects of your research problem are the most important to them.
 

One way to continue working on your research project is to start a flow chart (HINT: take a look at the Inspiration computer program, available in our Learning Resource Center). Diagram your key variables and the types of relationships among variables that you expect to find. Such a chart will alert you to the concepts you need to measure.

Each global concept, such as "reading assessment" or "instructional design plan" has a number of variable components and alternative definitions. Be alert to this multiplicity of definitions and make clear what your definition is, what your key variables are, and what is or is not an instance of your definition.

A variable is a characteristic or factor that has values that vary, for example, levels of education, intelligence, or physical endurance.

A variable has at least two different categories or values. If all cases have the same score or value, we call that characteristic a constant, not a variable.

CONCEPTUAL VARIABLES are what you think the entity really is or what it means. YOU DO NOT DISCUSS MEASUREMENT AT THIS STAGE! Examples include "achievement motivation" or "endurance" or "group cohesion". You are describing a concept.

On the other hand, OPERATIONAL VARIABLES (sometimes called "operational definitions") are how you actually measure this entity, or the concrete operations, measures or procedures that you use to measure the variable.

You usually begin your research problem with CONCEPTUAL VARIABLES and the relationships among them. One of the few exceptions is if your actual purpose is to study a particular operational variable, for example, perhaps you want to study the validity of the FCAT test, the achievement assessment test that kindergarten through twelfth grade students in Florida selected grade levels must take each year.

We will spend considerable time in the next week examing causal issues. For right now, you need to know about independent and dependent variables.

Causes are called INDEPENDENT VARIABLES.

If one variable truly causes a second, the cause is the independent variable. Speaking more statistically, variation in the independent variables comes from sources outside our causal system or is "explained" by these sources.

Independent variables are often also called explanatory variables or predictors.

Effects are called DEPENDENT VARIABLES.

Statistically speaking, we "explain" the variation in our dependent variable.

Dependent variables are also sometimes called outcome or criterion variables.

A research problem will describe the causal relationships between independent and dependent variables and explain how these relationships come to be.

• 1. Good research takes time. "Overnight polls" have terrible response rates and dubious generalizability. Experiments must be pilot tested, for example, to see if subjects even noticed your treatment manipulations (manipulation checks). Ethnographies can take months, or even years. No method can be done in a hurry. Even if you are only  in the field a short time, allow enough time for planning and pilot testing in your research.

• 2. No one study disproves (or worse yet, "proves") anything. While we like to think of the "definitive experiment," each study has strengths and weaknesses. Perhaps one cannot generalize well to a known population of individuals (groups) or situations (NOTE: this is EXTERNAL VALIDITY). Perhaps there are alternative causal explanations about what caused the outcomes (NOTE: this addresses INTERNAL VALIDITY). An aggregate of studies are usually needed to make strong assertions about the phenomena under study.

• 3. Always, ALWAYS pilot test before you go into the field. This way you will catch problems with the experimental manipulations, difficulties with field observational categories, strange ways that respondents interpret your survey research questions and much more. If you are using any type of questionnaire, be sure that you pilot test at least once by reading questions aloud (even if the questionnaire will be self administered).

• 4. Try to measure your variables as many ways as practicably possible. You want to rule out alternative explanations for your results. Do you want your survey results to represent acquiescence response set instead of substance? Of course not! Do you want your experimental findings to reflect experimenter demand effects instead of treatment effects? Of course not!

lalala
The process of measuring the same concept in different ways is sometimes called "TRIANGULATION." It is one way to try to ascertain CONSTRUCT VALIDITY (i.e., whether your operationalized variables really measure the construct you envisioned--or something else entirely).
 
• 5. Trust participants and respondents. Listen to what they are trying to tell you.  Your respondents may be trying to tell you (subtly, nicely) that they can't understand your questions (your colleagues had no trouble with the professional jargon). Your participants may see the experimental task as ridiculous although they will try to "help out" by completing it anyway. Debrief. You are bound by ethics to do so anyway. Ask your participants what they thought was the purpose of your experiment. Ask a random subsample of respondents to answer the question in their own words or why they answered the way they did (Schuman's "RANDOM PROBE" technique).

• 6. Watch your defined population. Who does it represent? Undergraduate educational psychology students only? All undergraduate college students? High school Spanish students at an upper income facility? Football coaches at AA universities? Graduate students enrolled in distance learning courses? You almost certainly will want to make generalizations later on if you gather quantitative data (or even inappropriately if you gather qualitative data...)

• 7. Try to avoid dichotomies in your measurements whenever possible. Likewise, don't collapse an interval level variable (e.g., years of education) into an ordinal one (unequal educational categories) if at all possible. The computer can aggregate several categories into one category in a matter of seconds (e.g., 9 through 11 years of formal education can be recoded as "some high school"). However, you cannot go the other way: "some high school" cannot be turned into a definitive number.

Try learning to think in conceptual continuums, degrees of "more" or "less" rather than "either" "or." Although your manipulated treatments in an experiment may be categorical, even the manipulations can be "levels" or degrees of a treatment.

• 8. Consider how you will analyze your data once you have collected it or hopefully even BEFORE you have collected.  I know that some of you have not yet taken a course in statistics. Therefore--consult a statistician or a friend/student who has elected several statistics courses. If you do an experiment and want to use analysis of variance, you will need interval level (or "sort of numeric" anyway) measures for your dependent variables. Regression typically requires interval dependent variables (see your statistics instructor for variations on this theme). If all your variables are nominal you will be more limited in the analytic methods you can use.

• 9. It is very difficult, and sometimes impossible, for sophisticated means of data analysis to compensate for poor data collection.  If your response rate is poor, your results probably cannot be generalized to any known population. Some behavioral scientists engage in elaborate weighting schemes so their data appear to be "representative" but the problem is that we seldom know how those who responded differed from those who refused or could not be located. If your measures are contaminated by response bias, you will not be able to disentangle effects without gathering more data. If you find out later that you neglected to measure important variables, you may no longer have access to your population to collect more information.

• 10. There is no such thing as "value free" research. Researchers are human beings who are the captives of their culture. This includes considering only research that produces "statistically significant" differences as "important" (consider for a moment what this perspective did to the field of "sex differences" if ONLY research finding differences got published).


So what is there to do? Try to understand your own values and how they might introduce biases into your research. Safeguard against your own biases. Don't do your own interviewing, don't open code your own data, and don't serve as your own experimenter. Trade off with another graduate student or seek funding to hire people for these positions.

Talk with men (if you are female) or women (if you are male). Show your research design to friends from other cultural backgrounds to see if your ideas--or your treatments--or your questionnaire items--might be misconstrued.
 

• 11. Your research will take at least twice as long and be at least three times as much trouble as you ever thought it would before you got started. Trust me on this one! Experimental participants don't show up and must be rescheduled. The survey lab goes broke while you are in the field (this one really happened to me). You must locate someone who speaks the language fluently. Allow time for the Human Subjects Committee to examine your design. You had to go out of town and the client pilot-tested on the wrong population (I had this one happen too.) MORAL: try to trouble shoot as much as you can at the very beginning!

• 12. DO THE BEST YOU CAN WITH WHAT YOU GOT.  No study (including mine or yours) will be perfect. You almost certainly will have a less than ideal level of funding (bake sale level, maybe?). You will have less than ideal assistance and your time will be constrained. If we all waited for perfection, we would never study anything. So, relax and enjoy!

Susan Carol Losh
September 2  2002
Revised February 10 2009
This page was built with Netscape Composer
 
 

METHODS READINGS AND ASSIGNMENTS

OVERVIEW