Selecting the type of survey you are going to use is one of the most
critical decisions in many social research contexts. You'll see that there
are very few simple rules that will make the decision for you -- you have
to use your judgment to balance the advantages and disadvantages of
different survey types. Here, all I want to do is give you a number of
questions you might ask that can help guide your decision.
Population
Issues
The first set of considerations have to do with the population and its
accessibility.
- Can the population be enumerated?
For some populations, you have a complete listing of the units that
will be sampled. For others, such a list is difficult or impossible to
compile. For instance, there are complete listings of registered voters
or person with active drivers licenses. But no one keeps a complete list
of homeless people. If you are doing a study that requires input from
homeless persons, you are very likely going to need to go and find the
respondents personally. In such contexts, you can pretty much rule out
the idea of mail surveys or telephone interviews.
- Is the population literate?
Questionnaires require that your respondents can read. While this
might seem initially like a reasonable assumption for many adult
populations, we know from recent research that the instance of adult
illiteracy is alarmingly high. And, even if your respondents can read to
some degree, your questionnaire may contain difficult or technical
vocabulary. Clearly, there are some populations that you would expect to
be illiterate. Young children would not be good targets for
questionnaires.
- Are there language issues?
We live in a multilingual world. Virtually every society has members
who speak other than the predominant language. Some countries (like
Canada) are officially multilingual. And, our increasingly global
economy requires us to do research that spans countries and language
groups. Can you produce multiple versions of your questionnaire? For
mail instruments, can you know in advance the language your respondent
speaks, or do you send multiple translations of your instrument? Can you
be confident that important connotations in your instrument are not
culturally specific? Could some of the important nuances get lost in the
process of translating your
questions?
- Will the population cooperate?
People who do research on illegal immigration have a difficult
methodological problem. They often need to speak with illegal immigrants
or people who may be able to identify others who are. Why would we
expect those respondents to cooperate? Although the researcher may mean
no harm, the respondents are at considerable risk legally if information
they divulge should get into the hand of the authorities. The same can
be said for any target group that is engaging in illegal or unpopular
activities.
- What are the geographic restrictions?
Is your population of interest dispersed over too broad a geographic
range for you to study feasibly with a personal interview? It may be
possible for you to send a mail instrument to a nationwide sample. You
may be able to conduct phone interviews with them. But it will almost
certainly be less feasible to do research that requires interviewers to
visit directly with respondents if they are widely
dispersed.
Sampling
Issues
The sample is the actual group you will have to contact in some way.
There are several important sampling issues you need to consider when
doing survey research.
What information do you have about your sample? Do you know their
current addresses? Their current phone numbers? Are your contact lists
up to date?
- Can respondents be found?
Can your respondents be located? Some people are very busy. Some
travel a lot. Some work the night shift. Even if you have an accurate
phone or address, you may not be able to locate or make contact with
your sample.
Who is the respondent in your study? Let's say you draw a sample of
households in a small city. A household is not a respondent. Do you want
to interview a specific individual? Do you want to talk only to the
"head of household" (and how is that person defined)? Are you willing to
talk to any member of the household? Do you state that you will speak to
the first adult member of the household who opens the door? What if that
person is unwilling to be interviewed but someone else in the house is
willing? How do you deal with multi-family households? Similar problems
arise when you sample groups, agencies, or companies. Can you survey any
member of the organization? Or, do you only want to speak to the
Director of Human Resources? What if the person you would like to
interview is unwilling or unable to participate? Do you use another
member of the organization?
- Can all members of population be
sampled?
If you have an incomplete list of the population (i.e., sampling
frame) you may not be able to sample every member of the population.
Lists of various groups are extremely hard to keep up to date. People
move or change their names. Even though they are on your sampling frame
listing, you may not be able to get to them. And, it's possible they are
not even on the list.
- Are response rates likely to be a
problem?
Even if you are able to solve all of the other population and
sampling problems, you still have to deal with the issue of response
rates. Some members of your sample will simply refuse to respond. Others
have the best of intentions, but can't seem to find the time to send in
your questionnaire by the due date. Still others misplace the instrument
or forget about the appointment for an interview. Low response rates are
among the most difficult of problems in survey research. They can ruin
an otherwise well-designed survey effort.
Question
Issues
Sometimes the nature of what you want to ask respondents will determine
the type of survey you select.
- What types of questions can be asked?
Are you going to be asking personal questions? Are you going to need
to get lots of detail in the responses? Can you anticipate the most
frequent or important types of responses and develop reasonable
closed-ended questions?
- How complex will the questions be?
Sometimes you are dealing with a complex subject or topic. The
questions you want to ask are going to have multiple parts. You may need
to branch to sub-questions.
- Will screening questions be needed?
A screening question may be needed to determine whether the
respondent is qualified to answer your question of interest. For
instance, you wouldn't want to ask someone their opinions about a
specific computer program without first "screening" them to find out
whether they have any experience using the program. Sometimes you have
to screen on several variables (e.g., age, gender, experience). The more
complicated the screening, the less likely it is that you can rely on
paper-and-pencil instruments without confusing the
respondent.
- Can question sequence be controlled?
Is your survey one where you can construct in advance a reasonable
sequence of questions? Or, are you doing an initial exploratory study
where you may need to ask lots of follow-up questions that you can't
easily anticipate?
- Will lengthy questions be asked?
If your subject matter is complicated, you may need to give the
respondent some detailed background for a question. Can you reasonably
expect your respondent to sit still long enough in a phone interview to
ask your question?
- Will long response scales be used?
If you are asking people about the different computer equipment they
use, you may have to have a lengthy response list (CD-ROM drive, floppy
drive, mouse, touch pad, modem, network connection, external speakers,
etc.). Clearly, it may be difficult to ask about each of these in a
short phone interview.
Content
Issues
The content of your study can also pose challenges for the different
survey types you might utilize.
- Can the respondents be expected to
know about the issue?
If the respondent does not keep up with the news (e.g., by reading
the newspaper, watching television news, or talking with others), they
may not even know about the news issue you want to ask them about. Or,
if you want to do a study of family finances and you are talking to the
spouse who doesn't pay the bills on a regular basis, they may not have
the information to answer your
questions.
- Will respondent need to consult
records?
Even if the respondent understands what you're asking about, you may
need to allow them to consult their records in order to get an accurate
answer. For instance, if you ask them how much money they spent on food
in the past month, they may need to look up their personal check and
credit card records. In this case, you don't want to be involved in an
interview where they would have to go look things up while they keep you
waiting (they wouldn't be comfortable with that).
Bias
Issues
People come to the research endeavor with their own sets of biases and
prejudices. Sometimes, these biases will be less of a problem with certain
types of survey approaches.
- Can social desirability be avoided?
Respondents generally want to "look good" in the eyes of others. None
of us likes to look like we don't know an answer. We don't want to say
anything that would be embarrassing. If you ask people about information
that may put them in this kind of position, they may not tell you the
truth, or they may "spin" the response so that it makes them look
better. This may be more of a problem in an interview situation where
they are face-to face or on the phone with a live
interviewer.
- Can interviewer distortion and
subversion be
controlled?
Interviewers may distort an interview as well. They may not ask
questions that make them uncomfortable. They may not listen carefully to
respondents on topics for which they have strong opinions. They may make
the judgment that they already know what the respondent would say to a
question based on their prior responses, even though that may not be
true.
- Can false respondents be avoided?
With mail surveys it may be difficult to know who actually responded.
Did the head of household complete the survey or someone else? Did the
CEO actually give the responses or instead pass the task off to a
subordinate? Is the person you're speaking with on the phone actually
who they say they are? At least with personal interviews, you have a
reasonable chance of knowing who you are speaking with. In mail surveys
or phone interviews, this may not be the case.
Administrative
Issues
Last, but certainly not least, you have to consider the feasibility of
the survey method for your study.
Cost is often the major determining factor in selecting survey type.
You might prefer to do personal interviews, but can't justify the high
cost of training and paying for the interviewers. You may prefer to send
out an extensive mailing but can't afford the postage to do
so.
Do you have the facilities (or access to them) to process and manage
your study? In phone interviews, do you have well-equipped phone
surveying facilities? For focus groups, do you have a comfortable and
accessible room to host the group? Do you have the equipment needed to
record and transcribe responses?
Some types of surveys take longer than others. Do you need responses
immediately (as in an overnight public opinion poll)? Have you budgeted
enough time for your study to send out mail surveys and follow-up
reminders, and to get the responses back by mail? Have you allowed for
enough time to get enough personal interviews to justify that
approach?
Different types of surveys make different demands of personnel.
Interviews require interviewers who are motivated and well-trained.
Group administered surveys require people who are trained in group
facilitation. Some studies may be in a technical area that requires some
degree of expertise in the interviewer.
Clearly, there are lots of issues to consider when you are selecting
which type of survey you wish to use in your study. And there is no clear
and easy way to make this decision in many contexts. There may not be one
approach which is clearly the best. You may have to make tradeoffs of advantages and
disadvantages. There is judgment involved. Two expert researchers may,
for the very same problem or issue, select entirely different survey
methods. But, if you select a method that isn't appropriate or doesn't fit
the context, you can doom a study before you even begin designing the
instruments or questions themselves.
|