>
Volume: | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 |
Permission is granted to distribute this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. Please notify the editor if an article is to be used in a newsletter. |
Robert B. Frary,
The purpose of this article is to offer tips in designing quality questionnaires and on avoiding
common errors. Some of the more prevalent problems in questionnaire development are
identified and suggestions of ways to avoid them are offered.
Do keep the questionnaire brief and concise. Some questionnaires give the impression that their
authors tried to think of every conceivable question that might be asked with respect to the
general topic of concern. The result is a very long questionnaire causing annoyance and
frustration on the part of the respondents resulting in non-return of mailed questionnaires and
incomplete or inaccurate responses on questionnaires administered directly. To avoid this first
potential problem the investigator must define precisely the information desired and endeavor to
write as few questions as possible to obtain it. Peripheral questions and ones to find out
"something that might just be nice to know" must be avoided. A clear-cut need for every
question should be established.
Do get feedback on your initial list of questions. Feedback may be obtained from a small but
representative sample of potential responders. A field trial of a tentative form of the
questionnaire is also desirable.
Do locate personal or confidential questions at the end of the questionnaire. The early
appearance of unsettling questions may result in respondents discontinuing the questionnaire.
Do order categories. When response categories represent a progression between a lower level of
response and a higher one, it is usually better to list them from the lower level to the higher in
left-to-right order, for example,
1) Never 2) Seldom 3) Occasionally 4) Frequently
Do consider combining categories. In contrast to the options listed just above, consider the
following:
1) Seldom or never 2) Occasionally 3) Frequently
Combining "seldom" with "never" might be desirable if responders would be very unlikely to
mark "never" and if "seldom" would connote an almost equivalent level of activity, for example,
in response to the question, "How often do you tell you wife that you love her?" In contrast,
suppose the question were, "How often do you drink alcoholic beverages?" Then the investigator
might indeed wish to distinguish those who never drink. When a variety of questions use the
same response scale, it is usually undesirable to combine categories.
Do ask responders to rate both positive and negative stimuli. There is sometimes a difficulty
when responders are asked to rate items for which the general level of approval is high (the
"apple pie" problem). There is a tendency for responders to mark every item at the same end of
the scale. By offering positive and negative responses the respondent is required to evaluate each
response rather than uniformly agreeing or disagreeing to all of the responses.
Do choose appropriate response category language and logic. The extent to which responders
agree with a statement can be assessed adequately in many cases by the options:
1) Agree 2) Disagree
However, when many responders have opinions that are not very strong or well-formed, the
following options may serve better:
1) Agree 2) Tend to agree 3) Tend to disagree 4) Disagree
These options have the advantage of allowing the expression of some uncertainty. In contrast,
the following options would be undesirable in most cases:
1) Strongly agree 2) Agree 3) Disagree 4) Strongly Disagree
Some would say that "Strongly agree" is redundant or at best a colloquialism. In addition, there
is no comfortable resting place for those with some uncertainty.
Avoid open-ended questions. In most cases open-ended questions should be avoided due to
variation in willingness and ability to respond in writing.
Avoid the response option "other." Careless responders will overlook the option they should
have designated and conveniently mark the option "other" or will be hairsplitters and will reject
an option for some trivial reason. An exception to the foregoing advice is any case in which the
categories are clear-cut, few in number, and such that some responders might feel uncomfortable
in the absence of an applicable response.
Avoid category proliferation. A typical question is the following:
Unless the research in question were deeply concerned with conjugal relationships, the
distinctions among all of these categories are not useful. Usually, such a question reflects the
need to distinguish between a conventional familial setting and anything else. If so, the question
could be:
Avoid scale point proliferation. In contrast to category proliferation, which seems usually to
arise somewhat naturally, scale point proliferation takes some thought and effort. An example is:
1) Never 2) Rarely 3) Occasionally 4) Fairly often 5) Often 6) Very often 7) Almost always
8) Always
Such stimuli run the risk of annoying or confusing the responder with hairsplitting differences
between the response levels. Psychometric research has shown that most subjects cannot reliably
distinguish more than six or seven levels of response. Offering four to five scale points is usually
quite sufficient to stimulate a reasonably reliable indication of response direction.
Avoid responses at the scale mid-point and neutral responses. The use of neutral response
positions had a basis in the past when crude computational methods were unable to cope with
missing data. In such cases, non-responses were actually replaced with neutral response values
to avoid this problem. The need for such a makeshift solution has long been supplanted by
improved computational methods. Consider the following questionnaire item:
1) Agree 4) Tend to disagree There is no assurance whatsoever that a subject choosing the middle scale position harbors a
neutral opinion. A subject's choice of the scale midpoint may result from: Ignorance,
Uncooperativeness, Reading difficulty, Reluctance to answer, or Inapplicability.
In all the above cases, the investigator's best hope is that the subject will not respond at all.
Unfortunately, the seemingly innocuous middle position counts, and, when a number of subjects
choose it for invalid reasons, the average response level is raised or lowered erroneously (unless,
of course, the mean of the valid responses is exactly at the scale midpoint).
In the absence of a neutral position, responders sometimes tend to resist making a choice in one
direction or the other. Under this circumstance, the following strategies may alleviate the
problem:
The preceding discussion notwithstanding, there are some items that virtually require a neutral
position. Examples are:
The amount of homework for this course was It would be unrealistic to expect a responder to judge a generally comparable or satisfactory
situation as being on one side or another of the scale midpoint.
Avoid asking responders to rank responses. Responders cannot be reasonably expected to rank
more than about six things at a time, and many of them misinterpret directions or make mistakes
in responding. To help alleviate this latter problem, ranking questions may be framed as follows:
By carefully evaluating the need of every question used in an instrument and carefully wording
the responses, you will collect information which will yield more satisfactory and meaningful
results.
Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: John
Wiley.
Frary, R.B. (1996) Brief Guide to Questionnaire Development. Washington, DC: ERIC
Clearinghouse on Assessment and Evaluation (30 pages)
Grunlund, N.E. (1993) How to make achievement tests and assessments. Needham Heights, MA:
Allyn and Bacon.
Hinkle, D. E., Oliver, J. D., & Hinkle, C. A. (1985). How large should the sample be? Part
II--the one-sample case. Educational and Psychological Measurement, 45, 271-280.
| ||||||||||||||||||||||
Descriptors: *Educational Research; *Questionnaires; *Research Design; Research Methodology; Responses; Scaling; *Test Construction; Test Format |
Sitemap 1 - Sitemap 2 - Sitemap 3 - Sitemap 4 - Sitemape 5 - Sitemap 6