Link to Figures Show Figures
	Article Text Figures References
	Send Link Send HTML Article

RESEARCH FORUM--Survey Research and Measurement Error

Terrie Nolinske, PHD, OTR/L, CO

ABSTRACT

Introduction

To conduct survey research is to look at the relationships between sociological and psychological variables by studying samples drawn from a population-at-large using interviews or questionnaires (1). Survey research may be the most commonly used method to collect information about public opinion, political perception or to assist in market research while interviews and questionnaires are used to gather qualitative data on attitudes, values, behaviors, perceptions, degree of participation and satisfaction (1-3).

Although popular, questionnaires may be the most misused method of collecting information. They often are prepared quickly with ambiguous questions crowded on each page to keep the questionnaire short and save on postage. Hurriedly prepared questionnaires likely will yield a low response rate with inaccurate, incomplete information. Effective survey research requires methodical planning, content expertise, resources, time and money. This article will introduce various types of survey research while focusing on the development and use of questionnaires. Ways in which measurement error can be controlled or accounted for when using questionnaires also will be discussed.

Purpose of Survey Research

Survey research is perhaps the best method to describe a population too large to observe directly (2). It may be used to explore, explain, describe or determine patient satisfaction. Surveying every patient who receives these services would be costly, time consuming and inconvenient. Selecting a sample of patients whose medical conditions, prostheses/orthoses, ages and life experiences represent the population is more realistic. Collecting information to measure the attitudes, behaviors and perceptions of the sample allows researchers to draw inferences about the larger population.

Types of Surveys

The research study's purpose is the single most important factor to consider, besides cost, resources and time, when deciding how to collect data. Survey research generally involves studies in which individual people respond to questions asked in a telephone or in-person interview or on a written questionnaire. Each mode of data collection has unique advantages and disadvantages which need to be considered before choosing the one most appropriate for the study since each type of survey might yield different information. Interviews and questionnaires may be used in conjunction with other techniques such as observations, review of written records, controlled observations and panels (1-3). Although these techniques are important, they are outside this article's scope.

Interviews

Interviewers play an important role when collecting survey data. After a thorough orientation and training to the research project, interviewers may help locate and then engage and motivate respondents to answer the questions to the best of their ability.

Interviews may be structured or unstructured. In a structured or formal interview, the same questions are asked in the same order, allowing for more consistent information and a more accurate comparison between responses. Structured interviews are best for the inexperienced researcher or for those studies in which interrater reliability is crucial since data from such interviews are relatively structured, easily organized and analyzed.

In an unstructured or informal interview, the interviewer must know more about the focus of the research because the questions asked may differ from one respondent to another. Interviewers also may probe, asking the respondent to clarify or further explain information. Interviewers can reword or reorder questions to put respondents at ease and allow them to respond at their own pace. This is particularly helpful when posing questions about sensitive issues. Although going off on tangents is an inherent risk in unstructured interviews, a skilled interviewer will redirect the respondent to the research questions.

Data obtained from unstructured interviews are generally more plentiful and ambiguous than those received from more structured interviews. Since unstructured interviews usually produce more data, more time is required to reduce and categorize it.

Because of the personal contact, the interviewer can establish rapport, clarify confusing questions and probe for more or different information, making interviews an ideal method of collecting data on complicated issues. Also, interviews typically yield higher, more complete response rates than mailed questionnaires since respondents are often more reluctant to refuse an interviewer than to throw away a mailed questionnaire (4).

In-Person Interviews

Getting information in-person is the most personal approach and most effective way of gaining trust and cooperation from the respondent. It is easier to read puzzled facial expressions, help the respondent follow complex directions, probe for clarification or redirect responses face-to-face. However, regardless of the interviewer's skill, respondents might be less truthful in their responses during in-person interviews if a topic is sensitive because they wish to appear more socially desirable or acceptable.

In-person interviews can be conducted for longer time periods because it is easier to talk about a topic than to write about it. However, written or self-administered questionnaires also might be used during in-person interviews.

The in-person interview often costs more than other modes, and much effort is required to arrange in-person interviews, including negotiating time and place, traveling and setting up the interview environment to minimize its affect on the process. The affect the environment has on the interview process should not be underestimated.

While interviewer and respondent could meet in many places, the most likely place is the respondent's home, which gives him or her the advantage of comfort and security. Supporting data such as photographs, medical, financial or legal records also are readily available at home should they be necessary to support responses. If a respondent is not at home or fails to keep the scheduled appointment. valuable time can be lost. In addition, the interviewer should wear neutral clothing during an in-person interview since clothing can convey socioeconomic status or reinforce stereotypes.

Interviewers must be trained in the process of interviewing (5). If a national sample is used, interviewers either must be found in the geographic areas under study and trained there, or flown in from a central area, which takes time and money. Even a local sample requiring interviewers to drive to locations and walk door-to-door requires time and financial resources. The sample might be limited or skewed if the interviewers avoid areas of low-density housing or high-crime rates since this could eliminate important information and inferences, depending on the study's topic.

Telephone Interviews

Telephone interviews generally are shorter than in-person interviews. Depending on the nature of the research, a questionnaire might have been mailed to the respondent ahead of the interview. More often, the interviewer reads from a fully written, prepared script and then records verbatim answers from the respondent. Depending on the scope of the research, telephone interviews may be less expensive than in-person interviews since housing, transportation and travel costs are minimized.

Fewer interviewers are needed to do telephone interviews than in-person interviews since phone calls can be made to almost anywhere in the world. Interviewers can clarify questions with one another if they call from the same location, which helps assure greater standardization and reliability. Telephone interviewing also allows interviewers to dress in any way they desire without fear of affecting respondents' answers.

Sampling limitations may occur since telephone interviews are limited to individuals who have telephones and any special equipment needed for hearing impairments. Alternatively, people living in unsafe areas or low-density housing may be more easily called than visited. Those who have phones, however, have become bombarded with direct telemarketing calls, making it difficult for most people to distinguish telemarketing calls from bona fide survey researchers. In general, decreased public trust and increased scams have made the public more skeptical about giving out information over the telephone, in person or through the mail. Therefore, engaging the respondent in the research's purpose and making the material relevant to him or her are important.

Respondents retain varying degrees of anonymity in telephone interviews, depending on how the telephone numbers were initially selected. Since respondents do not have to look a telephone interviewer in the eye, their responses may be more honest especially if the response is socially undesirable or about issues of a sensitive nature (2).

Questionnaires

Questionnaires usually are completed in one of three ways. Questionnaires may be sent through the mail and self-administered by the respondent; they may be delivered by a research team member who asks the respondent to complete it and return it by mail; or questionnaires may be given to a group of respondents gathered together to be completed in the presence of the researcher.

Mailed questionnaires make effective use of manpower since a large number of respondents can be reached with the efforts of few. Also, more people have an address than a telephone (2,5). Self-administered questionnaires received by mail allow respondents more time to think and use other resources than telephone interviews but provide no opportunity to clarify items or answer questions for respondents.

When using self-administered questionnaires the researcher also is limited to those who can read and write and who are physically able to complete the tasks. People who tire easily, have poor vision, or have difficulty reading or writing likely will have difficulty completing a questionnaire. If a friend or family member helps complete the questionnaire, the researcher may receive a collective opinion instead of just the respondent's view. It is especially important to avoid consensus responses if respondents have been selected because of specific knowledge, behaviors, attitudes or skills.

Mailed questionnaires also pose other risks. Data collection using mailed questionnaires often takes longer than most other methods. Up-to-date mailing lists can be costly and may be difficult or impossible to obtain. Once the first-class postage has been paid and the questionnaires mailed, the researcher has no control over or knowledge of if the questionnaires arrived at the intended destination. Questionnaires should be sent by some means where the researcher receives notification of receipt.

Whatever the method of distribution, it is important the questionnaire looks important and not like junk mail. If and when respondents bother to open the envelope, the cover letter must stress the importance of participating in the survey by telling them how the results will influence their lives.

Enclosing a self-addressed stamped envelope will help to increase the response rate. The deadline for returning the questionnaire, usually within two weeks of receipt, should be stated in the cover letter as should a return address in case the self-addressed stamped envelope is lost. The address to which the questionnaire should be returned and the deadline for receipt should be printed at the end of the questionnaire in case everything else is lost.

Unless the responses need to be anonymous, each questionnaire mailed should have a number or code that corresponds to a respondent's name and address. This allows the researcher to track who has and who has not returned questionnaires. Depending on the response rate received by the deadline, reminder postcards and follow-up telephone calls should occur no later than two weeks after the deadline. Reminders are an added cost, but the benefit is well worth it since the response rate is generally increased. This can help ensure sufficient data for a meaningful analysis.

Sources of Measurement Error

Contrary to popular belief, questionnaires are not easy to design and implement. Although there are many issues to consider throughout the process, thinking about them up front will minimize problems later. Some of the more common problems faced by researchers using questionnaires include selecting the population or sample, developing items, format design, question wording, social desirability, recall, context effects, modes of data collection and response rates. The second half of this article will provide an overview of these issues and offer ways in which to minimize the errors or control the biases they produce.

The Sample

Any criteria or method used to select a sample will include some individuals while excluding others. The researcher must define the criteria or characteristics that will describe those included in and those excluded from the sample. A sample can be selected from a relatively complete list of individuals in the population to be studied.

For example, an O&P facility might want to survey clients to determine their level of satisfaction with the services provided. The facility manager conducting the survey might randomly select names from a computer but, due to chance, end up with mostly prosthetic clients. A method to ensure a more equitable distribution of O&P client representation would be to compile separate lists of orthotic and prosthetic clients. If the sampling design calls for selecting a sample from one in every 10 clients on the list, the arrangement would ensure the selection of one of 10 orthotic clients and one of 10 prosthetic clients.

A sample also can be selected from a group of people who go somewhere or do something relative to the study's topic. For example, an orthotist might sample only clients referred from certain physicians or be interested in the stabilization effects offered by a particular thoracic-lumbosacral orthosis and include only those clients in the study sample. A prosthetist might be interested in the function of a specific terminal device and include only those clients in the study. Faculty in an O&P academic program might want to study all students who entered their program as a second career and compare their subsequent professional experience to a group similar in age from the same academic program but for whom prosthetics and/or orthotics was a first career choice.

A third method of sampling is done in two or more stages and can be centered around individuals or geographic area. Suppose a prosthetist wanted to draw a sample of all prosthetic students enrolled in academic programs west of the Mississippi River for recruitment purposes. The prosthetist would have to first obtain a list of all academic prosthetic programs in the prescribed region and then get lists of all prosthetic students enrolled in the program.

Regardless of the method used to select the samples, some degree of error will be present. Sampling error, or the imprecision in a survey, occurs because only part of an entire population is studied. Data are collected from a sample to draw conclusions about the population. Sampling error is calculated and described statistically as the standard error of a mean. Sampling error is generally a function of three things: diversity of what is being measured, sample size and population size (5).

Myths abound about determining sample size. One myth is that a fraction of the population must be specified for the sample (i.e., data must be gathered on 5 percent of the facility's clients). Not true. Another myth is the size of the population from which a sample is drawn has an impact on how well the sample will describe the population. If all else is equal, a sample of 150 will describe a population of 15,000 or 15 million with the same degree of accuracy (1). A third myth is national surveys should include about 1,500 people while state or local surveys should include about 500. This also is not true (1).

Rather, the study's scope should determine sample size. A thorough analysis of the overall research plan is critical to determine sample size. Will separate estimates be necessary for both males and females, upper-limb and lower-limb amputees, or differences between racial groups? For a national sample, will separate regional estimates be necessary? The subgroups are identified, and the researcher estimates how large a sample should be to provide a good representation of each subgroup.

How confident can the researcher be that the sample's characteristics accurately describe the population? The answer depends, in part, on whether the people or things in the population had at least some chance of being included in the sample. Coverage errors result when everyone or everything did not have the same chance. For example, in a telephone survey of orthotic clients, those who have no telephone have zero probability of being sampled. They also may have lower incomes or would have expressed more dissatisfaction with their orthotic services than the population with telephones. If the phenomenon being studied is correlated with the coverage/noncoverage issue, the accuracy of the survey's findings will be lowered (5).

It is important that the researcher makes sure sample error calculation is appropriate to the sample design. It also is important to consider all sources of error in evaluating the precision of sample size. Remember, there is no statistical basis for saying a sample is representative of the sampled population unless the sampling process gave each element selected a known probability of selection (1).

Nonresponse

Determining if the response rate is sufficient to justify the study is a concern when using mailed questionnaires. Low results generally mean that results cannot be generalized to other people or situations unless they are identical to those used in the study. Another concern is the bias from respondents who may have more of an interest in or strong opinion about the study than nonrespondents.

Consider a questionnaire sent to prosthetic patients to determine how satisfied they are with their prostheses. It is conceivable that responses will consist of those patients least satisfied with their prostheses since satisfied patients may not see the need to respond. The bias of dissatisfaction is introduced and skews the responses.

Other biases related to education, socioeconomic status, gender, culture or race also can occur depending on which subgroup within the study did not respond. If nonresponses are a problem, a new sample might need to be selected.

Researchers must define what constitutes a completed questionnaire for data analysis purposes. They must identify which items or what percentage of items must be answered for the questionnaire to be considered useful or complete. People who refuse to respond should be labeled as refusals and differentiated from nonrespondents. Some respondents might complete only part of a questionnaire. The researcher must determine whether the respondents answered the most relevant or important questions. If they didn't, the questionnaire is useless and should be classified as a refusal. A screening mechanism might exist for people who did not refuse to participate but who could not take part because they did not meet prescribed study criteria.

Researchers must define an acceptable response rate at the onset of the study. No standardized formula exists, however, for determining an adequate response rate. A review of the literature reveals diverse opinions on this subject. Bailey (4) reports a 30 percent response rate is reasonable if respondents are interested in the study while Babbie (2) suggests a response rate of at least 50 percent is adequate for analysis and reporting; 60 percent is good; and 70 percent is very good (2,4). Another researcher suggests that if mailed questionnaires are used, the response rate should be 80 to 90 percent (3).

An effort also should be made to determine whether nonrespondents differ from respondents. If the two groups don't differ, study findings can be extrapolated from one group to the other. If differences exist, results are limited strictly to respondents.

One way to determine the characteristics of nonrespondents and also the costs of nonresponse to a study is to interview or send a postcard or short follow-up questionnaire to several nonrespondents. This way researchers can obtain relevant demographics (i.e., age, diagnosis, adaptive aids and equipment used) and reason(s) for their original nonresponse. Be forewarned that events influencing information given by nonrespondents may occur between the time the original questionnaire was received and the follow-up call or postcard.

Comparing information from nonrespondents with information from respondents may not provide definitive descriptions about the two groups since the method used to obtain nonrespondent information may differ from the method used in the study-at-large. The comparison will, however, give some indication of how similar or dissimilar respondents are from nonrespondents. Regardless, it is still good information to have since it might affect the degree to which results can be generalized.

Nonresponse is generally biased toward age, being married and employed. Refusals to mail questionnaires almost always are biased toward education and interest in the study's topic (1). People with low levels of interest in the study who have little education are less likely to respond to and return the questionnaire than people who have more education and a higher interest level in the study. Time, money, study design and topic determine how much of an attempt should be made to increase response rates and obtain information on nonrespondents.

When using a mailed questionnaire, researchers should remind the sample to return the questionnaire. Researchers can send a reminder postcard two weeks after the questionnaires' original mailing; however, respondents who receive postcards may not be able to complete the questionnaire because they've already discarded it. Therefore, an ideal reminder method is to send another complete packet of materials to nonrespondents with a note emphasizing the importance of their contributions to the study. (In this or any other reminder, it is appropriate to advise respondents to ignore the reminder if they already have responded.)

In a study employing a self-administered questionnaire, this procedure was followed three times over a two-month period. The original mailing yielded a response rate of 48 percent while the second mailing resulted in a 78 percent response. The third mailing, coupled with phone calls to all nonrespondents, resulted in a 95 percent response rate (6).

Researchers must be assertive yet unoffensive when engaging respondents. To increase response rates, researchers might entice respondents with money, food or a promise to send a summary of the results (1). Mailing the questionnaire in an eye-catching, professional-looking envelope distinguishes it from junk mail. Most importantly, people are likely to respond, regardless of the questionnaire' s length, if clear, concise questions are presented in an uncluttered format and accompanied by a cover letter that makes the study's topic relevant to them.

Reliability and Validity

A reliable questionnaire suggests the same data will be collected in repeated observations of the same phenomena. Using a reliable questionnaire, the researcher can assume that differences in responses. are probably due to differences in respondents and not to differences in the questionnaire. Question wording is very important in determining questionnaire reliability. A questionnaire must contain items that prepare the respondent to provide accurate, consistent and complete responses. Questions must mean the same thing to each respondent. They should have a sense of what an acceptable response is based on the question or researchers' clues (1).

Reliability differs from the concept of validity. A questionnaire generally is valid if the measurement procedures accurately reflect the concept under study. Also, many specific types of validity exist. Are the responses to questionnaire items a valid reflection of the intended concept? To improve validity, questions should be worded to increase the likelihood that they'll mean the same thing to each respondent. An open-ended questionnaire item asking the respondent for his or her level of amputation might yield multiple answers with varying degrees of sophistication. These data may be too fragmented to be used by the researcher meaningfully. To obtain accurate, standardized responses and meaningful data, the question could be a multiple-choice one from which respondents choose the amputation level most similar to their own. Not only should multiple questions be used, but questions measuring the same concept should appear in different formats. Triangulation, where more than two methods of data collection are used to get at the same concept, can provide more detailed, credible and richer results (1, 2).

High degrees of reliability and validity are important in survey research. Both concepts are complex. Each variable under scrutiny requires care to identify the best set of questions to measure it. Pilot studies in which a draft of the questionnaire is sent to a small representative sample, perhaps five or 10 individuals from the target population, can help to determine how reliable and valid the items are If at all possible, the researcher should interview these respondents to determine where questions were unclear or misleading.

If it is not possible to talk directly with respondents in the pilot study, it is a good idea to include a feedback page that asks specific questions about the questionnaire-draft: Was the questionnaire easy to follow? Which items, if any, were confusing? Why? What sections or items were most difficult? Why? Feedback to these questions can greatly improve the questionnaire's design, item format and flow. It is not unusual to conduct multiple pilot questionnaires or studies prior to the study's final implementation. It also is useful to monitor the time it takes for respondents in the pilot test to complete the questionnaire.

Based on the results of pilot testing, the questionnaire must be revised and retested until each instrument item attains an acceptable level of validity and reliability. More often than not, the initial questionnaire will be much longer than the final version. Only items that make a direct and meaningful contribution to the study's purpose should be retained in the final questionnaire.

Social Desirability

Respondents may feel threatened by questionnaire items that are controversial or of a sensitive nature such as prejudice, money or illness (1,5). Also, some questions or topics threaten respondents who have low self-esteem or who fear a lack of confidentiality. In general, issues surrounded by social undesirability are underreported significantly more than other issues (1-3,5). Some respondents will give accurate responses about such issues during an in-person interview while others feel more comfortable reporting such information in a self-administered questionnaire. Others would not respond to such issues regardless of the method used.

How far should researchers go in pursuing information they believe they need? One question they might ask themselves is: "Does the end justify the means?" Causing respondents some emotional discomfort while answering questions might be worthwhile if the data will help resolve the problem.

The respondent's knowledge base will affect whether s/he can or will respond to an item. Consider a study to find out how much clients know about their disease or medical condition. Questionnaire items will have to be worded and ordered very carefully so respondents do not feel foolish or intimated by the questions.

Confidentiality and Anonymity

Many survey researchers promise respondents that all data will be kept confidential, and no one except the researchers will be able to link respondents with their answers. This should be stated in the cover letter, on the questionnaire or in signed agreements with the researcher. While any question has the potential to embarrass someone in the sample, respondents are more likely to respond to items if they trust the researcher and feel confident their answers will not be attributed to or adversely affect them. The promise to keep identifying information confidential still does not take into account how the data will be used (1). Even with the promise of confidentiality respondents may still feel the results could be used against them is a group. Or they may respond, but in a socially desirable manner, not wanting to appear undesirable when results are reported.

Confidentiality differs from anonymity. Confidentiality is when information about respondents is known by the researchers but to nobody else (single-blinded study). Anonymity is when respondent information is not known even to the researchers (double-blinded study). The guarantee of anonymity may increase the response rate but sacrifice other information. The researcher must examine the study's scope and determine the study design that best suits his or her needs.

Recall

Regardless of good intentions, respondents make errors when responding to questionnaire items because they cannot recall appropriate information. Respondents displace events in time, associate the behavior with the wrong time period or forget entirely. They distort the facts or they simply may not understand the question. Telescoping effects might bring the distant past nearer in memory (1, 5). Consider the multiple amputee who is asked the date or year of his or her most recent amputation. Due to the telescoping effect s/he will likely report a more recent date than the actual date of the surgery.

Respondents have a more accurate memory for recent events, especially if the events had an important impact on them. The respondents' motivational level also can affect how much effort they make to give an accurate report (7). Response accuracy may decline when too many questions are asked or if respondents are not given adequate time to recall and generate responses. Longer questions may give respondents the necessary time to think, sometimes yielding more accurate responses (8).

The researcher might consider several options to reduce errors resulting from recall effects. First, reduce the length of time respondents are asked to report about by decreasing the time between the event in question and the survey. The longer the reporting period, the less accurate the information will be (1).

Consider the prosthetist who is trying to determine how often a transradial (below-elbow) amputee uses the myoelectric prosthesis as opposed to the standard prosthesis throughout the year. Asking how often each prosthesis is used during the year is less reliable than asking the client how often each prosthesis is used during a month. The two numbers given in response to the latter question would be multiplied by 12 to get an estimated use for each prosthesis over the year.

Memory often is improved by asking more questions, especially if the questions relate or are designed to stimulate associations. Another way to improve recall is to ask questions that approach the topic from many different directions. Since there are many questions, respondents will need to spend a longer time working their way through them. Therefore, they have more time to think about the issues or events. In a questionnaire related to disease management, questions could be asked about visits made to health-care practitioners, symptoms, medications, and functional ability and limitations-numerous ways to approach a single topic. The greater the relevance perceived by respondents between the questionnaire item and themselves, the more likely they will recall their response with greater accuracy (1,7).

Memory also can be stimulated by asking for similar information in different formats on the questionnaire. An open-ended question might ask, "What problems do you have when putting on your hip-knee-ankle-foot orthosis?" It also might be included as a statement in a matrix format where the respondent is asked to read the statement "I have no problems when putting on my hip-knee-anklefoot orthosis." The respondent is then asked to make a mark under the appropriate response alternative-strongly agree, somewhat agree, somewhat disagree or disagree. Use of this technique can give respondents a second chance to think about their answers and to verify information reported elsewhere on the questionnaire.

Question Wording

Researchers writing items for questionnaires must consider the following: Is this a question that will mean the same thing to everyone? Is this a question people can answer? Does the question demand knowledge and information the respondent might not have? Is this a question that people will be willing to answer, given the methods used in data collection? (1,9) Does the question elicit information necessary to test research hypotheses or questions? Is the wording clear? Does the item lead respondents to a conclusion or allow them to draw their own? Does the question contain potentially threatening issues? Are questions loaded with social (un)desirability?

Adequate words. When writing items, make sure there are enough words to clearly express a thought. The response given will only be as accurate as the item was written. It is more accurate to ask "What was your age on your last birthday"? than to ask just "Age?". This will generate consistent responses and avoid fractional answers. Care must be taken not to include so many words as to confuse or lead the respondent (e.g., "What did you like best about the prosthetic/orthotic facility? We're especially interested in anything like courtesy shown by employees, facility cleanliness, available parking, etc.").

Well-defined words. The same words mean different things to different people so take care to clarify words or concepts that might be misconstrued. Consider the question: "How many times in the past month have you seen your doctor?" What does "doctor" mean? Does this category include discussions held with an osteopathic physician, a psychiatrist, podiatrist or chiropractor (1)? Do doctors' visits when seen by a nurse count? Provide respondents with definitions for terms at the beginning of the question or at the beginning of the questionnaire to avoid confusion and frustration. Some room for error still exists since many people don't know the titles of the professionals encountered at their doctor's office.

Negation. Using negation in a questionnaire can be misinterpreted easily. Often the negative word is missed when reading the question. Using "yes/no" answers can be confusing. Consider how a respondent might respond to the statement: "The following kinds of people should not be allowed to wear a SACH foot," followed by response alternatives of people with various conditions who may or may not find the SACH foot useful. When replying to this statement, respondents likely will respond by indicating a willingness for a person to use the SACH foot rather than to respond by indicating those who should be prohibited from using the foot. Either the stem of the statement should be reworded or the format of the item changed to be less ambiguous and confusing.

Biased items. Avoid biased items and terms. The meaning of a response depends largely on the question's wording. Take care so as not to bias respondents or give them leading questions. Simply identifying an attitude or remark with a prestigious practitioner or well-known facility might cause a biased response.

Question Format

Double-barreled questions are ambiguous and produce misleading and inaccurate answers (9). Responses to "The orthotic facility needs to add more parking spaces and help clients submit their insurance claims" will be accurate only if the facility is indeed currently deficient in both areas. The respondent may agree with part of the statement and answer in the negative, thinking a positive response requires agreement with both parts of the statement or vice versa. The statement should be reworded in one of several ways. It could be divided into two questions. Or a section on the questionnaire could be devoted to service improvements. The facility manager could list services being provided, and respondents could simply check off the functions they feel should be improved.

Open-ended questions allow respondents to create their own answers. Do not underestimate the feeling of empowerment this can give a respondent (1).

Open-ended questions should not give respondents much information or many clues about what the response should be. Consequently, respondents offer fewer opinions, and sometimes the researcher obtains qualitative data. While interesting, this can pose problems since relatively obscure answers are usually not easily analyzed. Data from open-ended questions are much more time consuming to read, reduce and analyze. Responses to open-ended questions generally become subject to the researcher's interpretation as s/he attempts to cluster them during the analysis.

Open-ended questions can be effective on topics of a sensitive nature or when used as follow-up "why" questions to capture respondent insight. To avoid constrained responses to open-ended questions, it is important to use clear, well-defined words and phrases.

Open-ended questions may be used in a pilot test to generate response sets or closed-ended questions for the final questionnaire. Consider an orthotist who, among other things, wants to determine the main reason clients prefer polymer orthoses over metal orthoses. A pilot test containing the item "Please give three reasons why you prefer plastic orthoses over metal orthoses" would be sent to a few identified clients. Imagine that the inquiring orthotist received 13 reasons from the pilot. The orthotist would then compile a list using those 13 responses as response alternatives on the final survey. The orthotist would then analyze the data to see which of the 13 alternatives were selected with greatest frequency.

Closed-ended questions present the respondent with a recognition, not a recall, task. Closed-ended questions offer a variety of responses the respondent might not have thought of which may be important to consider. The response alternatives should be exhaustive and mutually exclusive (2,9). The list of response alternatives should provide each respondent with the same perception of what constitutes an adequate response. This is one of the main advantages of using closed-ended questions since all respondents choose from the same categories. This increases the likelihood of enough responses in any one category to be able to use in data analysis.

Multiple-choice questions must have mutually exclusive response alternatives. It is generally a good idea to have no more than six response alternatives. Unless the design of the study is to match subjects by age or years wearing an orthosis or prosthesis, knowing the precise ages and years of experience of every respondent is unnecessary. This information also becomes messy to code and cluster during analysis.

It is often less threatening for the respondent and easier for the researcher to code and analyze if response alternatives include ranges of numbers. An example of a multiple-choice question for adults requesting age might look like this:

1. My age on my last birthday was:

1. 20-30 years
2. 31-40 years
3. 41-50 years
4. 51-60 years
5. 61-70 years
6. 71+ years

Matrix questions are used when listing several statements that have the same set of answers. Statements are listed, and respondents are asked, for example, whether they strongly agree, slightly agree, slightly disagree or strongly disagree with each statement. Is it undesirable to have a midpoint on an odd-numbered scale of response alternatives? No, although the odd-numbered scale should be used judiciously. If a numbered scale included "5-4-3-21" respondents could feel pulled toward both endpoints and likely select the midpoint of three (1, 9-10). The researcher also must work to define the midpoint category. Will it be a continuation of the other categories or will it be neutral? It might be important to include an undecided or not applicable category but, again, only after careful consideration since some respondents lazily select the undecided or not applicable categories. Many researchers prefer to force the respondent to make a choice without offering the latter two response alternatives.

To effectively use the matrix format so answers will be meaningful, every statement must be worded so it is located at one end of a continuum (1). Consider this situation: Respondents are asked to indicate the degree to which they agree with the statement, "Orthotic services provided by this facility are average." The word "average" is a descriptor that falls midway on a continuum between good and poor. If respondents agree with that statement, there is no problem. If respondents disagree with that statement, it means they rate the facility as either good or poor, which will not give the facility manager meaningful information. The statement could be rewritten as "Orthotic services provided by this facility fully address my needs" or "Orthotic services provided by this facility are excellent."

The matrix format makes for an efficient use of space so respondents tend to work through items in this format rather quickly. This can be a disadvantage to the researcher, however, since respondents can develop a pattern of agreeing with all statements without having stopped to think about any of them. This is a phenomenon called response-set and can be minimized by alternating statements representing different orientations (2).

Attempts have been made to characterize quantifying adverbs used in rating scales and response alternatives (11). Usually treated as categories with equal intervals between them, there are often real differences in values assigned to the responses. Using descriptors accompanied by a percentile range or simply using the percentile range alone are two ways to minimize confusion and misinterpretation of response alternatives (1). For example, if the response alternatives were "Always," "Often," "Sometimes" and "Rarely," they could be correlated to percentile ranges. "Always" would mean something was done or occurred 76-100 percent of the time; "often" would mean 51-75 percent; "sometimes" would mean 26-50 per- cent; and "rarely" would mean 0-25 percent. Or the percentages could be used without the adverb descriptors. Using this format helps respondents develop a more concrete image of the response parameters.

Don't know and contingency questions are a final option. The don't know option must be used judiciously since it can become an easy out for respondents who are unwilling to think about and give an answer. But many times questionnaire items do ask respondents about something they truly do not know. There are two ways to approach this problem. The questions may be asked, relying on respondents to volunteer that they do not know. To counter the invalidity of responses that can result, a contingency question can be used. This method might also be referred to as using screening or filter questions (1,2).

A contingency question asks respondents whether they are familiar with the topic. If they are, they should be directed to complete that section of the questionnaire. Whether they are to respond to the next item is contingent upon their response to the first question in the series. If they are unfamiliar with the topic, the researcher can explain the issue briefly and then ask for the respondents' opinion. Or, respondents can be excused from answering that section of the questionnaire by using a skip pattern that directs them to the next question. When skip patterns are used, clear instructions must be given to respondents to let them know exactly which questions they are to answer.

For example, consider facilities in which practitioners are board certified in both orthotics and prosthetics. The purpose of the questionnaire is to determine whether orthotic services are used more than prosthetic services. The initial question to clients might be, "Do you receive orthotic services when you come to this facility?" The options of "yes" and "no" are presented. If clients select "yes," they go on to the next numbered item, which most likely will ask them to check all orthotic services they have received. If clients select "no," a skip pattern printed immediately after the "no" directs them to the next numbered question they are to answer (i.e., No [skip to Q23]). Parallel items would be constructed about prosthetic services. Skip patterns should be very clear, making arrows and diagrams unnecessary. The cleanest skip patterns simply direct the respondent to "skip to Q23" and should be included in parentheses after the appropriate response alternative.

Context Effects and Question Order

Once a set of questions has been drafted, it needs to be put into a form or context. The context and ease of administration depend greatly on the format and item order. The questionnaire might begin with the statement of purpose. The first section should contain "easy" or "non-threatening" items. Difficult or sensitive items should be addressed in later sections of the questionnaire.

The goal when ordering questionnaire items is to make the tasks of the researcher and respondent as easy as possible. If writing a questionnaire for an interview, make certain everything the interviewer says is recorded in writing. Differentiate between words the interviewer is to read to respondents and words that serve only as instructions for the interviewer. Researchers often use uppercase letters to indicate instructions and lowercase letters for questions to be read aloud (1,5). Optional wording selections such as his/her or husband/wife are easy to recognize and use if set off by parentheses.

For self-administered questionnaires the goal is also to make items in the questionnaire flow and easy to use. The questionnaire must be self explanatory since no one will be present to clarify information. Separate or detailed instructions on how to complete each section of the questionnaire are usually not necessary since they take up room and are generally not read. Skip patterns, used sparingly, should employ simple instructions.

The layout of the questionnaire is very important. The questionnaire must be uncluttered and pleasing to the eye. Attempts to fit more copy onto a page by cramming it together or by reducing the font size should be avoided. Type should be no smaller than a 10 point size for easy reading. Studies have shown that strategies for cluttering pages actually reduce the response rate as compared with the same number of questions spread out attractively over more pages (1).

Questionnaire format might be influenced by available money, support staff and material resources. Most researchers remain adamant that each question and each response alternative appear on a separate line with the response box or code to the right.

The questionnaire should have a flow to it, with logical and smooth progression from one section to another. The researcher should consider whether a better answer will result if the questions are asked in a prescribed order. If similar questions are asked, the researcher should consider a short note to acknowledge that fact saying "Some questions may appear to you to be similar or the same. Please try to answer all to the best of your ability regardless of whether or not you believe you have already answered it." This will help minimize redundancy effects, which occur when a question is asked, and the response appears to be the answer to a different question. Later in the questionnaire that "different question" is asked, and the respondent either leaves it blank or repeats the answer given earlier.

In what order are general and specific questions asked? Some have found that if specific questions are asked first, the attention given to them will probably influence responses to general questions (12). More thoughtful answers to specific questions are given when respondents have had a chance to consider the material in a broader, more general context. Also, specific questions can give background on issues and help respondents answer subsequent general questions about complex issues (12,13).

Consider a section in a questionnaire that assesses clients' overall satisfaction with their quality of life since receiving their polymer knee-ankle-foot orthosis and with the design of this orthosis in particular. The researcher might begin this section of the questionnaire with items asking about the frequency and duration of use, and about the comfort and aesthetics of the orthosis. These items about specific aspects of the orthosis would be followed by a general question asking respondents to indicate their overall level of satisfaction with the orthosis by asking whether their quality of life has improved since receipt of the orthosis. The researcher, through item wording or control of variables during data analysis, must ensure that the client's quality of life has indeed been affected by the orthosis and not other factors such as medication or employment of a housekeeper or home health aide.

Response alternatives influence the respondents' interpretation of both questions and answers. Response alternatives also make assumptions about the range of opinions or behaviors held by the general population or the researchers. Items presented early in a list of choices are more likely to be selected than those presented later in the list (10, 12). Responses also may be influenced by whether the researcher is present (1).

The importance of pretesting or piloting a self-administered questionnaire cannot be overemphasized. When the items have been developed and ordered, the researcher should ask several people, with characteristics similar to potential respondents, to complete the questionnaire and provide feedback. In this way the researcher has an invaluable opportunity to make revisions that will strengthen the questionnaire and allow for the collection of reliable and valid data.

Ethical Considerations in Survey Research

Since survey research involves human subjects, ethical considerations are very important. Most organizations have a research committee or board of human investigations that must review and approve research proposals. The same committees or boards generally also approve research proposals submitted by someone outside of the organization wishing to conduct the study within that setting. Once the research proposal has been approved, the study may be implemented. However, respondents have a right to know about several things before answering the questions.

First, respondents have a right to know the name of the organization conducting the research, the name of the organization funding the research, the name of the interviewer, if applicable; the study's purpose; and the researcher's policy on confidentiality. Assuring respondents that they can skip any questions they do not want to answer sometimes helps relieve some of the pressure felt by the respondents. Signed consent forms generally are not necessary in survey studies; however, this may vary by research content, intent and organizational policy (1,2,14).

Second, respondents must be told how the information they are about to provide will be used and protected (1,2,14). While there are always exceptions, the following procedures can help assure that respondents are protected when participating in survey research: All research team members should sign a letter of intent to maintain confidentiality. Keep information such as respondents' names and addresses separate so one cannot be linked to the other (14). Make sure identifying characteristics of respondents are not on the questionnaire. Use of study numbers, assigned early on, is a good way to forget about names entirely. Individuals such as bosses who might be able to identify handwriting or people with their responses should not be allowed to look at the actual completed questionnaire (1). When analyzing data, care should be taken to report data in aggregate, avoiding small categories in which individual respondents could be singled out and recognized. Identifying information should either be destroyed at the end of the study or continue to be stored in confidential, locked files (1,14).

Just as researchers have a responsibility to the respondent, they also have a responsibility to the interviewer. Since interviewers are on the front line, they may be asked to visit sample addresses to make sure they feel safe. Interviewers may make evening calls or provide in-person interviewers with an escort as options to daytime calls and feeling unsafe in certain areas (1,5). Interviewers should be given adequate training on interpersonal skills as well as specific information related to the research study.

Survey research should be enjoyable for the respondent, the interviewer, if present, as well as the researcher. Most of the time the respondent feels satisfaction knowing s/he has contributed to a worthwhile project. If the researcher used motivators such as food, money, or small prizes or gifts, care must be taken to provide them in such a way so as not to breech any promises of confidentiality that were made (1).

Summary

Survey research can be conducted using a variety of methods, including in-person interviews, telephone interviews and self-administered questionnaires. Each method has its set of strengths and limitations, which should be taken into account along with the purpose of the study. Practitioners in the O&P profession can use survey research in many ways. Questionnaires can be sent to clients to determine customer satisfaction; sent to other practitioners to determine how they are positioning themselves in issues related to health-care reform; and sent to faculty and students in academic programs to find out whether what is being taught is compatible with what is being practiced.

The information received from any survey instrument is only as good as the words used to create it. Careful attention must be paid to every detail throughout the process of survey research. Every action, every question must somehow reflect back to the research questions or research hypotheses. Since the researcher puts it all together, perhaps the ultimate success of survey research depends on the dignity and ethics of that individual, as s/he interacts with the respondents and the data, ensuring all are treated with honesty and integrity.

TERRIE NOLINSKE, PHD, OTR/L, CO, is president of TNI: Consultants in Professional Development, 930 North Boulevard, Oak Park, IL 60301; and an associate professor at Rush University Department of Occupational Therapy, 1743 W Harrison St., Chicago, IL 60612; (312) 942-6988.

References:

1. Fowler FJ. Survey research methods. Newbury Park, Calif.: SAGE Publications Inc., 1988.
2. Babbie B. The practice of social research. Belmont, Calif.: Wadsworth Publishing Co. Inc., 1979.
3. Kerlinger EN. Foundation of behavioral research. New York: Holt, Rinehart and Winston Inc., 1973.
4. Bailey DM. Research for the health professional: A practical guide. Philadelphia: FA Davis Co., 1991.
5. Lavrakas PJ. Telephone survey methods: Sampling, selection and supervision. Newbury Park, Calif.: SAGE Publications Inc., 1987.
6. Nolinske TL. The value of the mentor-protege relationship as perceived by the mentor, the protege and the protege's boss. Dissertation Abstracts International 1994; 55:5508B (University Microfilms No. 94-33, 900).
7. Cannell CE Reporting of hospitalization in the health interview survey, U.S. Department of Health, Education and Welfare. Vital and Health Statistics 1965; 2:6.
8. Bradburn NM, Rips L, Shevell S. Answering autobiographical questions: The impact of memory and inferences on surveys. Science 1987;236:157-61.
9. Payne SL. The art of asking questions. Princeton, N.J.: Princeton University Press, 1951.
10. Schwarz N, Hippler Hans-J, Deutsch B, Strack F Response scales: effects of category range on reported behavior and comparative judgments. In: Singer B, Presser S [eds]. Survey research methods. Chicago: University of Chicago Press, 1989:165-72.
11. Bradburn NM, Miles C. Vague quantifiers. In: Singer E, Presser S [eds]. Survey research methods. Chicago: University of Chicago Press, 1989:155-64.
12. Strack F, Martin L. Thinking, judging and communicating: A process account of context effects in attitude surveys. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S [eds]. Measurement errors in surveys. New York: John Wiley & Sons Inc., 1991.
13. Sheatsley P. Questionnaire construction and item writing. In: Rossi PH, Wright JD, Anderson AB [eds]. Handbook of survey research. San Diego: Academic Press Inc., Harcourt Brace Jovanovich, Publishers, 1983.
14. Singer E. Informed consent: Consequences of response rate and response quality in social surveys. American Sociologi- cal Review 1878;43:144-61.

Website built by oandp.com