Link to Figures Show Figures
	Article Text Figures References
	Send Link Send HTML Article

RESEARCH FORUM-- Finding Answers to Your Research Question:The Art and Science of Data Collection

Caroline C. Nielsen, PhD
Robert S. Lin, CPO

ABSTRACT

Data collection is a critical component of any research initiative. The clinical practice of orthotics and prosthetics has the potential for many research endeavors requiring the aggregation or acquisition of information. Observation, record keeping, interviews, questionnaires, and prospective and retrospective reviews all exemplify methods of collecting data. It is crucial that this information is systematically coded and recorded for future reference if sound research is to be undertaken.

Introduction

The method of data collection chosen will be closely related to the research design. Concerns about reliability and validity are always important, but more so with an experimental or quasi-experimental design. A less-controlled research design, such as correlational or descriptive, will have less-strict requirements.

Reliability relates to the dependability, reproducibility or precision of a testing method or data collection method. For example, a prosthetist myotesting a transradial amputee for electrode site placement and strength of signal performs the same procedure assessing specific muscle sites multiple times. The closeness of the repeatedly measured outcomes would indicate a reliability of the instrument (myotester) and the assessment technique being used. It is reasonable for the prosthetist to believe he is consistent in the way he performs the myotesting, and there is a high degree of "intra-rater" reliability (the degree to which a practitioner can replicate a variable's measurements and obtain similar outcomes through numerous iterations).

Validity is the degree to which a testing instrument performs the intended task. Many subcategories relate to validity (e.g., content, criterion-related and construct), but for purposes of this text, one needs to understand that reliability and validity are closely interrelated and critical to sound research and data collection. It should also be noted that validity is less of a problem in dealing with physical sciences than with behavioral science when using measuring instruments (1).

An example would be an orthotist who performs multiple range-of-motion (ROM) measurements with a goniometer to assess knee flexion/extension for a given patient but finds that his mean values disagree with the values determined by the Biodex machine. The relationship of the two becomes suspect. If the Biodex is the most accurate standard of measurement, then all other methods (manual or mechanical) are compared with it for accuracy. The degree of agreement between the two techniques constitutes the validity of the technique (manual goniometric testing) being compared to the Biodex. This speaks to the validity of the instrument being tested.

The methods chosen for data collection are closely related to the overall research design, the question(s) asked, the available options and resources, and the plans for the information. Concerns about reliability and validity will also influence the selection of a data. collection technique (2).

Data collection can be highly formalized and controlled, or it can be as simple as watching and listening. Many researchers find it useful to collect information and data using multiple techniques to provide a complete answer to a question. Generally, data collection is comprised of one or more of these strategies:

• watching and listening
• asking questions
• obtaining and examining materials (3).

This article examines the pros and cons and strategies involved in observing and recording patient performance, developing questionnaires and conducting interviews, and reviewing patient records.

Methods

Observing and Recording Patient Performance

To demonstrate a change in a patient's condition after an intervention, accurate measuring and recording of behavior are necessary. Systematic measurement and recording allows researchers to monitor the exact behavior desired to be changed and to establish a baseline level of performance before a treatment procedure is implemented (4). Maintaining an accurate system of measuring and recording behavior is necessary during the intervention to demonstrate any changes. Objective, systematic records can be used to clearly demonstrate the value of a therapeutic intervention.

After clearly defining the behavior to be measured, several steps must be considered when observing and recording a client's behavior, including deciding the setting in which this behavior will be observed, how the behavior will be collected and coded, and how long the behavior will be observed. It is crucial the data be collected and coded systematically.

Several methods are commonly used to collect data on individual patients. Researchers may want to record events. In its simplest form, this is a tally of each occurrence of a well-defined response, for example, a count of the number of steps a patient is able to take unassisted with two different orthotic devices.

Also, researchers might focus on the length or duration of a particular response. A duration record is needed when the length of a response is a concern. Finally, researchers might want to specify the rate of a particular behavior. How frequently does a particular behavior occur within a defined time frame? In all of these protocols to evaluate individual clinical change, the key elements include a clear, measurable definition of the behavior to be evaluated and objective systematic measurement before, during and after the intervention.

The nature of O&P clinical practice lends itself to many research endeavors requiring data collection. Each time a practitioner adopts a new treatment modality, incorporates new componentry or combines a different design to achieve an unprecedented methodology, the results must be documented. This evidence may consist of scientific or diagnostic tests such as computerized gait analysis, radiographs, MRIs, etc., or careful visual observation by a trained eye using consistent baseline criteria. In all cases, the sampling should be large enough to be statistically sound with variables kept to a minimum.

Interviews and Questionnaires

Interviews and questionnaires may be structured like multiple choice exams or they may consist of open-ended questions. Interviews are conducted in person while questionnaires are generally mailed.

The choice of an interview or a questionnaire and a structured or open-ended format depends on the kind of research question to be answered and the amount of knowledge currently available on the topic. If the topic is relatively unknown, open-ended questions will allow for a wider range of responses. If considerable knowledge is available on the topic, a structured format will allow more specific data to be obtained.

Structured responses are easier to score and analyze. making them attractive for a large-scale study. Open-ended questions are particularly useful when you cannot anticipate the range of responses in advance and when you need in-depth information relevant to opinions, ideas or attitudes.

There are several important considerations when deciding to use personal interviews or questionnaires to collect data. Personal interviews are more costly and time consuming, but the researcher is likely to get a high response rate (usually over 90 percent) (5) and is able to reach low-income or culturally diverse clients who may not respond to mail surveys or may not have telephones. The interviewer also can establish rapport with the client, making this an effective technique for gathering personal or sensitive information. In addition, the interviewer can correct misunderstandings, allay concerns and probe inadequate responses from the small number of people in a contiguous geographic area.

Written questionnaires are an effective, low-cost means to gather data from a large sample. Clients may feel a greater sense of anonymity, but the response rate may be low (usually less than 50 percent) (5). Non-respondents may differ in important ways from respondents; so there is no way of knowing to what extent results are valid. With a mailed questionnaire, it is impossible to correct misunderstandings or to reach people who may have difficulty reading or writing.

Whether an interview or questionnaire is chosen, important guidelines for writing questions must be followed. First, needs must be clearly defined and it must be determined whether information can be received from people through these means.

Second, terms must be clearly defined. Some concepts, particularly those related to attitudes and feelings, such as self-esteem or self-empowerment, may be defined in many ways. Defining concepts before constructing questions ensures relevant data.

Constructing questionnaires that will provide the information needed requires a range of item-writing skills. The focus of a question should be clear and specific. Questions related to respondents' direct experience and actions are more likely to get truthful, useful responses. Biased words and phrases can unfairly influence responses.

To avoid confusion and meaningless responses, each question should address only one thought. It is generally a good idea to test the questionnaire with a group similar to the research sample. This "practice run" will allow researchers to identify and address any potential misunderstandings of questions before beginning the research (6).

Regardless of the questionnaire format or modality, the concept of participant confidentiality or anonymity is of paramount importance. While the investigator should document respondents' identities, specific references or inferences should always be avoided. The categorization of survey results, either geographically or by age, gender, diagnostic group, etc., is often a necessary component of the survey reporting process and should not be confused with participant confidentiality. As a precautionary measure, a consent form should be signed by all survey/study participants.

Many questionnaires exist regarding patient satisfaction and experiences during a specific procedure(s). In addition, several questionnaires have been developed recently to assess a patient's/consumer's knowledge to ascertain areas requiring enhanced consumer education. The recently published AOPA patient service evaluation form represents one such format (7).

Documents such as the Role Delineation Study conducted by the American Board for Certification in Orthotics and Prosthetics (ABC) is another prime example of data collection. In this case, questionnaires were mailed to a large sample of certificants relating perceived importance (criticality) and frequency of various treatment methodologies. The collected data were interpreted and used to identify the major treatment domains and the underlying task statements and knowledge requirements. The resulting document has since provided the O&P profession with a detailed blueprint of the O&P professional.

Archive or Record Reviews

Frequently in health care, the only option for collecting data on some subjects is to review existing medical charts or records. When a chart review is conducted or information is extracted from medical records, there are two critical issues: One relates to the reliability or consistency of the patient data, and the other concerns accuracy and the systematic way information is collected from the chart.

This retrospective approach (collecting data after the event has occurred) is useful when the researcher has 'limited time and funds and when it is not feasible to conduct a study with random assignment of patients to control and experimental groups. The purposes of these studies vary and may include a simple description of events, an examination of the relationship between variables or the exploration of possible causative relationships (8).

This form of data collection is relatively inexpensive and quick, and it does not interfere with the clinic situation. However, legal problems may arise with obtaining and using patient records, and documents may be incomplete or disorganized. In this type of study, the researcher has little control over whether complete information has been recorded consistently for all patients.

An example of a retrospective study (also known as ex post facto research) would be a review of scoliosis patient records at a prescribed minimum time after brace discontinuation. The data collected would be reviewed to search for patterns or causation relative to, for instance, successful curve arrest in the Boston versus Charleston Bending Brace. However, because variables are not manipulated, evidence of a strong relationship (success or failure) tells us only a relationship exists. It does not reveal anything about the underlying causes of the relationship. Any inference of causation from a correlation of data must be considered very carefully.

Conclusion

Three categories of data collection commonly used in health care research have been discussed. Frequently, if time and funds allow, it is useful to employ more than one data collection technique to gather information on a research problem. Each technique provides somewhat different kinds of information that can be used to answer research questions.

When selecting an appropriate data collection method, it is crucial to consider the kind of question being asked, what resources are available, how the information will be analyzed and how the results will be used. All the elements in a research design are closely related and interdependent.

Research and education are the lifeblood of every profession. To experience progress, innovation, sound technology transfer and legitimacy, O&P professionals must consider scientific research a required activity. Such endeavors cannot be isolated to university or hospital settings; they must be prevalent in all O&P environments. The appropriate collection of data is a critical first step toward this end.

Suggested Readings

1. Cox RC, West WL. Fundamentals of research for health professionals. Laurel, Md.: Ramsco Publishing, 1983.
2. Currier DP. Elements of research in physical therapy. Balti more: Williams & Wilkins, 1984.
3. DePoy E, Gitlin LN. Introduction to research. St. Louis: Mosby Publishing Co., 1994.
4. Grady KE, Wallston BS. Research in health care settings. Newbury Park, Calif.: Sage Publishing. 1988. Kerlinger FN. Foundations of behavioral research. New York: CBS Publishing, 1986.
5. Ottenbacher KJ. Evaluating clinical change: strategies for occupational and physical therapists. Baltimore: Williams & Wilkins, 1986.
6. Portnoy LG, Watkins, MP. Foundations of clinical research. Norwalk, Conn.: Appleton & Lange, 1993.

CAROLINE C. NIELSEN, PhD, is associate professor and director of research and graduate studies at the School of Allied Health Professions at the University of Connecticut in Storrs, CT 06269-2101.

ROBERT S. LIN, CPO, is director of pediatric clinical services and academic programs at the Newington Children's Hospital in Newington, CT 06111.

References:

1. Currier D, PhD. Elements of research in physical therapy. 3rd ed. Baltimore: Williams & Wilkins, 1984.
2. Oyster CK, Hanten WP. Introduction to research: a guide for the health science professional. Philadelphia: J.B. Lippincott Co., 1987.
3. DePoy E, Gitlin L. Introduction to research. St. Louis: Mosby Publishing Co., 1994.
4. Ottenbacher K. Evaluating clinical change. Baltimore: Williams & Wilkins, 1986.
5. Kerlinger F. Foundation of behavioral research. New York: Holt, Rinehart & Winston, 1986:380.
6. Portney LG, Watkins MP. Foundation of clinical research. Norwalk, Conn.: Appleton & Lange, 1993:259.
7. American Orthotic & Prosthetic Association, patient service evaluation form.
8. DePoy E, Gitlin L. Op. Cit., 81.