There was once a time when outcome studies involving only subjective
data were not published in The Journal of Bone and Joint Surgery.
Objective data, such as physical examination findings, were required when
reporting on the efficacy of an operation. This issue of The Journal
features an article on the outcome of surgery as determined on the basis of
retrospective questions only. What has led to this change in attitude?
"Outcome measures" are now in vogue. Questionnaires have now been
"validated." It is possible to fall into the trap that if an
author uses a validated questionnaire, then the result will be valid.
In the article entitled "Clinical Outcome at a Minimum of Five Years
After Anterior Cruciate Ligament Reconstruction," answers to five
questionnaires that were sent to patients more than five years after surgery
were compared with information gathered before and at the time of surgery. The
authors excluded ten failures from the analysis. They concluded that patients
who had gained more weight and had not changed their education level since the
time of surgery had worse outcomes compared with those who had not gained more
weight or who had changed their education level. Are these conclusions valid?
Is this information useful?
Validation of a questionnaire involves testing its test-retest reliability
(reproducibility), responsiveness (ability to detect clinically important
change), and validity. Face validity is the concept that questions are
relevant. Content validity is determined by the consensus of experts.
Construct validity is determined by correlating subjects' answers to the
questions with objective
However, there is no accepted standard of what constitutes validation.
Validation is self-proclaimed, usually after a study has been published in a
The Short Form-36 (SF-36) is a generic measure that includes questions
about general health, activities performed, problems at work, emotional
issues, physical activities, pain, and personal feelings. One of the
thirty-six questions (specifically, a question regarding the level of vigorous
activities) could relate to knee instability. The SF-36 has been validated for
quality of life, but not for knee problems.
The Western Ontario and McMaster Universities (WOMAC) Osteoarthritis Index
was developed to assess patients who have osteoarthritis of the hip and/or
knee. The index consists of twenty-four questions related to pain (five),
stiffness (two), and physical function (seventeen). The response to each
question is scored from 0 to 4. The maximum score is 96. None of the questions
are about instability or sports participation. A patient who has a tear of the
anterior cruciate ligament, causing the knee to give way with pivoting motions
and resulting in positive Lachman and pivot-shift tests, can still receive a
score of 96. The WOMAC scale has been validated for osteoarthritis of the
knee, but not for knee instability.
The Knee Injury and Osteoarthritis Outcome Score (KOOS) includes forty-two
questions related to symptoms (five), stiffness (two), pain (nine), function
of daily living (seventeen), sports and recreation (five), and quality of life
(four). The response to each question can receive a score of 5 points, for a
maximum of 210 points. A patient who has a knee with a torn anterior cruciate
ligament that gives way but that is not painful loses 5 points for symptoms of
instability and up to 16 points for reduction in quality of life. This form is
designed for osteoarthritis and has not been validated.
The International Knee Documentation Committee (IKDC) Subjective Knee Form
was designed to measure symptoms, function, and sports activity in patients
who have a variety of knee conditions. The IKDC questionnaire is generic for
knee problems and is not specific for sports or knee instability. The
questionnaire is part of a complete knee-evaluation system that includes
surgical findings and objective examinations. To validate the form, the
questionnaire was administered to 533 patients, 129 of whom had a torn
anterior cruciate ligament. The answers were compared with responses to
questions related to physical function (but not related to any knee disorder)
on the generic SF-36 quality-of-life questionnaire. A patient who has a torn
anterior cruciate ligament in a knee that is functionally unstable but that is
not painful or swollen can receive 95 of 100 points. The answers on the IKDC
questionnaire also were compared with answers to questions about quality of
life; however, quality of life may or may not be correlated with the status of
The ACL-QOL quality-of-life
used in this study) was developed, pretested, and validated for patients who
have a torn anterior cruciate ligament. This outcome measure is specific for
anterior cruciate ligament injuries. Perhaps the ACL-QOL is the appropriate
questionnaire to use when assessing the functional status of patients with
A study on the use of outcome scores in patients undergoing shoulder
surgery demonstrated that many outcome scores were applied
Scores often were used in a modified form. The authors concluded that the use
of outcome scores for cohorts for which they have not been validated casts
doubt on the validity of the results.
A questionnaire that has been validated for one clinical condition is not
valid when applied to a different clinical entity. Thus, an osteoarthritis
questionnaire should not be used for patients who have knee instability.
If a questionnaire is generic, it can be applied to a wide variety of
conditions but can have little meaning for a specific entity. An unstable knee
will only partly alter the quality of life, and it will do so to a different
extent in different patients. A generic knee questionnaire will give less
useful information about a patient who has a torn anterior cruciate ligament
compared with one that is designed for patients with this disorder. The more
specific a questionnaire, the more sensitive it will be for discriminating
outcomes between patients who have the disorder.
A one-time use of a questionnaire on the status of a condition is not a
"measure of outcome." The answers to the questions are merely
descriptions of subjective symptoms at a single point in time. Comparing
responses to the same questions asked before and after intervention are
subjective measures of outcome.
I believe that subjective information is insufficient for evaluating the
results of an operation. Objective measures, such as physical examination
findings, radiographs, and arthrometer measurements, need to be correlated
with symptoms to give a complete picture. Furthermore, finding a relationship
between independent variables does not mean that one has caused the other. The
existence of a relationship does not necessarily mean that it is significant.
A well-conducted study should use a validated questionnaire that is specific
for the condition being studied. Symptoms are important components of a
clinical study. However, subjective data, no matter how valid, should not