0
Editorials   |    
Are Validated Questionnaires Valid?
Bertram Zarins, MD
View Disclosures and Other Information
The Journal of Bone and Joint Surgery, Incorporated
J Bone Joint Surg Am, 2005 Aug 01;87(8):1671-1672. doi: 10.2106/JBJS.E.00554
5 Recommendations (Recommend) | 3 Comments | Saved by 3 Users Save Case
There was once a time when outcome studies involving only subjective data were not published in The Journal of Bone and Joint Surgery. Objective data, such as physical examination findings, were required when reporting on the efficacy of an operation. This issue of The Journal features an article on the outcome of surgery as determined on the basis of retrospective questions only. What has led to this change in attitude? "Outcome measures" are now in vogue. Questionnaires have now been "validated." It is possible to fall into the trap that if an author uses a validated questionnaire, then the result will be valid.
In the article entitled "Clinical Outcome at a Minimum of Five Years After Anterior Cruciate Ligament Reconstruction," answers to five questionnaires that were sent to patients more than five years after surgery were compared with information gathered before and at the time of surgery. The authors excluded ten failures from the analysis. They concluded that patients who had gained more weight and had not changed their education level since the time of surgery had worse outcomes compared with those who had not gained more weight or who had changed their education level. Are these conclusions valid? Is this information useful?
Validation of a questionnaire involves testing its test-retest reliability (reproducibility), responsiveness (ability to detect clinically important change), and validity. Face validity is the concept that questions are relevant. Content validity is determined by the consensus of experts. Construct validity is determined by correlating subjects' answers to the questions with objective measurements.1 However, there is no accepted standard of what constitutes validation. Validation is self-proclaimed, usually after a study has been published in a medical journal.
The Short Form-36 (SF-36) is a generic measure that includes questions about general health, activities performed, problems at work, emotional issues, physical activities, pain, and personal feelings. One of the thirty-six questions (specifically, a question regarding the level of vigorous activities) could relate to knee instability. The SF-36 has been validated for quality of life, but not for knee problems.
The Western Ontario and McMaster Universities (WOMAC) Osteoarthritis Index was developed to assess patients who have osteoarthritis of the hip and/or knee. The index consists of twenty-four questions related to pain (five), stiffness (two), and physical function (seventeen). The response to each question is scored from 0 to 4. The maximum score is 96. None of the questions are about instability or sports participation. A patient who has a tear of the anterior cruciate ligament, causing the knee to give way with pivoting motions and resulting in positive Lachman and pivot-shift tests, can still receive a score of 96. The WOMAC scale has been validated for osteoarthritis of the knee, but not for knee instability.
The Knee Injury and Osteoarthritis Outcome Score (KOOS) includes forty-two questions related to symptoms (five), stiffness (two), pain (nine), function of daily living (seventeen), sports and recreation (five), and quality of life (four). The response to each question can receive a score of 5 points, for a maximum of 210 points. A patient who has a knee with a torn anterior cruciate ligament that gives way but that is not painful loses 5 points for symptoms of instability and up to 16 points for reduction in quality of life. This form is designed for osteoarthritis and has not been validated.
The International Knee Documentation Committee (IKDC) Subjective Knee Form was designed to measure symptoms, function, and sports activity in patients who have a variety of knee conditions. The IKDC questionnaire is generic for knee problems and is not specific for sports or knee instability. The questionnaire is part of a complete knee-evaluation system that includes surgical findings and objective examinations. To validate the form, the questionnaire was administered to 533 patients, 129 of whom had a torn anterior cruciate ligament. The answers were compared with responses to questions related to physical function (but not related to any knee disorder) on the generic SF-36 quality-of-life questionnaire. A patient who has a torn anterior cruciate ligament in a knee that is functionally unstable but that is not painful or swollen can receive 95 of 100 points. The answers on the IKDC questionnaire also were compared with answers to questions about quality of life; however, quality of life may or may not be correlated with the status of the knee.
The ACL-QOL quality-of-life questionnaire1 (not used in this study) was developed, pretested, and validated for patients who have a torn anterior cruciate ligament. This outcome measure is specific for anterior cruciate ligament injuries. Perhaps the ACL-QOL is the appropriate questionnaire to use when assessing the functional status of patients with this injury.
A study on the use of outcome scores in patients undergoing shoulder surgery demonstrated that many outcome scores were applied inappropriately2. Scores often were used in a modified form. The authors concluded that the use of outcome scores for cohorts for which they have not been validated casts doubt on the validity of the results.
A questionnaire that has been validated for one clinical condition is not valid when applied to a different clinical entity. Thus, an osteoarthritis questionnaire should not be used for patients who have knee instability.
If a questionnaire is generic, it can be applied to a wide variety of conditions but can have little meaning for a specific entity. An unstable knee will only partly alter the quality of life, and it will do so to a different extent in different patients. A generic knee questionnaire will give less useful information about a patient who has a torn anterior cruciate ligament compared with one that is designed for patients with this disorder. The more specific a questionnaire, the more sensitive it will be for discriminating outcomes between patients who have the disorder.
A one-time use of a questionnaire on the status of a condition is not a "measure of outcome." The answers to the questions are merely descriptions of subjective symptoms at a single point in time. Comparing responses to the same questions asked before and after intervention are subjective measures of outcome.
I believe that subjective information is insufficient for evaluating the results of an operation. Objective measures, such as physical examination findings, radiographs, and arthrometer measurements, need to be correlated with symptoms to give a complete picture. Furthermore, finding a relationship between independent variables does not mean that one has caused the other. The existence of a relationship does not necessarily mean that it is significant. A well-conducted study should use a validated questionnaire that is specific for the condition being studied. Symptoms are important components of a clinical study. However, subjective data, no matter how valid, should not stand alone.
Mohtadi N. Development and validation of the quality of life outcome measure (questionnaire) for chronic anterior cruciate ligament deficiency. Am J Sports Med.1998;26: 350-9.26350  1998  [PubMed]
 
Harvie P, Pollard TC, Chennagiri RJ, Carr AJ. The use of outcome scores in surgery of the shoulder. J Bone Joint Surg Br.2005;87: 151-4.87151  2005  [CrossRef]
 

Submit a comment

References

Mohtadi N. Development and validation of the quality of life outcome measure (questionnaire) for chronic anterior cruciate ligament deficiency. Am J Sports Med.1998;26: 350-9.26350  1998  [PubMed]
 
Harvie P, Pollard TC, Chennagiri RJ, Carr AJ. The use of outcome scores in surgery of the shoulder. J Bone Joint Surg Br.2005;87: 151-4.87151  2005  [CrossRef]
 
Accreditation Statement
These activities have been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education (ACCME) through the joint sponsorship of the American Academy of Orthopaedic Surgeons and The Journal of Bone and Joint Surgery, Inc. The American Academy of Orthopaedic Surgeons is accredited by the ACCME to provide continuing medical education for physicians.
CME Activities Associated with This Article
Submit a Comment
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discretion of JBJS editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe





John E. Kuhn
Posted on November 08, 2005
Perspective Bias
Vanderbilt University Medical Center, Nashville, TN 37212

To The Editor:

I read with interest the editorial by Zarins titled ““Are validated questionnaires valid?” Dr. Zarins makes clear his preference for “objective” measures of outcome, and shows his disdain for “subjective questionnaires”. Unfortunately, Dr. Zarins suffers from a bias in perspective. A patient presents with symptoms of knee instability, which prevent that individual from playing in his or her sport. The surgeon identifies a tear of the ACL as the source of the patient’s concerns. The patient seeks treatment to be able to play on a stable, pain-free knee. The surgeon will operate to restore ACL integrity. These are very different perspectives of the same problem. The outcome of treatment can be approached from either perspective. Which outcome is more important? It depends on the question you are asking.

If you are interested in the clinical outcome of a treatment, then clearly the patient’s perspective is most important. The KT-1000 can not tell us that the patient is playing without pain and without episodes of giving way. Only validated subjective questionnaires can answer this question. If you are interested in a new ACL graft option, or comparing your surgical method to a normal knee then the KT-1000 can tell us if there is a difference in anterior laxity in 30 degrees of knee flexion. Just as the KT-1000 cannot tell us is a patient returns to sport, the validated questionnaire cannot tell us if the surgery was successful in correcting laxity with the knee at 30 degrees of flexion. Every measurment tool has potential for bias and error, and both subjective and objective measures can be valid.

The real issue is what is the question driving the research effort, and then did the authors use the best tool to measure the outcome in question? As surgeons we must remove out blinders, stop thinking about patients as their pathology, and welcome patient-oriented measures of outcome.

Bertram Zarins, M.D.
Posted on October 11, 2005
Dr.Zarins responds to Drs. Anderson and Irrgang
MGH/Harvard Medical School

Drs.Anderson and Irrgang make excellent points in their critique of the editorial. Validated questionnaires are useful if applied correctly, and are better than traditional methods of gathering subjective information from patients. The point of the editorial, though, was that the pendulum has swung too far. Validated questionnaires are being used independently as sole measures of outcomes and are sometimes applied inappropriately.

Using health related quality of life as the primary outcome measure of the success of a knee operation could lead to erroneous conclusions. Many factors influence the quality of life, most of which have nothing to do with the knee. To evaluate the result of a knee operation I think we can get information that is more meaningful if we study the quality of knee function rather than the quality of life.

Bertram Zarins, M.D. Consulting Editor for Sports Medicine

Frederick A. Matsen, M.D.
Posted on October 09, 2005
Counterpoint to “Are Validated Questionnaires Valid?”
University of Washington, Seattle, WA 98195

To The Editor:

In the August 2005 volume of the Journal, my respected colleague, Bert Zarins, wrote a provocative piece with the title quoted above. While what he says rings true, there are other aspects to the “validity” issue that should be added to our thoughtful consideration. I will enlist the help of another Massachusetts General surgeon who is often, but incompletely, quoted:

“Already in 1900 I had become interested in what I have called the End Result Idea, which was merely the common-sense notion that every hospital should follow every patient it treats, long enough to determine whether or not the treatment has been successful, and then to inquire ‘if not, why not?’” “We had found that this routine tracing of every case, interesting or uninteresting, had brought to our notice many things in which our knowledge, our technique, our organization, our own skill or wisdom, and perhaps even our care and our consciences, needed attention.” When he presented this idea in 1913 in the great hall of the Philadelphia Academy of Medicine, Codman pointed out that answering these questions is of primary interest to the patient, the public, and the medical field. He then asked, “Who represents or acts for these interests” and answered “Strangely enough the answer is: No one.” In his infamous cartoon of the Back Bay Golden Goose Ostrich, he showed the bird producing golden eggs of profit while hiding her head in the sand so she could not see how much (or how little) the care was benefiting the patient. (1). For his insolence he was fired from the hospital.

A century later we are struggling with the same issues.

If we are to have a valid assessment of the effectiveness of an operation, we first need a method for determining the result in a reasonable statistical sample of patients having this procedure. As an example of the difficulty in achieving this objective, Hassan and others have shown that the great majority of shoulder arthroplasties are done by surgeons who perform fewer than four per year (2). Because these common cases are not included in studies of outcomes, the surgical results data published in the peer reviewed literature are not particularly relevant, because they lack common external validity – that is, they cannot be applied to the common surgical experience. We are surely a long way from complying with Codman’s notion of following “every” patient. The issue is that, as he pointed out, no one is representing or acting in the interest of the average patient having the surgery.

The two key steps in helping us get valid data on the common experience are to apply the simplest effective tool for measuring the result of treatment (such as a series of ‘yes or no’ questions about comfort and function that the patient can complete before and sequentially after their procedure); and to exert leadership at all levels for the broad collection and analysis of these data. The argument for the most practical metrics for this purpose must be made forcefully to balance the views expressed by Dr. Zarins that more costly “objective measures, such as physical examination findings, radiographs, and arthrometer measurements" are required for validity. The impracticality of rigorously applying these “objective measures” in common practice systematically and selectively excludes the largest and most important groups of patients and surgeons from the sample, those in general orthopaedic practices.

As the expense of medicine becomes progressively unaffordable, the need for externally valid, generalizable analyses of surgical effectiveness will be pressed on us by the employers, who are responsible for the medical benefits of their employees, and by the government, which is responsible for the medical benefits of those without medical benefits from their employers. It is in their interest.

The benefit from a surgical procedure can only be determined if we assess the patient in the same way before and after the treatment; a point emphasized by Dr. Zarins. While there is inconsistent use of the word ‘outcome’ in our literature, common sense indicates that it means what comes out of the treatment in terms of the chosen measurement tool applied at a reasonable time after surgery. In order to determine the effectiveness of the treatment we need also to know what goes into the treatment or the “ingo”. Thus the effectiveness or result of the arthroplasty is the difference between the outcome and the ingo as indicated by the same metric.

We recognize that the result of surgery is determined by three major factors: the patient; the surgeon; and the implant or prosthesis, if one is used. Let us consider these determinants in reverse order.

The prosthesis. There are essentially no data indicating that ‘improvements’ in prosthetic design have a significant effect on the benefit of an arthroplasty to the patient. For example, two recent articles by Mileti et al (3) and Churchill et al (4) found no difference between ‘first’ and ‘second’ generation humeral components. In this light it is of concern that many tens of millions of dollars have been spent on trying to ‘improve’ the humeral prostheses used in shoulder arthroplasty. For sure, the charges for implants are rising faster than either the established benefit to the patient from their use or the funds available in the health care system to cover the increased charges. This is in contrast to the application of vascular stents, which both increased the quality of result and reduced the cost of treating arterial occlusive disease. Were he alive today, Codman might ask, “in whose interest is the design of additional and more expensive varieties of orthopaedic prostheses?”

The surgeon. It has been determined that patients cared for by surgeons who performed fewer than two shoulder arthroplasties per year have more postoperative deaths, more surgical complications, and longer lengths of stay than those managed by surgeons with a volume of five procedures or more per year. In one study, Jain et al. (5) concluded, “...patients who have a total shoulder arthroplasty or hemiarthroplasty performed by a high - volume surgeon or in a high-volume hospital are more likely to have a better outcome.” Hammond et al. (6) found “...that the patients of surgeons with higher average annual caseloads of total shoulder arthroplasties and hemiarthroplasties have decreased complication rates and hospital length of stay compared with patients of surgeons who perform fewer of these procedures.” “Who holds the knife” appears to have a strong effect on the surgical result of a procedure.

The effects of inter -surgeon differences extend beyond the surgical technique in the operating room. Harryman et al.(7) demonstrated that groups of patients with the same diagnosis in the practices of different surgeons were quite different from each another. Thus the generalizability of the results obtained by a given surgeon treating his or her own patients may not be any more generalizable to all patients than the results obtained by a particular math professor in an elite university would be generalizable to all college students.

The surgeon is the method.

The third factor is the patient - the complex of the specific surgical problem and the human being that is affected by it. William Osler is credited with the statement: "It is more important to know what sort of patient has the disease than to know what sort of disease has the patient." Although it would seem intuitive that a patient’s physical, emotional and social welfare would all have a powerful effect on his or her ability to benefit from a surgical reconstruction, there has been little research on this important determinant of surgical effectiveness. Yet, as shown by Rozencwaig et al,(8) tools for this critical research, such as the SF 36 patient self-assessment, are at hand providing a compelling and practical approach for exploring the relationship of the patient’s perceived physical and mental health to the result of orthopaedic procedures.

As Dr. Zarins and we consider the validity of approaches to clinical research in orthopaedic surgery, we must ask, “what question are we trying to answer?” and “will the answer we get from our study apply generally across our specialty, or only to a relatively few patients, practices, or surgeons?” If we are looking for a way to determine “is prosthesis A or prosthesis B better in the hands of a surgeon who does nothing but this type of surgery?” we would want a sophisticated set of tools for controlling the variability in patients and their pathology as well as for documenting the function, mechanics and radiographic anatomy before and sequentially after surgery – much as suggested by Dr. Zarins. If we are asking “do surgeons with high volumes, special training, or memberships in specialty societies get better results?”, we need a method for including as many patients and as many surgeons as possible so that we have a statistically and externally valid sample. This will require the simplest and least taxing methodologies if it is to have a chance of succeeding. Finally, if we want to know what patients do best and what patients fail to benefit from a procedure, we need to capture data from the broadest possible sample of what is really happening. It is not a question of subjective versus objective; validity comes from matching the tool to the task.

Frederick A. Matsen III, M.D.

Edward V. Fehringer, M.D.

References:

(1) Codman E.A. (1934). The Shoulder: Rupture of the supraspinatus tendon and other lesions in or about the Subacromial Bursa. Boston, MA: Thomas Todd. Preface

(2) Hasan, S., Leith, J., Smith, K.L., Matsen FA3rd. (2003). The distribution of shoulder replacements among surgeons and hospitals is significantly different than that of hip or knee replacements. J Shoulder and Elbow Surg. Mar-Apr,12(2), 164-169.

(3) Mileti J, Sperling JW, Cofield RH, Harrington JR, Hoskin TL. (2005). Monoblock and modular total shoulder arthroplasty for osteoarthritis. J Bone Joint Surg Br. Apr, 87(4), 496-500.

(4) Churchill RS, Kopjar B, Fehringer EV, Boorman RS, Matsen FA 3rd. (2005). Humeral component modularity may not be an important factor in the outcome of shoulder arthroplasty for glenohumeral osteoarthritis. Am J Orthop. Apr, 34(4), 173-6.

(5) Jain N, Pietrobon R, Hocker S, Guller U, Shankar A, Higgins LD. (2004). The relationship between surgeon and hospital volume and outcomes for shoulder arthroplasty. J Bone Joint Surg Am. Mar, 86-A(3), 496-505.

(6) Hammond JW, Queale WS, Kim TK, McFarland EG. (2003). Surgeon experience and clinical and economic outcomes for shoulder arthroplasty. J Bone Joint Surg Am. Dec, 85-A(12), 2318-24.

(7) Harryman II, D.T., Hettrich, C., Smith, K.L., Campbell, B., Sidles, J.A., and Matsen III, F.A. (2003). A prospective multipractice investigation of patients with full thickness rotator cuff tears: The importance of co-morbidities, surgeon, and other co-variables on self-assessed shoulder function and health status. J Bone Joint Surg Am. 85A(4), 690-96.

(8) Rozencwaig R, van Noort A, Moskal MJ, Smith KL, Sidles JA, Matsen FA 3rd. (1998). The correlation of comorbidity with function of the shoulder and health status of patients who have glenohumeral degenerative joint disease. J Bone Joint Surg Am. Aug, 80(8), 1146-53.

Bruce D. Beynnon
Posted on September 27, 2005
Are Validated Questionnaires Valid?
The University of Vermont

To the Editor:

The focus of Dr. Zarins recent editorial was on the use of different types of outcome measures to assess the treatment of anterior cruciate ligament injuries. He appropriately outlined the procedure which should be followed to establish that a patient-oriented outcome measure has been validated. There is no doubt that this is critical for outcomes that measure a subject’s perspective of an injury and subsequent treatment.

Dr. Zarins made a very important point when he highlighted that the validation of a questionnaire, or an objective test, should involve testing its test-retest reproducibility, responsiveness to detect clinically important changes, face validity, content validity, and construct validity. We also agree with Dr. Zarins’ perspective-- all outcome measures should be validated for the specific application for which they are intended.

Dr. Zarins also stated that the Knee Injury and Osteoarthritis Outcome Score (KOOS) “is designed for osteoarthritis and has not been validated for the study of anterior cruciate ligament injuries”. This is simply not true. The KOOS has undergone extensive validation for the assessment of anterior cruciate ligament injuries(1,2) This work has included evaluating its test-retest reproducibility, responsiveness to detect clinically important changes, face validity, content validity, and construct validity. In addition, it has been validated for orthopaedic interventions such as meniscectomy.(3) The KOOS meets the basic validity criteria (outlined above) to evaluate injury and treatment of anterior cruciate ligament tears, and this research has been published for some time.(1,2,3) The KOOS is a valid, reliable, and responsive self- administered instrument that can be used for short- and long-term follow- up of different types of knee injuries. <

We take this opportunity to congratulate Dr. Spindler and co-authors for their interesting study on long-term outcomes after ACL injury.(4) Their results highlight the importance of using several complementary and validated outcome measures to evaluate the patient-relevant aspects of this complex injury. The clinical, patient relevant, endpoint serves as our gold standard for the pharmacological, surgical or other form of medical treatment.

Sincerely,

Bruce D. Beynnon, PhD, Director of Research, The University of Vermont, Burlington, VT, USA

Ewa Roos, PT, PhD, Lund University Hospital, Lund, Sweden

Stefan Lohmander, MD, PhD, Lund University Hospital, Lund, Sweden

Robert J. Johnson, MD, The University of Vermont, Burlington, VT, USA

References:

1) Roos EM, Roos HP, Lohmander LS, Ekdahl C, Beynnon BD. Knee injury and osteoarthritis outcome score (KOOS) – Development of a self- administered outcome measure. J Orthopaedic and Sports Physical Therapy. 1998;78(2):88-96.

2) Roos EM, Lohmander LS. The knee injury and osteoarthritis outcome score (KOOS): from joint injury to osteoarthritis. Health and Quality of Life Outcomes. 2003;1:64.

3) Roos EM, Roos HP, Ekdahl C, Lohmander LS. Knee injury and Osteoarthritis Outcome Score (KOOS) – Validation of a Swedish version. Scand J Med Sci Sports 1998;8:439-48.

4) Spindler KP, Warren TA, Callison JC Jr, Secic M, Fleisch SB, Wright RW. Clinical outcome at a minimum of five years after reconstruction of the anterior cruciate ligament. J Bone Joint Surg Am. 2005;87:1673-1679.

Allen F. Anderson, M.D.
Posted on September 26, 2005
Are Validated Questionnaires Valid?
Tennessee Orthopaedic Alliance/The Lipscomb Clinic Fdn. for Research & Education

To The Editor:

We were concerned by the general tenor and inaccuracy of the editorial “Are Validated Questionnaires Valid?” 2005:87:1671-2. The editorial ostensibly sanctions a return to historical methods of outcome assessment that primarily focused on the structure of the knee joint, not function of the patient. Unfortunately, this unscientific approach of evaluating outcomes based on impairment measures and reporting results derived from knee evaluation forms that were neither valid nor comparable often resulted in erroneous conclusions by orthopaedic researchers. The editorial fails to acknowledge that patient reported “outcome measures are now in vogue” because critical analysis of the orthopaedic literature demonstrated a number of deficiencies that call into question our fundamental basis of learning.

We disagree with the implication that validation is of questionable value, because “there is no accepted standard of what constitutes validation” and consequently, “it is self-proclaimed”. According to current theory among psychometricians, validation includes the accumulation of evidence to provide a sound scientific basis to support interpretation of test scores for an intended purpose(1). In reality, it is the interpretation of the test scores required by the proposed uses that are evaluated, not the outcome instrument itself.

Having said that, the editorial makes a cogent point that use of a “validated instrument” does not necessarily insure that the results of a study will be valid. Validity of a study (i.e. the degree to which one can make cause – effect conclusions) is dependent on the design of the study, which includes evidence for the validity of the outcomes instrument and the procedures that are used to implement the study. Researchers should refrain from stating that an instrument is “validated”, implying that it is validated for all uses. When patient-reported outcome instruments are used in more than one way, each interpretation/use requires validity evidence. Consequently, the comments regarding the SF- 36, WOMAC and KOOS are appropriate, as there is little validity evidence to support the interpretation of these instruments for a population of patients following ACL reconstruction.

The statement that construct validity is determined by correlating subject’s answers to questions with objective measurements is an over simplification. Construct validity is the degree that test scores conform to hypothetical relationships to other variables, not just objective measures(2). Testing the hypothesis that scores of an outcomes instrument would be related to the SF-36 physical function score, but not related to the mental component score provides some evidence of construct validity. Some researchers refer to this as convergent and divergent validity evidence. Additional evidence of construct validity may be provided by demonstrating hypothesized relationships with other measures of knee function (some would call this criterion-related validity) as well as demonstrated hypothetical relationships between groups known or hypothesized to have different levels of function. For example, it may be hypothesized that younger subjects would have higher scores than older subjects, subjects with normal knees would have higher scores than those with a history of knee problems, or, patients would have higher scores after ACL reconstruction than patients with chronic knee conditions such as osteoarthritis. Construct validity of IKDC Subjective Knee Form was demonstrated by the degree that subject test scores conformed to these and other hypothesized relationships(3,4,5).

We also disagree with the comments regarding the IKDC Subjective Knee Form. The editorial stated that because the IKDC form is generic for knee problems and not specific for sports and instability, that it can have little meaning for evaluating patients with ACL injuries. A major objective in the development of the IKDC Subjective Knee Form was to create a region specific, rather than a disease specific, form that was appropriate to evaluate patients with a variety of knee impairments including ligament and meniscal injuries, patellofemoral pain and osteoarthritis, because function of the knee is dependent on more than just one structure (i.e. the ACL). Outcome instruments that are disease specific may not be appropriate for evaluation of outcomes of treatment for ACL tears when other conditions occur concomitantly. Development of a single form that is valid for patient’s with a variety of conditions affecting the knee simplifies data collection because the same form may be used for all patients. Item response theory demonstrated that the IKDC Subjective Knee form functioned equally well for patients with ACL injuries and other conditions(3). The alternative would be to use multiple forms, like Spindler, et al,(8) to document the outcomes of patients with traumatic arthritis and other conditions that frequently occur in the ACL deficient knee.

As evidence to support the contention that the IKDC Subjective Knee Form has little meaning for evaluating patients with ACL injuries, the editorial stated that a patient who had a torn ACL with a knee that is functionally unstable can receive 95 of 100 points. On the contrary, our data does not support that contention. The mean IKDC Subjective Knee Form score for patients with an ACL deficient knee was 47.6 at baseline and 80.1 at final followup(5). The mean IKDC score for males from 18 – 24 years of age with normal knees is 89.1(4). There were also few ceiling effects which are common in general health measures(3). Therefore, it is highly unlikely that a patient with an unstable ACL deficient knee would score 95 on the IKDC Subjective Knee form. In addition, the IKDC Subjective Knee score has been shown to be significantly correlated with patient satisfaction after ACL reconstruction(6).

Historic methods of orthopaedic evaluation were primarily based on impairment measures such as range of motion, strength and laxity. Although we agree with the editorial that both subjective and objective outcomes are necessary for complete evaluation of results, it is important to understand the relationship between impairment and resulting activity limitations and participation restrictions is not direct. For example, several authors have demonstrated that there is no relationship between KT -1000 measurements and patient-reported activity and participation(6,7). We believe that activity and participation are of most concern to the patient. Therefore, health related quality of life should be the primary outcome measure. The secondary, objective outcome measures complete the picture by evaluating effectiveness of treatment in reducing impairment.

Recent analysis of the orthopaedic literature has demonstrated deficiencies in our past methods of assessment. Despite the negativity of this editorial, orthopaedic surgeons should continue to improve methodological quality by embracing valid clinical outcomes research that includes patient-reported outcome measures, an evolution of assessment that is scientifically rigorous, allowing critical evaluation of results and refinement of knee surgery.

Allen F. Anderson, M.D., Tennessee Orthopaedic Alliance/The Lipscomb Clinic, The Lipscomb Clinic Foundation for Research and Education, Nashville, Tennessee, U.S.A.

James J. Irrgang PhD PT ATC, Director of Clinical Research, Department of Orthopaedic Surgery, University of Pittsburgh School of Medicine

REFERENCES:

1) American Educational Research Association, American Psychological Association; National Council on Measurement in Education: Standards for Educational and Psychological Testing. Published by American Educational Research Association, Wash DC 1999.

2) Steiner DL, Norman GR: Health Measurement Scales: A Practical Guide to their Development and Use 2nd Ed. Oxford University Press, Oxford, 1995.

3) Irrgang JJ. Anderson AF. Boland AL. Harner CD. Kurosaka M. Neyret P. Richmond JC. Shelborne KD. Development and validation of the International Knee Documentation Committee Subjective Knee Form. Am.J.Sports Med. 2001; 29: 600-613.

4) Anderson AF. Irrgang JJ. Kocher M Mann B, Harrast J., Members of the IKDC. International Knee Documentation Committee (IKDC) Subjective Knee Form: Data from a nationally representative sample. Am.J.Sports Med. (in press)

5) Irrgang, J. J., Anderson, A. F., Boland, A. L., Harner, C. D., Neyret, P., Richmond, J. C., Shelborne, K. D., and International Knee Documentation Committee. Responsiveness of the International Knee Documentation Committee Subjective Knee Form. American Journal of Sports Medicine. (In revision)

6) Kocher MS. Steadman JR. Briggs KK. Sterett WI. Hawkins RJ. Relationships between objective assessment of ligament stability and subjective assessment of symptoms and function after anterior cruciate ligament reconstruction. American Journal of Sports Medicine. 32(3):629- 34, 2004 Apr-May.

7) Snyder-Mackler L, Fitzgerald GK, Bartolozzi AR, Ciccotti MD: The relationship between passive joint laxity and functional outcome after anterior cruciate ligament injury. Am J Sp Med 25: 191-195, 1997.

8) Spindler KP, MD, Waren TA, NP, ATC, Callison, Jr. JC, et al. Clinical outcome at a minimum of five years after reconstruction of the anterior cruciate ligament. J Bone Joint Surg. 2005;87-A, No. 8, pp. 1673-1679.

James G. Wright, M.D.
Posted on September 10, 2005
Are Validated Questionnaires Valid?
Associate Editor JBJS

To the Editor:

I write to disagree with Dr. Zarin’s recent editorial [Zarins B.: Are Validated Questionnaires Valid? J Bone Joint Surg Am. 87:1671-1672, 2005]. Dr. Zarins concludes that “subjective data are insufficient for evaluating the results of an operation” and “subjective data, no matter how valid, should not stand alone”.

The essence of Dr. Zarins’ argument is that subjective measurements are sometimes used inappropriately. His main example is that measures developed in one context are sometimes being used inappropriately in another. He also states that finding a relationship between independent variables does not mean that one caused the other.

Neither of these examples is convincing or relevant because exactly the same concerns apply to objective measurements. The choice is simple--in evaluating a surgical procedure, if I have to choose between an objective measure, and a subjective measure which validly addresses patient clinical concerns, I know which measurement the patient would ask me to choose.

Yours truly,

James G. Wright, MD, MPH, FRCSC

Related Content
The Journal of Bone & Joint Surgery
JBJS Case Connector
Topic Collections
Related Audio and Videos
Clinical Trials
Readers of This Also Read...
JBJS Jobs
01/22/2014
Pennsylvania - Penn State Milton S. Hershey Medical Center
01/08/2014
Pennsylvania - Penn State Milton S. Hershey Medical Center
02/05/2014
Oregon - The Center - Orthopedic and Neurosurgical Care and Research
04/16/2014
Georgia - Choice Care Occupational Medicine & Orthopaedics