Explanation & Elaboration

11. Methods of evaluation

1. Describes instruments and procedures (qualitative, quantitative, or mixed) used to assess a) the effectiveness of implementation, b) the contributions of intervention components and context factors to effectiveness of the intervention, and c) primary and secondary outcomes
2. Reports efforts to validate and test reliability of assessment instruments
3. Explains methods used to assure data quality and adequacy (for example, blinding; repeating measurements and data extraction; training in data collection; collection of sufficient baseline measurements)

Example

"All indicators were dichotomous variables thought to be amenable to simple quality improvement measures. In general, the quality indicators allow for a longer time frame to administer the clinical intervention (vaccine, aspect of physical examination, or laboratory test) than suggested by the ADA [American Diabetes Association] guidelines. This leniency means that performance for these indicators should be better than for the ADA guidelines because decreasing the time frame for the clinical intervention would probably decrease average indicator performance.

We ascertained from the medical record whether the following appropriate care was performed for each eligible patient at least once during the 18-month period: (1) measurement of long-term glucose control as reflected by at least 1 test of glycosylated hemoglobin (hemoglobin A1c) or fructosamine, (2) measurement of serum cholesterol, (3) measurement of serum triglycerides, (4) measurement of serum creatinine, (5) performance of in-office foot examination, and (6) administration of influenza vaccine. Performance on an indicator was quantified by dividing the number of eligible patients who received the item by the total number of eligible patients.

...All data for the quality measures were obtained from chart review according to methods previously described. Charts were photocopied and abstracted centrally using MedQuest, publicly available software developed for HCFA [Health Care Finance Association]. The ACQIP [Ambulatory Care Quality Improvement Program] investigators developed a standardized chart review protocol and refined the protocol through pilot testing. As part of the protocol, abstractors underwent intense training with competency certification. The MedQuest chart review module contained standard lists for variable synonyms, medications, diagnoses, and procedures. Throughout the chart abstraction period, 5% of charts were randomly sampled for dual abstraction and physicians evaluated chart abstractions for validity. Validity and reliability of all key variables were at least 95%." [31]

Elaboration

The description of the methods of evaluation outlines what the study used to quantify improvement, why the measures were chosen and how the investigators obtained the data. The measures chosen may be outcomes or process measures, continuous or categorical, biological or behavioral. The measures also must be sensitive enough to detect meaningful change in the processes and outcomes. Measures should have an accepted, clear operational definition so that changes in the values can be determined statistically and, more importantly, significant to the clinical problem being studied. Different perspectives, such as provider, patient, payer, or societal, should also be considered during the delineation of measures.

In the example, the improvement project focused on achievable benchmarks in diabetes care. The study's measures were the proportion of the population receiving certain tests or interventions. These are process measures. If the study had looked at the change in hemoglobin A1c values, the number of influenza cases prevented or change in the quality of life of the patients, then the measures would have been outcome measures.

Whether process or outcome, measures should reflect a reasonable range of potential changes that may result from the intervention. Hemoglobin A1c is frequently used as an outcome measure to determine adequacy of glycemic control in diabetes related studies; however, both the disease and the change effort may cause or alleviate a burden to the patient or to the provider. Therefore, it may be appropriate to measure behavioral or functional factors such as quality of life or satisfaction. Cost is often an important variable in change efforts. It is imperative that the investigator look at the process and the proposed improvement plan and determine a reasonable, balanced approach (sometimes referred to as a "data dashboard") to measure those factors which are likely to be affected by the improvement intervention. Consensus panel guidelines and quality indicators are helpful in guiding the investigator's choice of measures, but the local environment and problem should be considered to optimize the measures chosen.

Once the measures have been chosen, the investigator needs to develop operational data definitions, collection forms, and determine how the data will be collected. The methods of data collection and data quality management should be described in the paper so that others may replicate the project. In the example above, chart abstraction was used to collect the data. The authors explain in detail how the data were abstracted and how a random sample was chosen for dual abstraction to confirm the adequacy of the abstraction method and the data.

The manuscript should also ensure that the factors being study are reliably measured. For biological specimens, this may include describing the laboratory process used or the method of specimen collection and handling. Behavioral domains should be measured with validated instruments. If a new scale is created for the change effort, then the reliability and validity of the scale should be described. The validity should include a construct validity model or comparison to a gold standard, if available. This informs the reviewer and reader that the measure accurately represents the factor or domain that was being assessed.

References

31. Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA : the Journal of the American Medical Association. 2001;285(22):2871-2879.

to top ^