Explanation & Elaboration

12. Analysis

1. Provides details of qualitative and quantitative (statistical) methods used to draw inferences from the data
2. Aligns unit of analysis with level at which the intervention was implemented, if applicable
3. Specifies degree of variability expected in implementation, change expected in primary outcome (effect size), and ability of study design (including size) to detect such effects
4. Describes analytic methods used to demonstrate effects of time as a variable (for example, statistical process control)

Example

1. "By monitoring the number of days between infections, the g chart has greater detection power for rare events compared with conventional binomial based approaches. Whenever a CR-BSI [catheter related bloodstream infection] occurred we added the data point to our g chart and calculated the number of days from the previous infection. ...The g chart is a type of statistical process control (SPC) chart and therefore requires a basic understanding of the principles inherent to SPC. ... At the start of the project we constructed a baseline (preintervention) g chart by querying the NNIS [National Nosocomial Infection Surveillance System] database from 1 January 2000 to 31 October 2002 (fig [1], observations 1-39). We only plotted CR-BSIs on the g chart for catheters inserted by the MICU - for example, dialysis catheter CR-BSIs were not included on the g chart..."
   
   
   "We measured the average time between infections (27 days) and used Benneyan's method for calculating an upper control limit (UCL=109 days) at three standard deviations above the pre-intervention mean. During the pre-intervention period the number of days between infections was consistently below the UCL. The absence of any points above the UCL suggested that the variation in time between CR-BSIs was inherent to our current process of care (that is, common cause), and that the proper way to reduce the CR-BSI rate was through process redesign or CQI. Our goal was to reduce the CR-BSI rate and thereby increase the time between events. Based on the g-chart, we considered our intervention successful if data points fell above the UCL since this would correspond to a decreased CR-BSI rate." [21]
2. "All audiotaped interviews were transcribed verbatim. Two researchers...independently reviewed the manuscripts and marked comments about barriers to adherence. Remarks of professionals were compared and classified into categories of potential barriers to physician adherence according to a conceptual model.... The two reviewers discussed all the remarks that they had individually highlighted and classified until consensus was reached. They consulted a third researcher to make a formal judgment about differences in classification. If controversy remained, the comment was considered ambiguous and was excluded." [32]

Elaboration

The analysis plan is intimately related to the study design. A defining characteristic of quality improvement is to show that the strategy for change (which is often multi-faceted) works to bring about a measurable difference in the process or outcome measures. Such demonstrations have resulted in a plethora of before-after studies comparing two data points, one value representing the pre-intervention period and the second value representing the post-intervention period. Such two-point before-after analyses are weak demonstrations of change; consequentially, they are generally considered pre-experimental, and hence acceptable as secondary, hypothesis-generating contributions, which require additional hypothesis-testing analysis.

Strong demonstrations of improvement projects overcome the limitation of before-after analysis by capitalizing on the concept of replication. The strength of replication is that it accumulates confidence that the intervention produces the pattern of change observed in the results. Example (a) from Wall et al. uses replication in the form of statistical process control (SPC) analysis to demonstrate a change from pre-intervention (baseline phase) to post-intervention (implementation phase). The authors state the type of SPC that was used ("g-chart"), the timeframe for the baseline data ("1 January 2000 to 31 October 2002"), and the inclusion criteria for the data ("for catheters inserted by the MICU"), thus specifying the analysis for the main outcome measure in the study. SPC is within the family of time-series analyses that plot multiple points, and where each point represents the operationally defined unit of measurement (such as a daily, weekly, or monthly proportion, mean, or time between events). We recommend consultation with a statistician familiar with time series analysis at the start of a study to ensure that the analysis is appropriate for the questions that are posed.

SPC requires a stable baseline from which to evaluate changes (improvements) that occur, but healthcare systems are also subject to secular trends which may appear in any given period of pre-intervention and during the intervention period. For these instances, other analytic techniques may be more appropriate. For example, the interrupted time series technique estimates the intervention's effects and tests for statistical significance of change even with background noise from secular trends33, 34. Alternatively, some authors may choose to use sophisticated statistical techniques that have been employed in longitudinal or time-dependent studies.29 These are appropriate for use in analysis of quality improvement, but these more sophisticated statistical techniques usually require consultation with a statistician for appropriate design, analysis, and interpretation.

Also note that replication is a two-edged sword in the evaluation of complex social interventions. Particular interventions work well in some contexts but poorly, or not at all, in others. In the aggregate, therefore, introducing a complex social intervention into many different contexts is likely to lead to a null result overall, even when the intervention is effective in at least some settings.

Finally, not all quality improvement research is quantitative. Example (b) by Schouten et al. demonstrates a brief description of qualitative analytic methods. Qualitative research is rich in discovering knowledge of root causes, understanding flow of process, formulating concept classifications and themes, and gaining insight into mechanisms of change, context, perspective, and perception. While simply relating anecdotes from qualitative studies can be powerful in affecting memory, attitudes, and beliefs, qualitative research involves the use of systematic, rigorous, and robust study techniques. Schouten's example clearly describes how the interview data were transcribed, extracted, summarized, and adjudicated. Other qualitative methods might use ethnographic or qualitative analysis software, provide evidence of interrater reliability, or provide procedural detail for replicable data classification schemes. Authors who use qualitative study designs should also consult the guidelines for reporting of observational studies (STROBE)4 to be sure the methods and descriptions are consistent with those recommendations.

References

21. Wall RJ, Ely EW, Elasy TA, et al. Using real time process measurements to reduce catheter related bloodstream infections in the intensive care unit. Quality & safety in health care. 2005;14(4):295-302.

32. Schouten JA, Hulscher ME, Natsch S, Kullberg BJ, van der Meer JW, Grol RP. Barriers to optimal antibiotic use for community-acquired pneumonia at hospitals: a qualitative study. Quality & safety in health care. 2007;16(2):143-149.

to top ^