Explanation & Elaboration

16. Limitations

1. Considers possible sources of confounding, bias, or imprecision in design, measurement, and analysis that might have affected study outcomes (internal validity)
2. Explores factors that could affect generalizability (external validity), for example: representativeness of participants; effectiveness of implementation; dose-response effects; features of local care setting
3. Addresses likelihood that observed gains may weaken over time, and describes plans, if any, for monitoring and maintaining improvement; explicitly states if such planning was not done
4. Reviews efforts made to minimize and adjust for study limitations
5. Assesses the effect of study limitations on interpretation and application of results

Example

This study included multiple health care organizations and community agencies in a single metropolitan area rather than a single health care organization and multiple community agencies from several regions. This approach minimized concerns of health care organizations in a single region about possible adverse selection of patients with dementia after the creation of enhanced services, and it more realistically reflected patterns of use of community resources encouraged by the chronic care model.

We adjusted for health care organization provider but not for care manager because 2 health care organizations employed only 1 care manager each, thereby precluding our ability to distinguish health care organization from care manager effects. We decided a priori to adjust for health care organization rather than for care manager: The care managers received identical training and used the same assessments and treatment protocols, whereas we purposively recruited health care organizations with diverse characteristics and believed that health care organization rather than care manager differences would influence outcomes. Dyads in the intervention group could be referred to multiple community agencies, depending on the service needs identified through the assessment of health care organization care manager; referral to a community agency was a study outcome. Thus, we did not adjust for community agency care manager.

Our study sample was well-educated, was predominantly white, had relatively few comorbid conditions, and was not institutionalized. Accordingly, the intervention may need to be modified for institutionalized patients and for those without a usual source of care and stable insurance. Secondary outcomes and some care process measures were self-reported, but multi-item scales met standards for reliability, and support for validity has been reported for several measures. As with almost any quality improvement intervention study, medical record abstractors could have discerned aspects of the study intervention, and we did not assess the extent to which abstractors were blinded to intervention status. It is possible that medical record documentation may incompletely reflect actual care processes, but we believe that observed differences reflect actual differences in care rather than differences in documentation because this care management intervention was based on a model in which multidisciplinary teams were engaged in delivering or facilitating the delivery of much of the recommended care... [39]

Elaboration

The authors of this report identified a number of factors that could have introduced bias and affect the study measures. These included self-reporting of outcome and process measures, lack of blinding of medical record abstractors to the study interventions, and the decision to adjust for health care providers but not for the community agency care managers. A description of efforts to control for these factors and minimize their effects either in the study design or the analysis of results should be included in the limitations section. In addition, insight into how the identified factors could have potentially affected the study measures, by either artifactually increasing or decreasing the effects of the described intervention, should be included in the limitations section.

This section should also explore why the particular study design was chosen and whether it reflects clinical practice in a realistic fashion. The authors in this example note that they chose their design with multiple health care agencies in a single area to reduce adverse selection of patients and to better reflect practice based on the chronic care model. Although this study design may be helpful for those who are planning care for multiple practices, this study design may not be as helpful for the individual practitioner who may be interested in implementing the described interventions into a single practice location.
This particular study was a randomized, controlled trial, a study design that typically makes use of multiple inclusion and exclusion criteria to ensure that the intervention and control groups are similar. Although this may improve the internal validity of the study by reducing the effects of confounding or selection bias from different study populations (or in this example, different clinical practices), it may also reduce the external validity or generalizability of the study as the included patient populations or practices may not resemble the diverse practices, providers, or patients that exist in actual clinical care.

Finally, the limitations section should identify particular factors in the context and patient population that may affect generalizability. The success of improvement interventions depends on the specific contexts in which they occur; therefore, identification of unique traits or characteristics of the patient population, providers, institution, or geographic setting is critical. A clear understanding of the local setting in which the intervention is implemented is important to the reader who is interested in applying the described interventions in their own particular setting and population of patients. The authors identified their patient population as unlikely to be representative of other populations across the country, suggesting that the study's providers and practice settings are likely to be different from many other practices that care for patients who suffer from dementia. The impact of this discrepancy could be mitigated by the inclusion of more information about particular aspects of the providers and practice settings that could affect the study outcomes.

References

39. Vickrey BG, Mittman BS, Connor KI, et al. The effect of a disease management intervention on quality and outcomes of dementia care: a randomized, controlled trial. Annals of Internal Medicine. 2006;145(10):713-726.

to top ^