Diagnostic Safety Research Resource Center

Back to Toolkit Home

 

This section provides a listing of different definitions of errors and different ways to categorize the errors. We will also describe several of the types of studies that evaluate diagnostic error.

Conceptual Models & Definitions View

Error Taxonomies View

To better understand and measure diagnostic errors, we must develop organized systems to collate and categorize them. Several error taxonomies have been suggested for this purpose and are briefly described below.

Overview of Data Sources View

  • Records of clinical encounters
    • Description: Review and evaluation of medical records for diagnostic errors.
    • Sampling: Records are identified based on symptoms, diseases, health care units/processes, or other criteria. "Trigger" criteria can be to select scenarios that are suspected to have the highest risk of error, which may help reduce the number of records requiring expert review.
    • Error Assessment: The determination of whether a diagnostic error has occurred is usually made by one or more expert reviewers.
    • Strengths:
      • Often easily accessible within institutions.
      • Can be used to estimate the incidence and prevalence of errors within included group.
    • Weaknesses:
      • Dependent on quality of clinical documentation which often lacks sufficient information to know whether an error occurred.
      • Dependent on inter- and intra-expert agreement of error.
      • Unlikely to allow conclusions regarding the causes of errors, especially when unrecognized.
      • Time-intensive to review records to identify cases, especially when error rates are low.
      • HIPAA can limit access to covered entities across institutions.
    • Recommendations:
      • Predetermine assessments and make them as objective as possible.
      • Consider multiple reviewers to account for inter-expert variability.
      • Consider re-review of a subset of records to evaluate intra-expert agreement.
      • Consider appropriate ethical approach to patient notification if unrecognized error is identified.
  • Autopsies
    • Description: Autopsies are detailed post-mortem examinations. They are often considered to be the "gold standard" for determining diagnostic accuracy. Major diagnostic discrepancies are identified in 10-20% of cases.
    • Sampling: Autopsies are typically generated from inpatients and emergency department settings and generally require consent of families or are required for legal reasons (e.g., in cases of suspected murder).
    • Error Assessment: Autopsy provides a systematic gross and microscopic evaluation of the entire patient.
    • Strengths:
      • Systematic examination of the entire body or large regions of it.
      • Able to provide definitive pathological evidence of many conditions, confirming or refuting the diagnoses received while the patient was alive.
    • Weaknesses:
      • In the United States, autopsy is obtained only a small subset of cases, but they remain more common in other parts of the world.
      • Families may be more likely to consent to autopsy when they suspect a diagnostic error has occurred.
      • Physicians may either be more likely to recommend autopsy for difficult cases where they had lingering diagnostic uncertainty or to discourage autopsy when they are concerned they may have made an error.
      • Autopsy findings may not directly correlate with clinical findings and cannot always determine the cause of physiological changes.
      • A large number of incidental findings of no clinical relevance are uncovered by autopsy and should not be construed as diagnostic error.
    • Recommendations:
      • Encourage system-wide approaches to consent for autopsy to reduce selection biases.
      • Recognize the potential for autopsy to identify incidental findings and exclude these from diagnostic error assessments.
  • Voluntary reports
    • Description: Patients, families, and healthcare workers file reports of suspected diagnostic error.
    • Sampling: Usually reporting occurs within a healthcare system and is voluntary.
    • Error Assessment: The person reporting makes the initial determination of error. While these can be reviewed further for accuracy, reporting is often anonymous and edge cases are difficult to verify.
    • Strengths:
      • Allows a unique opportunity to explore both individual and system factors that contribute to error.
      • Provides a mechanism for various stakeholders (patients, families, nurses, physicians, staff, etc.) to report errors.
    • Weaknesses:
      • Substantial underreporting.
      • Lack of common format for error reporting.
      • Quality of data can vary widely especially.
      • Often no mechanism to investigate reports further.
      • Time, fear of reprisal, and reluctance to call attention to one’s own errors are barriers to reporting.
    • Recommendations:
      • If possible, provide a mechanism for further investigation, e.g., follow-up interviews, although this must be balanced by the benefits of anonymity.
      • Develop common formats for reporting.
  • Re-review of Source Data
    • Description: When source diagnostic data is persistent (e.g., radiographs, pathology slides, patient photographs/videos), additional experts can re-review the same source data and discrepancies can be determined.
    • Sampling: Sampling is only limited by the frequency of the underlying condition of study and the availability of an appropriate data source and experts.
    • Error Assessment: Agreement/disagreement among expert reviewers.
    • Strengths:
      • Easily allows for controlled experiments since the assessments are not confounded by time.
      • Experts may even disagree with themselves permitting an opportunity to study situational factors.
    • Weaknesses:
      • Primarily limited to visual specialties, e.g., dermatology, radiology, ophthalmology, pathology
      • Tends to overestimate frequency of diagnostic error
    • Recommendations:
      • Mask experts to purposes of experiment, if possible.
      • Maintain a base rate of abnormality typical of "real world" conditions.
  • Computerized screening algorithms
    • Description: Software systems designed for the semi- to fully- automatic detection of medical conditions and potential errors.
    • Sampling: Systems can be applied to a wide variety of settings, but are usually applied automatically and consistently within the chosen setting.
    • Error Assessment: Software makes diagnostic decisions or identifies errors. Systems can be completely automated without human input after development. Alternatively, systems identify definitively normal cases while flagging uncertain cases for human review or augment the human reviewer by highlighting areas of concern.
    • Strengths:
      • Automation improves the systematic application of the diagnostic technology.
      • Algorithmic approaches can be adjusted in a clear and consistent fashion based upon the system’s successes and failures.
    • Weaknesses:
      • Performance requires periodic monitoring against a reference standard to ensure accurate and precise performance.
      • In most cases, substantial resources are required for development.
      • Rarely applicable to general situations due to caveats and complexities.
    • Recommendations:
      • Periodic quality control and adjustment should be part of best practices.
      • Provide system fail-safes to allow for unexpected circumstances.
  • Surveys
    • Description: Providers and patients are surveyed about diagnostic error.
    • Sampling: Surveys are distributed to the population of interest and individuals decide whether or not to respond voluntarily.
    • Error Assessment: Whether to report an error or not is determined by the person receiving the survey. The study team can assess the appropriateness of completed reports based only on the data provided.
    • Strengths:
      • Provides a clear denominator for the study population.
      • Allows sampling of a large population relatively easily and quickly.
    • Weaknesses:
      • Response frequencies tend to be low leading to underestimation and selection biases.
      • When providers are surveyed, detail may limit ability to determine if unique cases are being reported.
      • Limited ability to follow-up for additional information in uncertain cases.
    • Recommendations:
      • Provide appropriate incentives to offset time required to fill out the survey and to encourage responses.
      • If possible, provide a mechanism for further investigation, e.g., follow-up interviews, although this must be balanced by the benefits of anonymity.
  • Diagnostic testing audits
    • Description: Audits are performed in diagnostic testing settings (e.g., the clinical laboratory or radiology) to estimate the incidence and prevalence of diagnostic error.
    • Sampling: Samples are limited to those related to diagnostic testing.
    • Error Assessment: Control samples are used to assess test performance.
    • Strengths:
      • Control samples permit analytical errors to be very effectively identified.
    • Weaknesses:
      • Modern quality control methodologies have resulted in a very low rate of analytical error.
      • The vast majority of errors result from preanalytical (test ordering) and postanalytical (test reporting and interpretation) than from analytical errors.
    • Recommendations:
      • Continued diagnostic testing audits are a critical part of maintaining the successes achieved in reducing analytical errors.
      • Future research should be primarily directed at pre- and postanalytical errors.
  • Standardized patients
    • Description: Standardized patients (real or simulated) with "classic" presentations are sent surreptitiously into real clinics.
    • Sampling: The need for standardized patients practically reduces the number of medical practices that can be evaluated by this methodology:
    • Error Assessment: Errors are determined by whether the patients are correctly diagnosed with their condition.
    • Strengths:
      • Use of "real world" settings.
      • Greater control over patient presentation allows study of factors related to misdiagnosis.
      • Allow estimation of both the frequency of error and etiologies underlying error.
    • Weaknesses:
      • Only a small subset of conditions can be studied in this fashion due to overlap in presentations and need for standard patients.
      • Standardized patients are less likely to have significant comorbidities and other complexities that are common in typical patients.
    • Recommendations:
      • As much as possible, standardized patients should be representative of typical patients to permit generalization of results.
  • Malpractice Claims, Risk Management and Safety Reports
    • Description: Evaluation of claims sent to malpractice insurers or flagged incidents
    • Sampling: The underlying claim/incident is assessed to determine whether it is diagnosis-related
    • Error Assessment: The determination of whether a diagnostic error has occurred is usually made by one or more expert reviewers.
    • Strengths:
      • Focused on errors that cause harm
      • Requires fewer cases to review to identify errors
    • Weaknesses:
      • Because we only examine cases that led to claims or flagged incidents, it is a biased sample, so unable to determine prevalence of errors
    • Recommendations:
      • Predetermine assessments and make them as objective as possible.
      • Consider multiple reviewers to account for inter-expert variability.
      • Consider re-review of a subset of records to evaluate intra-expert agreement.

Outline of Diagnostic Error Data Sources and Methods View