National hospital quality rating systems from The Leapfrog Group, Consumer Reports, U.S. News & World Report, and Healthgrades rarely agree on which hospitals are the nation’s best and worst, a recent study has found.

National Hospital Ratings Systems Share Few Common Scores And May Generate Confusion Instead Of Clarity,” which appeared in the March 2015 Health Affairs, compared the four systems for overlap in definitions, methodologies, transparency, and results. The study, with a stated limitation of one round of ratings from each system, created high-performer, medium performer, and low-performer hospital categories to provide a comparison across the different ratings systems.

Researchers reported these findings:

  1. The Leapfrog Group and Consumer Reports focus their ratings on safety, but define it differently – Leapfrog defines it as “freedom from harm,” while Consumer Reports defines it as “a hospital’s commitment to the safety of their patients.” Healthgrades focuses on outcome measures of mortality and complication rates, and U.S. News focuses on ability to provide care for high-acuity patients.
  2. Starting with initial hospital eligibility for rating, the methodologies for evaluating hospitals for quality vary. Each uses some degree of process and outcomes measures. Consumer Reports is the only one that incorporated patient experience measures. Unlike the others, U.S. News uses a reputational survey of providers. Both U.S. News and Leapfrog include hospital structure (such as computerized physician order entry) into their ratings.
  3. Only two of the systems offer full methodological transparency: Leapfrog and Consumer Reports. The study discusses the need for full transparency so the ratings have value to providers, for quality improvement, and to patients, for choosing a provider.
  4. There were 27 cases of hospitals being rated as a high performer in one rating and as a low performer in another rating system. Performance categories often didn’t translate across the other ratings systems for the high-performing and low-performing categories.

The study also found considerable variation among ratings of hospitals with the largest percentages of Medicaid patients. These hospitals are statistically overrepresented among the low-performers by Leapfrog and Consumer Reports. Conversely, U.S. News has them statistically underrepresented among the low-performers. This inconsistency is of particular interest to America’s Essential Hospitals member hospitals, where more than half of all inpatient discharges and outpatient visits are for uninsured and Medicaid patients.

In a recent National Public Radio blog post, the study’s lead author, J. Matthew Austin, MS, PhD, said the findings suggest that consumers should ask, “What is that rating for? What does that rating represent?”