NEW YORK, April 08 (Reuters) -- The US News & World Report's annual "America's Best Hospitals" list is based on flawed methodology, and may end up misleading consumers, say researchers.
Although it purports to assure consumers of hospital quality based on rigorous scientific analysis, the list "remains primarily an opinion survey, much as it has always been," according to a team led by Dr. Jesse Green from the Department of Clinical Evaluation and Outcomes Research at New York University (NYU) Medical Center. Their analysis of the 1995 edition of the rankings is published in this week's issue of The Journal of the American Medical Association (JAMA).
But a statement issued Tuesday by Avery Comarow, assistant managing editor of US News & World Report counters the JAMA piece. "Is it better for consumers of critical health care to have information that falls short of perfection? Or to have none at all?" asks Comarow.
The US News & World Report list was first instituted in 1990, in response, the NYU researchers say, to "growing concern about the effect of cost-containment efforts on the quality of health care." The annual report quickly became an influential best-seller. "Sales of the (list issue)... exceed the magazine's averages, and high-ranking hospitals cite the report in advertising and publicity materials," the NYU authors write.
But just how are those rankings arrived at? The list criteria stem from a 1980 quality assessment 'model' developed by A. Donabedian in his book "The Definition of Quality and Approaches to Its Assessment." Donabedian linked three separate components: "structure" -- the underpinnings of care such as staff, experience, and technology available; "process" -- the actual nature of the ongoing care itself; and "outcomes" -- the effectiveness of care based on patient death rates versus expected death rates for a particular illness.
Green and the other researchers say the National Opinion Research Center (NORC), which develops the rankings for US News & World Report, included Donabedian's trinity of care factors in their own assessment criteria. Yet the researchers say they discovered serious flaws in each component of the "Best Hospitals" list methodology.
First of all, they say, the list compilers tried to measure the physician talent pool by assessing the number of certified specialists at a given hospital. The problem was that although many specialists are certified by more than one specialty board, the 'list' recognized only one certification per doctor. This "allowed four prominent institutions, whose cardiology departments were ranked in the top 40 nationwide, to report no board-certified cardiologists." And the researchers say "some hospitals counted cardiologists, oncologists, and gastroenterologists as internists, receiving no credit for the medical expertise of their staff in these specialties."
Other hospitals went ignored due to perceived technological 'deficiencies'. The researchers say "some hospitals lost credit for providing services if the hospitals contracted with outside firms to furnish equipment." And specialty hospitals "scored poorly on the technology index because they did not require the broad range of equipment needed by a general hospital."
"Process" was perhaps the most flawed of the three assessment categories, the NYU researchers say. "It was based on a single result," a "reputation score" gleaned from a nationwide poll of doctors. Many, if not most, of the doctors, chose long-respected hospitals in cities located far from their actual practices. Because "the top-ranked hospitals are also the ones with the most name recognition, the survey confirms popular conceptions... being little more than a feedback loop that allows fame to be perceived as quality."
And because the vast majority of doctors surveyed chose the same 8% of hospitals, "the remaining 92% of the hospitals had no reputation points at all, eliminating all credit for the activities encompassed by process of care."
Finally, the researchers believe "outcomes" (measurements of patient survival rates), to be flawed as well. The "list" contrasts hospital death rates with mortality rates published by the federal government's Health Care Financing Administration (HCFA). But HCFA admitted in 1994 that its rates were probably erroneous, and pulled them from publication. However, "although the federal government had lost confidence in mortality data, the 'America's Best Hospitals' analysts continued using it in 1995," say the NYU researchers.
Green's team believe that US News and World Report's list still has some merit, labeling it "a pioneering effort." But they lament that, despite the estimated $1 trillion spent every year on health care in the United States, there remains an "astonishing paucity of data sources available for such research."
More thorough study, which might give Americans a more accurate depiction of hospital quality, would cost money, however. The researchers believe that "organizations, including managed care plans, that are profiting from the reduction of health care costs could shoulder more of the burden, instead of leaving the job of health care quality assessment to a news magazine."
"We have always advised readers to use the rankings as a starting point, not as the sole resource," says Comarow at US News & World Report. "And while we disagree with many of the points raised by Dr. Green, we are in total agreement with him that all Americans should have access to unshakable performance data in this vital area of their lives. The data available fall short of that standard because the health care industry has declined to develop them."
SOURCE: The Journal of the American Medical Association (1997;277(14):1152-1155)
