CARDIOVASCULAR JOURNAL OF AFRICA • Vol 23, No 10, November 2012
546
AFRICA
Statistical profiling of hospital performance using acute
coronary syndrome mortality
SAMUEL OM MANDA, CHRIS P GALE, ALISTAIR S HALL, MARK S GILTHORPE
Abstract
Background:
In order to improve the quality of care deliv-
ered to patients and to enable patient choice, public reports
comparing hospital performances are routinely published.
Robust systems of hospital ‘report cards’ on performance
monitoring and evaluation are therefore crucial in medi-
cal decision-making processes. In particular, such systems
should effectively account for and minimise systematic
differences with regard to definitions and data quality, care
and treatment quality, and ‘case mix’.
Methods:
Four methods for assessing hospital performance
on mortality outcome measures were considered. The meth-
ods included combinations of Bayesian fixed- and random-
effects models, and risk-adjusted mortality rate, and rank-
based profiling techniques. The methods were empirically
compared using 30-day mortality in patients admitted with
acute coronary syndrome. Agreement was firstly assessed
using median estimates between risk-adjusted mortality
rates for a hospital and between ranks associated with a
hospital’s risk-adjusted mortality rates. Secondly, assessment
of agreement was based on a classification of hospitals into
low, normal or high performing using risk-adjusted mortal-
ity rates and ranks.
Results:
There was poor agreement between the point esti-
mates of risk-adjusted mortality rates, but better agreement
between ranks. However, for categorised performance, the
observed agreement between the methods’ classification of
the hospital performance ranged from 90 to 98%. In only
two of the six possible pair-wise comparisons was agreement
reasonable, as reflected by a Kappa statistic; it was 0.71
between the methods of identifying outliers with the fixed-
effect model and 0.77 with the hierarchical model. In the
remaining four pair-wise comparisons, the agreement was,
at best, moderate.
Conclusions:
Even though the inconsistencies among the
studied methods raise questions about which hospitals
performed better or worse than others, it seems that the
choice of the definition of outlying performance is less criti-
cal than that of the statistical approach. Therefore there is a
need to find robust systems of ‘regulation’ or ‘performance
monitoring’ that are meaningful to health service practition-
ers and providers.
Keywords:
Bayesian methods, health provider performance,
league tables
Submitted 11/1/11, accepted 28/9/11
Cardiovasc J Afr
2012;
23
: 546–551
DOI: 10.5830/CVJA-2011-064
Incidents of professional failure and the necessity to improve
efficiency and quality of care in the health service have led to
increasing demand for quality assurance and audits of medical
institutions.
1-5
This has allowed quality appraisal and optimal
targeting of resources to areas of need. These processes have
led to significant improvements in health outcomes; however,
variation in hospital performance remains.
5,6
A widely used and acceptable method to control variation in
health outcomes is based on case mix adjustment.
7-9
However,
failure to adjust appropriately for differences in case mix
may result in unfairly targeting hospitals admitting high-risk
patients. Indeed, the identification of hospitals having unusual
performance depends on the variables used in the risk-adjustment
model.
7,8
Furthermore, comparing hospitals on the basis of a risk-
adjustment model could be erroneous, as the risk model may be
wrong, or suffer from incorrect inclusion of prognostic factors.
4
More importantly, the disparity in risk-adjusted outcomes
may result from a variety of factors including definitions, data
quality, structural and institutional management factors, and
resource characteristics that have a direct effect on clinical
processes.
4-6
To this end, differences in case mix should be
accounted for in a suitable risk-adjustment model and differences
in definitions and data quality kept to a minimum. Any residual
variation in outcome between hospitals would therefore reflect
hospital quality of care, the basis for medical institutional
profiling methods.
7-15
However the extent to which these hopes
are satisfied remains uncertain.
There is a large literature base on statistical methodology for
health provider profiling.
10-13
Simple methods use ratios of the
observed to the expected outcomes (indirect standardisation) or
odds ratios from a logistic regression analysis.
8,15
A number of
studies have shown disagreements between different frequentist
or Bayesian methods for profiling hospital performance
(
Marshall and Spiegelhalter,
11
Austin,
12
Ohlssen
et al.
,
15
Delong
et al
.
16
and Leyland and Boddy
17
).
In particular, random-effects
models are found to be more conservative in classifying
institutions as performance outliers.
11
There is therefore a need
for research to identify statistical models and ways that robustly
differentiate between hospitals and remain meaningful to the
medical practitioner.
12
Normand
et al.
,
10
Marshall and Spiegelhalter,
11
Austin
12
and Ohlssen
et al.
15
advocated using hierarchical Bayesian
random-effects methods for provider profiling. These methods
easily permit data pooling across institutions; thus overcoming
uncertainty associated with small institutions, which might be
Biostatistics Unit, Medical Research Council, Pretoria,
South Africa
SAMUEL OM MANDA, PhD,
Centre for Epidemiology and Biostatistics, University of
Leeds, Leeds, United Kingdom
CHRIS P GALE, PhD
ALISTAIR S HALL, PhD
MARK S GILTHORPE, PhD
