Measuring GP outcomes: a crisis of validity and meaning

IN recent years, GPs have been nudged into changing their clinical practice through the analysis of proxy variables. Our billing, prescribing and pathology testing data and our recording of other proxy variables are analysed by government, and then linked to rewards, or threats designed to change our clinical practice, all without any knowledge of the context in which these data may or may not be meaningful.

Instead of collecting the data that GPs want, others assign meaning to the data that we have. Urban psychiatrists who never see our patients comment on our mental health care by analysing our billing patterns. Health economists believe they can improve patient care by sending “nudge” letters to change the clinical behaviour of GPs who are statistical outliers on a range of measures, without measuring the clinical outcomes of their interventions at all. There are randomised controlled trials conducted on us without our knowledge or consent, designed to determine which intervention will change our clinical behaviour the most.

Nevertheless, the rhetoric that changing proxy outcomes improves patient care shows remarkable resilience.

There are very few absolutes in general practice. There is a place for opiates, antibiotics, x-rays and pathology tests, and because GPs serve different populations, it is highly likely that appropriate care can still involve statistically high use of all of these interventions among some practitioners. Nevertheless, the statistical outliers are being nudged towards the mean with no measure at all of the impact on the patient, who is supposed to be the centre of our care.

The relentless criticism of general practice is dispiriting at best and harmful at worst, and the narrative that “GPs should”, “GPs don’t”, and “GPs fail to” causes the sort of occupational distress that decimates our already fragile workforce. Inappropriate outcome measurement can and does cause harm. The harm, of course, is not just to doctors. Changing our clinical behaviour changes patient care, which many of these campaigns do not measure either before or after the intervention.

Choosing proxy variables: the entrenched inequity within evidence

So where do proxy variables come from?

When we measure outcomes for individual diseases, we often rely on established evidence, so it’s important to understand what the limitations of that evidence is, and how that impacts on the populations we serve.

We know that much of the health research we have comes from studies that focus on white, middle class, urban men. If we imagine a patient in the centre of that evidence, we are likely to be considering a 40–50-year-old, relatively privileged man. Generalisability is problematic in non-representative trials for diseases such as pancreatic cancer, even though we assume the pancreas is the same from Bondi to Broome. It is a public health disaster for illnesses such as depression when we attempt to impose standardised “evidence-based” strategies on diverse populations that are excluded from the generation of the evidence.

Our guidelines, outcomes and indicators are often based on this narrow understanding of evidence. When I am explaining this to students, I often try to personalise the narrative by contrasting two representative patients: Norm and Maria. Norm, the middle aged, urban man, is the person who is at the centre of the evidence, because he is well represented in clinical trials, and so he will benefit most from the evidence the trial produces. In mental health research, Norm also tends to be literate, have no comorbid conditions, and sufficient capacity to remain in the trial for a period of time.

“Normism” in research is getting worse, with the problem of non-representation getting larger. One author in the US applied the exclusion criteria of a series of depression trials to an outpatient population, and found that “across all 158 studies, the percentage of patients that would have been excluded ranged from 44.4 to 99.8%”. The lack of representation in trials means indicators developed for Norm have less saliency and relevance to people like Maria.

Maria has multiple conditions to manage. She has diabetes, chronic obstructive pulmonary disease, and depression. She lives with the consequences of a lifetime of trauma. Her small town has few health resources, and she doesn’t have the resources to travel. She cares for her children; her husband, who is an alcoholic; and her elderly mother with dementia. To imagine we can measure her health, or the care her GP provides, with a simple depression inventory or a quality indicator such as the recording of her smoking status is disrespectful and almost meaningless.

Evidence from the UK suggests that the correlation between proxy measures and actual clinical performance is low and inconsistent, especially for patients like Maria. She is likely to have depression that is “difficult to treat” and doesn’t respond to standardised therapies, even if she had access to care and the resources to manage it.

Norm and his mates already live with health privilege, compared with the Marias of the world. Optimising their health is important, but on a population basis, they are not the people with the greatest need. We see this in our national statistics on mortality and morbidity, where the most privileged Australians continue to lead longer and healthier lives than those who have low socio-economic status.

For the GPs working at the deep end of disprivilege, measuring performance based on Norm’s indicators can be deeply misleading.

The surgeons and obstetricians know this. League tables are misleading because the best specialists see the most complex patients with the poorest potential outcomes. Performance measures in this setting can often drive exceptional doctors providing exceptional care out of complex clinical practice so they can manage risk. Even in business settings we know that metrics that are too narrow can privilege mediocre performance.

Want better outcomes? Choose easier patients

The logical response of GPs to the current climate is to stop seeing patients like Maria and focus on patients like Norm. Seeing patients like Norm means shorter consultations (which are rewarded by the Medicare structure), fewer interventions (which means fewer nudge letters) and less complexity (which is professionally less draining and less risky). The easiest way to do this is to build a Norm clinic, which attracts a segment of the population with similar needs.

Norm clinics focus on a narrow scope of practice and are being developed outside of general practice using multidisciplinary teams. These clinics can get satisfying outcomes on a range of metrics, improving effectiveness and efficiency by carving out the patients for whom performance indicators make sense. However, segmentation has unintended consequences for the patients left behind, the undervalued core of what GPs do.

Even if GPs continue to provide exceptional care, Norm’s indicators will be low in a non-Norm context, because they will be applied to patients who sit outside the core evidence. When we remove segments of straightforward patients from general practice, GP performance on standardised performance metrics falls.

At the moment, we have a perfect storm of incentives, designed to reduce care for the most vulnerable patients GPs see. This may not be the goal that policymakers would articulate, but it is the goal that the current health care system seems designed to create.

What outcomes should we measure?

We hear a lot about outcome-based funding, but increasingly, general practice is seeing the patients with needs that are too complex to be captured in standardised metrics. We have not yet implemented outcome-based funding in environments with less diverse populations and health needs, including hospital-based procedures. Interventions such as hip replacements have clear outcome measures, logical times to measure capacity before and after surgery, and relatively consistent cohorts of patients.

Until we can optimise outcome-based models for these segments of health care in the Australian context, it seems premature to implement scattergun outcome measures in the highly complex and diverse general practice environment.

Since we cancelled the funding for the Bettering the Evaluation and Care of Health research in 2016, we really have no idea what GPs actually do. Lots of people feel they could do what we do much more effectively and efficiently, but until we solve the great mystery of what happens during an item 23, this is all speculation – and speculation with conflicts of interest for high investment, standalone services.

We should work with general practice to clearly articulate the current standard of care before we start segmenting, fracturing and changing it. We know that what gets measured, gets done. We need to ensure we are measuring what is important.

We need to consider the paradox of primary care. GPs follow guidelines less than other disciplines, but on a population level they get better outcomes. We need to find out why. Because the corollary is if you measure adherence to guidelines as a measure of quality, you might reduce patient outcomes in general practice.

Outcome measurement is an intervention designed to change practice. Every intervention needs to articulate the impact on the patients left behind. Otherwise, Norm gets better outcomes, and Maria can potentially get significantly worse care. We are here to provide health care for all Australians, not just segments of interest. If the US has taught us anything, it is that subspecialisation is expensive, ineffective and grossly inequitable.

Complex health care systems privilege those with the literacy, health literacy and resources to manage it. Primary health care, particularly in mental health, is like a website full of dead links that are doors to nowhere. We need to measure this sort of health care waste. Some patients may want choice, but every time we offer choice, we increase complexity, and this can create systemic barriers to care for the patients with most needs we serve.

Finally, we need to consider the impact of each intervention on the workforce. Every time non-experts nudge us towards inappropriate standardised care, the performance of general practice falls and the demoralisation of the workforce rises. Losing general practice in communities that have few health care resources is a devastating outcome.

At the moment, outcome measurement in general practice is nudging us towards inequity, reduced performance for patients with complex needs, and a continued fall in workforce capacity. We need to reconsider whether this is the outcome that we want for the Australian community.

Louise Stone is a GP with clinical, research, teaching and policy expertise in mental health. She is Associate Professor in the Social Foundations of Medicine group at the Australian National University Medical School, and works in youth health.

The statements or opinions expressed in this article reflect the views of the authors and do not necessarily represent the official policy of the AMA, the MJA or InSight+ unless so stated.

Subscribe to the free InSight+ weekly newsletter here. It is available to all readers, not just registered medical practitioners.

 If you would like to submit an article for consideration, send a Word version to mjainsight-editor@ampco.com.au.