Understanding the brain: beware small samples

WHEN functional magnetic resonance imaging (fMRI) technology was developed in the early 1990s, hopes were high: the ability to look inside our working brains, to see how they fired in response to stimuli or when undertaking cognitive tasks would surely be a powerful clinical tool.

Potential benefits were touted in conditions as varied as autism, attention deficit hyperactivity disorder, chronic pain, addiction and a range of mental illness.

Since then, neuroscientists have used fMRI and other neuroimaging technologies to study everything from what makes us laugh to how we fall in love, but the hoped-for clinical benefits have largely remained elusive.

Neuroimaging has certainly become a useful presurgical tool, allowing surgeons to map an individual’s motor, language and memory brain function to help minimise damage to vital areas.

But the anticipated improvement in diagnosis and treatment of neurological and psychiatric conditions has not really materialised.

A recent article in Nature might help to explain why.

Lead author Scott Marek, a neuroscientist at Washington University, initiated the study believing it would demonstrate the reliability of conclusions from previous neuroimaging research.

To do that, he used data from one of his own previous studies that appeared to show a link between children’s brain function and performance in intelligence tests. Including more than 2000 children, the study offered an exceptionally large dataset for a neuroimaging study, a field where the median study size is around 25, according to Dr Marek.

He and colleagues split the data from their original study and analysed each half separately, confident the large sample size would ensure the two groups delivered consistent results. They didn’t.

“I thought it was going to look exactly the same in both sets,” Dr Marek told Nature. “I stared out of my apartment window in depression, taking in what it meant for the field.”

Concerned that even a study with 2000 participants might not be large enough to detect significant links between brain function and behaviour, the research group turned their attention to other published research in the field.

Although their analysis of other large neuroimaging studies, with 50,000 total participants, did reveal some replicable correlations with behavioural traits, those associations tended to be much weaker than those reported elsewhere.

Smaller studies delivered much stronger associations, they found from simulations, but these findings could rarely be replicated.

Even a large study with 2000 participants had only a 25% change of replication, while a more typical study of fewer than 500 participants had only around a 5% chance.

“Reproducibility requires samples with thousands of individuals,” they concluded.

Neuroimaging studies are expensive and time-consuming to conduct. It’s understandable that study sizes have traditionally been small. The problem is the sweeping conclusions that have sometimes been drawn on the basis of those small studies.

A decade ago, while writing a book on the science of sex and gender, I was astonished at the number of studies claiming to have identified fundamental differences between the sexes’ cognitive and emotional brain function based on studies conducted in a dozen or so people.

Study sizes have increased since then, but it seems not enough.

“There’s a lot of investigators who have committed their careers to doing the kind of science that this paper says is basically junk,” Professor Russell Poldrack, a neuroscientist and psychologist from Stanford University and one of the paper’s peer reviewers, told Nature. “It really forces a rethink.”

Not everybody agrees, and perhaps Dr Marek’s findings also require replication, but researchers in this or any field do well to be cautious about making big claims from small studies.

Our magnificent, plastic brains are too complex to be summed up in simple conclusions.

After all, as physicist Emerson Pugh famously said, if the human brain were so simple that we could understand it, we would be so simple that we could not.

Jane McCredie is a Sydney-based science and health writer.

The statements or opinions expressed in this article reflect the views of the authors and do not represent the official policy of the AMA, the MJA or InSight+ unless so stated.

Subscribe to the free InSight+ weekly newsletter here. It is available to all readers, not just registered medical practitioners.

 If you would like to submit an article for consideration, send a Word version to mjainsight-editor@ampco.com.au.