“Representative” research isn’t really

BACK in 2008, psychologist Jeffrey Arnett wrote a much-cited article criticising American psychology research for focusing almost entirely on just 5% of the human population (Americans) while claiming to reveal generalised truths about the whole of humanity.

His examination of research published in six leading psychology journals had found around 70% of authors and their sample populations were American, with almost all the remainder coming from other English-speaking countries or Europe.

The studied population was even narrower in one journal, where two-thirds of participants in American studies were undergraduate psychology students, prompting Dr Arnett to suggest a more accurate title for the publication might be Journal of the Personality and Social Psychology of American Undergraduate Introductory Psychology Students.

Enrolling students is a cheap and easy way to recruit subjects, but it’s hardly representative of even American society, let alone a broader range.

“No other science proceeds with such a narrow range of study,” Dr Arnett wrote of his findings.

“It is difficult to imagine that biologists, for example, would study a highly unusual 5% of the world’s crocodile population and assume the features of that 5% to be universal. It is even more difficult to imagine that such biologists would be aware that the other 95% of the world’s crocodile population was vastly different from the 5% under study, and highly diverse in habitat, eating habits, mating practices, and everyday behavior, yet show little or no interest in studying that 95% and continue to study the 5% exhaustively while making universal claims.”

Humans are not crocodiles, but there’s plenty of evidence a failure to include diverse populations in research does not lead to the best health care for those outside the fence.

When dermatology algorithms are based on research conducted predominantly in pale-skinned people, for example, they can fail those with darker skin.

The historical tendency to study predominantly male populations has led to poorer outcomes for women in conditions ranging from cancer, to cardiometabolic disease, to mental illness, as this MJA article highlights.

And, in psychology, it’s been suggested clinicians who brought American-style therapeutic interventions to Sri Lanka in the wake of the 2004 tsunami, without adequate consideration of local factors, may have done more harm than good.

In 2010, a group of Canadian researchers looked at comparative psychological data to examine just how valid those generalisations made on the basis of research in limited populations might be.

Not very, they concluded, coining the acronym WEIRD (Western, Educated, Industrialised, Rich and Democratic) to describe the most-studied group.

In domains ranging from visual perception, to analytic reasoning, to fairness and cooperation, they found members of WEIRD societies were “among the least representative populations one could find for generalizing about humans”. In fact, they were often outliers.

“We need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity,” they suggested.

Not much changed over the following decade, according to lead author Professor Joseph Henrich, despite the paper having been cited almost 10 000 times.

“On one level, I feel like there’s a lot more enthusiasm around addressing sample variability,” he told UNDARK magazine. “But if you actually look at the numbers, the latest numbers coming in the last few years don’t actually show any shift in the diversity of samples.”

A recent follow-up to Dr Arnett’s 2008 paper, co-authored by him and looking at the same journals, found there had been some change.

Over the intervening years, American representation had dropped to just over 60% of authors and samples, although the gap was largely filled by studies from other Western nations.

The proportion of the world’s population now represented in the journals may have risen from 5% to 11% as a result, but the majority of humanity is still being ignored, the authors wrote.

Improving representation of the other 89% is an “ethical, intellectual and professional imperative”, they argue, given the influence these journals wield in current practice and future directions in psychology.

There’s nothing wrong with studies conducted in specific populations, even American psychology undergraduates.

But, if we want to make grand claims about the whole of humanity, we’d better make sure they’re included in the process.

Jane McCredie is a health and science writer.

The statements or opinions expressed in this article reflect the views of the authors and do not represent the official policy of the AMA, the MJA or InSight+ unless so stated.