misleading medical-science stories – & why most published research findings are false

It’s been pointed out to me that this post could be construed as dissing the EPP drug & its producers. This was not in any way my intention – the post was about the fairly poor reporting around a couple of medical science stories, ending with a wish that science journalists were more aware of the fact that much medical research tends (for a whole lot of perfectly good reasons!) to be ‘wrong’. (Hopefully this awareness would be followed by a decline in the hyperbolic reports about new breakthroughs & discoveries…)

Two stories in this morning’s Herald caught my eye – & made me sigh. One announced the imminent release of a ‘UV vaccine’, & the other elevated another fruit to the level of ‘superfood’. Both were examples of how not to present research findings to the public.

The first (originally from the UK paper The Telegraph) carries the headline UV protection drug prevents skin cancer & the first paragraph tells readers that [an] anti-sunburn drug that protects even the fairest-skinned people from skin cancer and ultra-violet rays could soon be made available to the public. It’s difficult to assess all the claims made in the article because, unfortunately, the actual research study on which this story appears to be based has yet to be published. However, there are a number of statements in the story that set my teeth on edge (beginning with ‘the results [of the study] will be published soon’).

The drug in question (afamelanotide) has been developed by an Australian company to treat people with a condition called EPP, or  erythropoietic protoporphyria. Porphyrins are chemicals involved in a number of important biochemical processes within the body, including the production of red blood cells. The metabolic pathway your body uses to produce porphyrins can be interrupted by any one of several different mutuations, with the result that intermediate chemicals build up in the body. In the case of EPP (a rare disorder that affects less than 1 in 200,000), a mutation sees an excess of the chemical protoporphyrin in patients’ blood & skin – & in the skin it absorbs and reacts with light to give an extremely painful burning sensation. Individuals with the most severe (& extremely rare) form, congenital EPP, can’t tolerate any natural light, as their skin blisters & burns on exposure. Not nice at all.

It sounds as if the drug – which has just gone through a 12-month trial in EPP patients – acts to stimulate eumelanin production in the skin (not ‘between light and a person’s skin’, as the newspaper story would have it), providing a protective barrier against the worst of the sunlight. However, I couldn’t find anything in the company’s own available documents to suggest that it might also protect against skin cancer – and anyway, while skin cancer is more common in fair-skinned individuals, it does still affect those with more heavily pigmented skins. (A tan is not a 100% guarantee of immunity from skin cancer.) For example, the frequency of melanoma in African Americans is 1 in 100,000 (compared to 22 in 100,000 for US Caucasians). In any event, if afamelanotide did ‘prevent’ skin cancer, then you might expect to see it licensed  approved for this use – but the US FDA has approved it for treatment of further clinical trials in people with EPP, & this is how the company concerned sees it being used: [it] is proposed that afamelanotide will assist those patients who are most severely affected by UV and light, with the aim of improving their quality of life.

So the absolute statement that the new drug ‘prevents skin cancer’ is misleading, at the least. And while the drug may well reduce the rates of skin cancer in EPP patients, where any sun exposure is damaging, there is as yet nothing to demonstrate that it would have the same protective effect in the wider population. Nor should it be marketed to that wider population, as it sounds as if there is the potential for damage to immune & cardiovascular systems unless use of the drug is carefully monitored. Unfortunately, this information comes at the end of the news story, which otherwise hypes up the new pharmaceutical to the population at large.

The other story (also not yet on the Herald website, but carried in The Guardian in the UK) is called Fruit could be answer to hospital superbug. The latest addition to the ‘superfood’ pantheon is the pomegranate (which must be something of an acquired taste – one which so far I have yet to gain). We’re told that [s]cientists have discovered the power of fruit as a potential new weapon in the fight against MRSA and other hospital superbugs. Now, methicillin-resistant Staphylococcus aureus (MRSA) is a real nasty, resistant to methicillin & other related penicillin-type antibiotics, so the finding that a novel treatment may be available is cause for cautious optimism. And apparently a combination of pomegranate rind, vitamin C, & a metal salt ‘greatly enhanced [the] infection-fighting properties’ of this particular fruit. But not against Staph.aureus – the study on which this report seems to be based used the a multidrug-resistant strain of the bacterium Pseudomonas aeruginosa (Gould et al. 2009). A picky point, but how hard is it to be accurate? Would this detract from the story?

The article quotes one of the research team as saying that "It was the mix that fantastically increased the activity – there was synergy, where the combined effects were much greater than those exhibited by individual components…"  And then goes on to say that The tests were conducted using microbes taken from hospital patients. Scientists found that pomegranate rind mixed with metal salts was most effective against MRSA, while adding vitamin C helped tackle other common hospital infections.

But – and there are at least a couple of big ‘buts’ here… How well does this concoction compare to other currently available drugs (like vancomycin) that are still effective against most strains of MRSA (or of P.aeruginosa)? The story doesn’t tell us; the way it reads, the study looked at pomegranate rind, alone & variously combined with the other two ingredients. This is borne out by the abstract of the research paper itself, in which the authors conclude that [t]hese results warrant further investigation of PRE as a potential source of new antimicrobial agents. And – we don’t actually know if the pomegranate mixture would have the same antimicrobial effects in humans. Why not? Because these preliminary trials were in vitro ie on bacteria growing on agar in petri dishes. (To be fair, the article does – eventually – get around to pointing this out.) There’s a very long way to go indeed before this one makes it into regular use in hospital care – something the authors of the paper were careful to say, but which the article’s headline, & its first several paragraphs, gloss over.

All this makes me wonder whether a paper by John Ioannidis – published 4 years ago & freely available on-line – ought to be required reading for journalists writing stories about medical ‘breakthroughs’. Writing in PLoS Medicine, Ioannidis makes the point that in the field of medical research, ‘most published research findings are false’. (I need to emphasise here that he is talking about medical research – his essay has been used elsewhere in attemtps to support claims that most research in all fields of science is false, which is not necessarily the case.)

Ioannidis begins by noting that [p]ublished research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. No surprises here, to anyone working in science; this simply reflects how science operates. Hypotheses are tested, & tested again, & it’s by no means out of the ordinary for an hypothesis to be overturned as a result of this rigorous evaluation. Unfortunately I suspect many non-scientists find this confusing. Having been sold the line that science gives us ‘the truth’; people can’t really be blamed for thinking, well, if scientists were wrong about this, what else are they wrong about? (They can be blamed if they abuse this misunderstanding in others to promote their own particular agenda, however.)

Anyway, Ioannidis goes on to point out that a number of factors can influence the production (& dissemination) of research findings that are false ie not a good reflection of what is act ually going on in the biomedical system being studied. One of these is bias: the combination of various design, data, analysis and presentation factors that tend to produce researh findings when they should not be produced. (This can also at times obscure ‘true’ research findings, by the way.) Inefficient use of data, conflicts of interest (potentially a real problem with commercial development of pharmaceutical products), and a failure to notice when findings are statistically significant, can all have an impact on how research findings are presented.

There’s also an unfortunate tendency to trumpet findings by single teams of researchers, viewing their work in isolation at a time when there may be several different teams investigating the same topic. This can be exacerbated by pressure to publish & to get results out there. As Ioannidis says, [w]ith many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritise on pursuing and disseminating its most impressive ‘positive’ results.  In addition, size of the study (or size of its sample population) is important – studies with only a few patients involved may show a ‘positive’ outcome when in fact the apparent results are due to chance alone (or to some other factor that the design of the study failed to take into consideration).

Required reading? I think so – but I know it’s not likely to happen any time soon. "Gee whiz" & "latest scientific/medical/technological breakthrough" stories are perceived as more attractive to readers, & for whatever reason scientific accuracy seems to take a back seat. Which is a real pity, as it distorts the public view of what science is & how it operates, at a time when more than ever we need people to have at least some understanding of the nature & value of science.

Sigh.

S.W.Gould, M.D.Fielder, A.F.Kelly, W.El Sankary & D.P.Naughton (2009) Antimicrobial pomegranate rind extracts: enhancement by Cu(II) and vitamin C combinations against clinical isolates of Pseudomonas aeruginosa. British Journal of Biomedical Science 66(3): 129-132

J.P.A.Ioannidis (2005) Why most published research findings are false. PLos Medicine 2(8): e124. doi: 10.1371/journal.pmed.0020124