Here’s a really interesting quote from Ryan, commenting over on Orac’s blog:
Sometime in the 60s, education in America started putting a greater emphasis on skepticism. For the first time, kids were encouraged to question what they were told. This is a good thing, or course. But I wonder – have we been encouraging people to distrust and challenge authoritative voices without properly equipping them to properly judge authoritative voices?
Orac speaks authoritatively and quotes research papers as his sources. The anti-vax movement also has authoritative-sounding voices who are able to quote research papers: http://www.generationrescue.org/studies.html
How are people from outside the medical and scientific community supposed to parse this? At what point in High School were we supposed to have been taught how to properly evaluate the findings in research papers? I don’t recall that chapter. I was made to memorize the periodic table, but I don’t recall that anyone ever taught me how to tell the difference between good science and bad science.
It’s a good point, isn’t it? Just how well do we teach students these things – to tell the difference between good science & bad science?
Well, the short answer is that at school, anyway, we probably don’t do a particularly good job of teaching students to assess scientific papers. (Not intended as a criticism of teachers, by the way, & I did say ‘we’, having been there myself! But it’s just that there are so many other things that place demands on classroom time.) And maybe that’s too narrow a focus – perhaps we should be teaching how to assess scientiific claims, or rather, statements that claim some level of scientific authority.
And there are ways of doing this – tools we can use, things to remember when reading or hearing about something that supposedly has scientific validity or the support of scientists. One place to go is the ‘Bogus Science’ page, which discusses 7 signs that something might instead be bogus:
- The claim is first published in the media, rather than in a scientific journal (rather like that ‘gender-&-voting-preferences’ study I wrote about earlier this week).
- It’s claimed that this wondrous discovery is being actively suppressed by the scientific establishment – take the Discovery Institute’s claims about bullying of scientists who promote ‘intelligent design’ as an example; another would be the attitude of alt-health practitioners…
- The effect that’s being described is almost impossible to detect – & may only show up through some particularly inventive use of statistics.
- The supposed data are in fact anecdotal (& you know the value of this).
- The claimed effect is said to be valid because it’s been around for centuries. (A lot of ‘alternative medicine’ falls into this camp.)
- The person making the claims has worked in isolation on his/her great discovery (relates back to the second point). But this is rare – most scientists work collaboratively with others.
- And they propose some new law of nature to account for their observations. Unfortunately these laws conflict with everything else we know about the way the world works, & so are almost certainly wrong. (Hahnemann’s ‘Laws’ related to homepathy are a case in point.)
And not completely at a tangent – you might also enjoy this post on Pharyngula 🙂 – on interpreting graphs, assessing the accuracy of calculations, & our sources of information.
Of course, the graph that PZ’s discussing is essentially worthless – no more than pretty colours on the screen.
The version below is equally useful/useless (& thanks to Blake Stacey!) – & perhaps says more about ‘popular’ sources of information than the one it’s spoofing 🙂
So don’t be fooled by lots of pretty colours on a graph!