When is my result incorrect?

I’ve been talking today with a PhD student about some measurements he’s made in the lab. In physics, like all sciences, when we measure something we don’t just make one measurement, but we measure it several times. That way you get a more accurate result. Now, with most physical measurements, we expect there to be a single answer (e.g. the mass of an electron is what it is, we don’t expect it to be different tomorrow from what it is today; nor do we expect two different electrons to have different masses. ) Of course, every time we measure the mass, we’ll get a slightly different result, because of the uncertainties in our measurement, but we expect there to be a ‘true’ value.

My student’s been measuring a physical property of a biological system. In this case, there won’t be a true answer, since every specimen will be different. No two plants are identical, no two animals are identical. He’s ended up with data that goes something like this. Don’t worry what the measurement exactly is, or what the units are, it’s not important. (It’s very important to him, but not to this blog entry – that’s what I mean.)

0.308   0.327  0.291  0.356  0.328  6.23  0.310  0.451  0.299  0.320

Now, casting your eye over this one there seems to be an obvious problem with one of the data measurements. 6.23? I don’t think so. It’s nothing like the others. Something has most likely gone a bit awry when this measurement was made, and it most likely doesn’t belong in this set. Let’s throw it away.

But then we have another problem. What about the 0.451 measurement? That’s rather higher than the rest, but it’s not outrageously higher (or is it?). Could it be a dodgy measurement? Or is it just that this particular specimen exhibited our property particularly strongly. Can we throw it away? Hard to tell. And if we did, what about the 0.356 measurement? That’s a touch higher than typical, too.

Where do we draw the line with saying that one measurement is OK, but another isn’t.

I don’t think there’s a clear cut response to this one. But it serves to point out to the student that when you are collecting data, just think before you write. If we see something that is obviously way out (the 6.23) then query it. What’s gone wrong here. Can we repeat the measurement on the same specimen and get a better answer? In fact, in this experiment, it’s difficult to do a second measurement on the same specimen. So we need to think carefully about how we know when we’ve got a good measurement – i.e. what would give us confidence that it’s been done correctly?

That’s one of the subtleties of doing real science, on real systems – how you do a measurement, how you ensure it’s robust and means what you think it means, is really important. That often gets glossed over in the way we teach the subject. So we have some thinking to do here.