This afternoon I’ve been discussing with a PhD student a question that is really at the heart of the scientific method. He’s measuring something in the lab that is a bit variable. Everytime he takes a reading of Y it is a little bit different. Essentially, he wants to know that if he does X to his experiment, do his results Y change? (In other words, does changing X produce changes in Y? And, deeper, HOW does it cause Y to change.)
His question to me was ‘how many measurements should I take?’ It is a very interesting one. Whatever we measure, we’ll find there is always variation in it. Some things exhibit really large variation, in others the variation can be so small it is below your ability to measure it. There is a huge raft of statistical techniques that can be called on to help you describe these variations and what they mean. But, essentially, it boils down to this: The more measurements you take, the better you will know the average (mean) result. More specifically, if you want to halve your uncertainty (i.e. be twice as certain about something), you need to take four times the number of measurements. This means that the number of measurements you require to show something can often be very large indeed.
Just how many measurements you should take is governed by the size of the effect you are trying to demonstrate. If the change that X is likely to have on Y is very pronounced, you will only need a few measurements before you can see it. But if the change is really small, a lot of measurements are called for.
In some cases we might have a good idea of what the size of the effect is likely to be (e.g. someone has done a theoretical calculation on it) and so we can design an experiment accordingly. But often we have only a vague idea. So we’ll do a few trial measurements, and see what results are turning up. And then we’ll do a few more. Hopefully we’ll then get some clue as to the magnitude of the effect we have (or don’t have, as the case may be), and this will inform a decision on how many measurements to take when we (by which I mean my student) do this ‘for real’.
After all that, however, we still won’t be able to say anything for certain. Statistics in the physics context is about quantifying your uncertainty. Uncertainty will never go away – even if your effect is very pronounced you could always say that there is a smidgen of a chance that this is just due to statistical variation, and that, if you did the experiments again, you’d get a different result. But we can quantify that smidgen, and, if it’s small enough (again, how small is small enough?…5%, 1% are just arbitrary choices…so nothing is absolutely proven beyond any doubt whatsover) we can say with some confidence that X does have an effect on Y.