In this article –
What a significant result means
In my last blog about research we looked at the fact that it’s almost impossible to prove something. Even if you thought you had proved it quite likely a new method of research or a way of seeing things or new/better data will likely come along eventually and either change what we think about something or allow us to see it in a new and hopefully better way. So the best we can say about any research is that the findings are the best, most up-to-date findings that we have right now. We always have to be open to the idea that later research may add to, change or even completely overturn our understanding of the topic under consideration.
Be impressively well informed
Get the very latest research intelligence briefings, videos and infographics sent direct to you
This is why only the uninformed talk about research proving things. Researchers and academics understand that you can only show that something isn’t the case, for example that feeding your employees chocolate causes customers to buy more of your product. We accept a hypothesis based on the fact that the best evidence at the moment shows that the opposite, the null hypothesis, can’t be true (that feeding your employees chocolate does not increase product sales).
So research shows you the best evidence to support something at the moment or what theory to explain something best fits the available data right now.
The term significant to a researcher means how likely are we to have got this result by chance?
Say some researchers found that 50% of people in America believe in Santa Claus. Is that high or low or are you likely to get that result by random chance?
Ok, the first thing is that given the fact that there are somewhere over 318 million people in America, the researchers are very unlikely to have asked and got a response from every person in America. So they will have sampled a range of the population. The question is, given that it is a sample of the population, how likely are these results to occur just by chance, given the size of the sample and how representative that sample is of the population generally?
There are a series of statistical methods (which I won’t go into now) that can give us an answer to that. If, say, you only asked two people, then not only would this not represent the entire population of the US, but any result you get could easily happen due to random change. The results would be described as not significant. The result is not significant because if you asked just two people, the chances are that you would get almost any result anyway. It doesn’t really tell us anything. However if you sampled a large enough percentage of the population controlling for (or taking into consideration) the age, education, socio-economic status and other characteristics so that the sample truly represented the population of the US, then you might find that the chances of getting a result of 50% believers by random chance is less than 95%. In other words there is only a 5% likelihood that you could get this result by random chance. This result could then be described as significant. This doesn’t mean it is proved, it means that you are only 5% likely to get this result by random chance.
In fact 95% is the usual minimum level researchers use as evidence that their results are significant. There tends to be two cut off points for research, we are 95% sure that the results are what we think they are and 99%. These are called levels of significance and obviously being 99% sure the results are not as a result of random chance is rather better than being 95% sure.
Be impressively well informed
Get the very latest research intelligence briefings, video research briefings, infographics and more sent direct to you as they are published
Be the most impressively well-informed and up-to-date person around...