Is significance significant? (part II)
We must remove the word significance from our vocabulary in management reports. After our test two weeks ago, one thing is for sure: our audience will misunderstand us. And most likely, we don’t even know what we are actually saying.
Did you take our test two weeks ago? Based on an old study from Haller and Krauss*, we asked what you can derive from a statistically significant test result. Did you pass? Excellent! If not, you were at least in good company. Most people answered at least one of the questions as ‘correct’ and failed the test. Here are the failure rates:
| Methodology-instructors: | 80,0 % | (n=30) |
| Scientist non-teaching methods: | 89,7 % | (n=39) |
| Psychology students: | 100,0 % | (n=44) |
| Blog readers: | 71,1 % | (n=384) |
In the case of so many misinterpretations, significance is a problem and not a solution. Even instructors are affected.
In most cases, neither the author nor the reader would pass the test. Yet that doesn’t stop most people from claiming something – that could be statistically significant but not proven in a long shot – to be significant. We find proof in the media almost daily. If you want to see for yourself, simply use our small preconfigured Google search of a German newspaper on your left and one of the key words on your right:
Proving something statistically is just as desirable as turning invisible, gaining superhuman powers, or finding the fountain of youth – and as realistic as well. If we want to manage companies, we need to be levelheaded. So let’s recap what we have already discussed.
- Regardless if you use statistical means or not: Differences among groups are normal. Identical results are odd. A significant result says that the difference is not random. A non-significant result, however, says that the difference may still not be random.
- What the significance test doesn’t say is where the difference lies. It’s similar to correlations. The drop in the German stork population is similar to the drop in Germany’s birthrates. Whoever wants to see the connection, can do so – or just forget it.
- In the case of social, medical or business experiments, we can neither fully identify the influencing variables nor keep them constant. This is even difficult, albeit somewhat easier, in the case of technical experiments.
- Since variation is normal, significance tests only confirm that time and again. And since they cannot say anything about the true reasons, we can discard them once and for all without any second thoughts. Instead, we can just observe the distributions ourselves.
As a result, we should permanently erase these and similar phrases from our vocabulary:
- It is statistically proven (B is the result of A).
- The difference is significant.
- There is a 95 % probability that the new campaign will be effective.
- The relationship is statistically proven.
As far as management reports are concerned, we are already prepared. Statistically significant is only an attribute – and we avoid attributes already.
* Source: Haller, H., Krauss, S. (2002). Misinterpretations of Significance: A problem students share with their teachers? Methods of Psychological Research Online, Vol. 7, No. 1, P. 1–20.
