For those who rely on statistics (in particular t and p values) for their work, this recent Nature article may be of interest, Scientific method: Statistical errors. In it, Regina Nuzzo argues the fairly well-known statistical weakness of relying just on the p-value to support a
study’s statistical significance. A
stronger guidebook for researchers and policy makers is to further examine the
quality of one’s chosen null and alternative hypothesis distributions. And further examine the ex-ante choices on both
sampling part of the data (for replicatablity), and the choice of a 1-tail versus 2-tail tests. This is a better focus for analysts and researchers looking to strengthen their overall study approach.
The p-value works well in limited settings. But more often in the real world across business, policy, and science, we are dealt small sets of clumpy data, with no conviction on alternative hypotheses. This makes Regina's suggestion of pre-judging their likelihood highly improbable. The result is a push on one of a few statistical measures, such as the p-value, which is meaningless only in that it violates the underlying assumptions for using such a statistical test from one's initial research approach. It’s unfair after all to presume (as the Nature article suggests) an extreme likelihood occurrence, for why then are we performing a hypothesis test to begin with?
Incorrect distributions, which are not well sampled at the tails, paves the way for committing statistical errors of different varieties, and misdiagnosing the costly tail risks that an organization could face in their work. This applies to the p-value, or on the stronger “power" tests, which too are often applied in the same setting.
To solve the weaknesses identified here on this note, critical research modifications generally needs be done (this blog site covers many interesting applications of this), including generating larger data samples. Even if costly and time consuming. Other transformations involve taking a deeper look at the data, the costs of which usually come to a couple degrees of freedom. For example, focusing on the higher order moments, between the variables.
The p-value works well in limited settings. But more often in the real world across business, policy, and science, we are dealt small sets of clumpy data, with no conviction on alternative hypotheses. This makes Regina's suggestion of pre-judging their likelihood highly improbable. The result is a push on one of a few statistical measures, such as the p-value, which is meaningless only in that it violates the underlying assumptions for using such a statistical test from one's initial research approach. It’s unfair after all to presume (as the Nature article suggests) an extreme likelihood occurrence, for why then are we performing a hypothesis test to begin with?
Incorrect distributions, which are not well sampled at the tails, paves the way for committing statistical errors of different varieties, and misdiagnosing the costly tail risks that an organization could face in their work. This applies to the p-value, or on the stronger “power" tests, which too are often applied in the same setting.
To solve the weaknesses identified here on this note, critical research modifications generally needs be done (this blog site covers many interesting applications of this), including generating larger data samples. Even if costly and time consuming. Other transformations involve taking a deeper look at the data, the costs of which usually come to a couple degrees of freedom. For example, focusing on the higher order moments, between the variables.
Hello, Salil!
ReplyDeleteYour blog is very interesting (even for me as a Ph.D. candidate and a beginner in statistics and quantitative research). I'd like to add two resources on this post's topic, which might be of your interest. First is the online book by Alex Reinhart, called "Statistics Done Wrong" (http://www.statisticsdonewrong.com). Second is the work by Geoff Cumming on "new statistics", represented by his paper "The New Statistics: Why and How" (http://pss.sagepub.com/content/25/1/7). I hope that you will enjoy these resources and would be happy to read your opinion on them. Using this opportunity, I'd like to ask whether you could answer a couple of my questions, related to statistics, including sampling, factor analysis and SEM. Please let me know and, if 'Yes', I'll point you to the questions, as they're posted online.
Sincerely,
Aleksandr Blekh
Thanks much Aleksander. While I can not assist with your project, a number of similar-minded people to yourself are both reading this site and using some of the statistics forums on the internet. All from which you may benefit.
DeleteYou're very welcome, Salil. Fair enough - I already communicate with some people in statistical, SEM and data science circles, but thank you for the suggestion, anyway. Best wishes, Aleksandr.
Delete