Serving the Quantitative Finance Community

 
User avatar
katastrofa
Topic Author
Posts: 7930
Joined: August 16th, 2007, 5:36 am
Location: Event Horizon

Re: Move over p-values...

July 25th, 2019, 12:24 pm

Motivation p-values and standard null hypothesis testing have come under intense scrutiny in recent years (Wasserstein et al., 2016, Benjamin et al., 2018); s-values and safe tests offer several advantages. Most importantly, in contrast top-values, s-values behave excellently under optional continuation, the highly common practice in which the decision to perform additional tests partly depends on the outcome of previous tests. A second reason is their enhanced interpretability, and a third is their flexibility: s-values based on Fisherian, Neyman-Pearsonian and Bayes-Jeffreys’ testing philosophies all can be accommodated for. These three types of s-values can be freely combined, while preserving Type I error guarantees; at the same time, they keep a clear (monetary) interpretation even if one dismisses ‘significance’ altogether, as recently advocated by Amrhein et al. (2019).
If one considers Tukey's 1960's celebrated works recent... I don't want to be mean, but if they write in the paragraph "Motivation" about how great their method is, where's the paragraph "Our great method" which must contain the accidentally switched motivation?
 
User avatar
Cuchulainn
Posts: 22927
Joined: July 16th, 2004, 7:38 am

Re: Move over p-values...

July 25th, 2019, 5:53 pm

Here come S-values! https://arxiv.org/abs/1906.07801

It looks interesting - I will give it a read and comment further.
BTW which of the 300 authors do we write to if we have queries?
Image
 
User avatar
katastrofa
Topic Author
Posts: 7930
Joined: August 16th, 2007, 5:36 am
Location: Event Horizon

Re: Move over p-values...

August 12th, 2019, 11:09 pm

I have two remarks:

Obvious: the proposed method (the concept of "optional continuation") applies only to systems in equilibrium. In practice, everything changes in time because of drift, mutations, etc. (even changes change!). Ergo, you can rsrely go "let's make one more experiment". There are also all sorts of practical errors which accumulate: from measurement errors to the researchers getting "overfitted to themselves" (vide your last paper). Well, science is by design a one big overfitting contest...

Technical: why not simply use the existing false discovery rste method? The Bayesian version which suits here is common in some disciplines (in which the chance of a discovery is very low - p-values weren't made for such cases).
 
User avatar
katastrofa
Topic Author
Posts: 7930
Joined: August 16th, 2007, 5:36 am
Location: Event Horizon

Re: Move over p-values...

August 13th, 2019, 10:30 am

Am I right: their contribution is that they calculate the s-value distribution for the whole family of priors, while - in a standard Bayesian approach - optimising the test for power and controlling the type I error rate? It may be too complex to use in the real fuzzy practice (that's why researchers often resort to simplistic tests like p-values).
 
User avatar
katastrofa
Topic Author
Posts: 7930
Joined: August 16th, 2007, 5:36 am
Location: Event Horizon

Re: Move over p-values...

August 13th, 2019, 7:17 pm

That's more or less how I understand this work.
If you want more: q-value
 
User avatar
katastrofa
Topic Author
Posts: 7930
Joined: August 16th, 2007, 5:36 am
Location: Event Horizon

Re: Move over p-values...

August 16th, 2019, 9:57 pm

Yes. They are often tested for different assumed probabilities of H0 and compared against p-values. It tells the researcher what is the expected the false discovery rate for the discovery which is significant according to the p-value. All those tests are tailored to specific problems (q-values are used in genome analysis).
 
User avatar
katastrofa
Topic Author
Posts: 7930
Joined: August 16th, 2007, 5:36 am
Location: Event Horizon

Re: Move over p-values...

August 17th, 2019, 12:04 am

Image