Stopping rules in NHST and Bayes

(Sorry  – I wrote this post after our last BayesClub meeting, but I forgot to actually post it! Better late than never, I hope.)

Let’s say you’re collecting data and you check on your results every once in a while as the data are coming in. You decide to stop collecting data once you see that the results look “good”, and you report your findings using a standard NHST test.  Then the Stats Police break down your door, handcuff you, and throw you in the back of an armored van that takes you to Stats Prison. A common tale, but true.

But what happens if you’re analyzing your data using Bayesian inference instead of NHST? We know that multiple comparisons aren’t a problem when you’re dealing with a posterior distribution instead of running NHST tests, so maybe it’s also okay to run your analyses multiple times and then stop collecting data whenever you’re satisfied. 

Turns out, no. If your stopping rule is based on your results, your results will be biased, no matter what flavor stats you’re running. 

http://datacolada.org/2014/01/13/13-posterior-hacking/

One thing to note is that Simonsohn is using the Bayes Factor to get a result analogous to a significance test in his Bayesian analyses, which operates differently from the HDI and ROPE method we’ve been reading about in Kruschke’s work. Kruschke actually has a blog post on this as well, in which he discusses stopping rules with NHST, Bayes Factors, and HDIs in more detail:

http://doingbayesiandataanalysis.blogspot.com/2013/11/optional-stopping-in-data-collection-p.html