Presentation: Decision-Theoretic Approach to Post-Hoc Test Assessment
Speaker: Chloe Krakauer, Graduate Student, UW Biostatistics
Abstract: After conducting significance tests, researchers often (and perhaps should) wonder how much faith may be placed in the inferential result. In particular, in the case of an insignificant test, do the data suggest lack of evidence to reject the null or strong evidence in favor of the null hypothesis? Similar questions persist with significant results. After the value of post-hoc power assessments were disproven in the early aughts, some methods to quantify strength of evidence have been suggested. However, most suffer from poor interpretability, and with the exception of reporting confidence intervals, none have gained wide popularity. Our method to quantify this evidence starts with a Bayes’ rule resulting from a loss function that retains the interpretation and approximate inference of a frequentist significance test when flat priors are used (with additional flexibility to allow for more informative priors). In this talk, we explore the ability of data to aid in estimating the loss or risk incurred by a significance test, and therefore the credibility of the inference.