01/07/2016 Mathematics Psychology
DOI: 10.1080/13645579.2015.1091235 SemanticScholar ID: 55123978 MAG: 1868623645

What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’

Publication Summary

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The ‘number needed to disturb’ a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or ‘effect’ size disappears, minus the number of cases missing key values. This way of presenting the security of findings has several advantages over the use of significance tests, effect sizes and confidence intervals. It is not predicated on random sampling, full response or any specific distribution of data. It bundles together the sample size, magnitude of the finding and the level of attrition in a way that is standardised and therefore comparable between studies.

CAER Authors

Avatar Image for Stephen Gorard

Prof. Stephen Gorard

University of Durham - Professor in the School of Education

Share this

Next publication

2009 Psychology

The Dynamics of Category Conjunctions

R. Hutter, R. Crisp, G. Humphreys, Gillian. M. Waters + 1 more