Misunderstanding and misrepresentation: a reply to Hutchison and Schagen

Publication Summary

In this journal, I previously published a paper discussing how to conduct an analysis based on a cluster sample. In that paper, I outlined several widely adopted alternative approaches, and pointed out that such approaches are anyway not needed for population figures, and not possible for non-probability samples (Gorard 2007). Thus, I queried the prevalence and relevance of complex approaches such as multi-level modelling (MLM) for most real-life analyses. I illustrated the lack of substantive difference in the results obtained via MLM and other approaches in a number of recent studies. And I repeated my challenge (from Gorard 2003a) for those advocating the use of MLM to find and publish a real study in which the policy or practice implications of the research would differ depending on whether MLM or a sensible alternative was used. Instead of taking up this challenge, Hutchison and Schagen (2008) responded with a paper that misrepresents (and even misquotes) what I said, in an attempt to defend their favoured technique of MLM. In my reply to them here, I hope to establish their misunderstanding of this important topic. This might lead readers to imagine that this will be a highly technical paper about a statistical issue. It is not. I also hope to establish their misrepresentation of my paper. The issue is one of wider and greater concern than the surface argument about whether MLM is of any great value. It is about the kind of intellectual processes we are prepared to endorse as a field.1 This paper deals first with areas in which we agree, and then areas in which Hutchison and Schagen (2008) simply restate their position, before turning to the more substantive consideration of why it is not necessary or appropriate to use MLM in most real analytical situations.