Best practice in conducting RCTs: Lessons learnt from an independent evaluation of the Response-to-Intervention programme

Publication Summary

This paper presents the findings of the first independent UK evaluation of a large-scale randomised controlled trial of Response-to-Intervention, used as a catch-up literacy intervention. A total of 385 pupils in their final year of primary school (Year 6) were involved in the study (181 treatment and 204 control). These were identified as those at-risk of not achieving the threshold Level 4 in English at Key Stage 2. The pupils came from 49 schools across England. Twenty-seven schools were randomised to receive treatment immediately and 22 schools, which formed the control, were randomised to receive the intervention later. RTI was delivered in the summer term in preparation for pupil transfer to secondary school. The overall impact based on the standardised New Group Reading Test (NGRT) showed an ‘effect’ size of +0.19, and of +0.48 when considering only free school meal eligible pupils. However, these results must be viewed with considerable caution given the high attrition (over 25%) especially from the control group, and unclear identification of pupils eligible for the intervention. The fact that the evaluators did not have direct contact with schools when trying to identify eligible pupils, and that the developers wanted to use the pre-test to identify eligible pupils, led to this being a weak trial. The significance of the work therefore lies at least as much in the lessons learnt as in the headline figures. We learnt that ideally no more than two parties should be involved in communicating with schools, so that relevant instructions are passed quickly and acted upon promptly. This helps minimise the risk of misunderstanding and dropout post-allocation. Prior training on the technicalities of trials and research in general is necessary for both developers and any staff delivering the intervention so that all parties involved understand their commitment and the need to provide accurate and complete data. In future trials of RTI, it would be better for individual eligible pupils to be randomised rather than schools. RTI should ideally be given a whole year to allow the full cycle to be implemented, and it should be delivered daily for at least 30 minutes.