As regular readers of this blog will know, I’m a great believer that those of us who are interested in evidence-based education can learn much from the experience of colleagues working in the field of evidence-based medicine.  So I was delighted to have recently come across Evans et al’s (2011) book Testing Treatments: Better research for better healthcare which set out to explore how we ensure that medical treatments best meets the needs of patients.  In particular it looks at how to ensure ‘that research is soundly based, properly done, able to distinguish harmful from helpful treatments, and designed to answer questions that matter to patients, the public, and health professionals.

What I like about the book was how it was upfront about the limitations of research i.e. all it can really do is help us become a little less uncertain about how to proceed.  High quality research can indicate the probability that a particular treatment will lead benefits or harms for the patients, and which is equally true of research into interventions within education.    

Helpfully, at the end of each chapter Evans et al come up with a list key points to think about, which for me, many of which equally applicable in education.  Below is an edited list  key points, which have been amended for educational settings.

• Testing new teaching strategies/interventions is important because new teaching strategies are as likely to be worse as they are to be better than existing teaching strategies

• Biased and unfair tests of interventions can leads to pupils losing out and having reduced life chances

• Just because a major body recommends the approach, does not mean it might not harm pupils

• The beneficial effects of interventions are often overplayed, and the harmful effects underplayed

• Neither theory or professional opinion is a reliable guide to effective interventions

• Just because an intervention or practice has been used for years does not mean that it is effective

• Even if pupils are not ‘harmed’ by the intervention, using them is a waste of resources

• More intensive use of an intervention is not necessarily beneficial, and can sometimes do more harm than good

• Better diagnosis does not necessarily lead to better outcomes, sometimes it makes matters worse

• Screening programmes should only be introduced on the basis of sound evidence of their effects

• Dramatic effects of an intervention are rare

• Uncertainties about the effects of an intervention are common

• Fair tests of interventions are needed because we will not otherwise sometimes conclude that interventions are useful, when they are not, and vice versa

• Comparisons are fundamental to all fair tests of treatments

• A single study rarely provides enough evidence to guide intervention choices in education

• Assessments of the relative merits of alternative interventions, should be based on systematic reviews of all the relevant, reliable evidence

• New research should only proceed if an up-to-date review of earlier research shows that it is necessary

• Much research is of poor quality and done for questionable reasons

• Input from teachers, pupils and school stakeholders can lead to better research

 Of course, this is not an exhaustive list of what needs to be done to improve the quality of and usefulness of educational research. On the other hand, will help you become more sceptical about some of the claims made about educational interventions. And by being sceptical, I’m not talking about someone who is a continual ‘naysayer’ but rather someone who withholds approval or disapproval until they have made appropriate and rigorous inquiries. 

Reference

Evans, I., Thornton, H., Chalmers, I., & Glasziou, P. (2011). Testing treatments: better research for better healthcare. Pinter & Martin Publishers