Teacher and school-led research - some ethical considerations

This is the second in a series of posts that looks at the issue of ethics within the context of teacher-led and school-led research.  In the first post I am examined senior leaders’  lack of awareness of issues pertaining to research ethics.  In this post, I will pose some questions about ethics and teacher/school led research and provide some tentative answers. So here goes.

1.     Do I need to take into account ethical issues when conducting teacher/school led research?

Yes – as a professional educator you have an obligation to conduct research in an ethical manner, Robinson and Lai (2005).

2.     Where can I gain guidance on ethical issues?

The best places to start are BERA’s Ethical Guidelines for Educational Research - BERA (2018) alongside the code of conduct of your professional association

3.     Do I need to get ethical approval to conduct teacher/school research?

Maybe – if the research is being conducted as part of a formal programme of study or in partnership with a higher education institutions, then in all likelihood you will require ethical approval from the relevant ‘ethics committee.’ If not, it depends upon the activity you are undertaking and the procedures within your school.

4.     If I – or the school - are undertaking research outside of the auspices of a HEI are there occasions where it would make sense for the research to be subject to some appropriate form of internal approval process?

Yes – the following activities are likely to be deemed ‘research’ and should have some form of approval

·      the testing or development of hypotheses 

·      comparing interventions 

·      collecting quantitative and qualitative data over and above data generally collected within the school - though this could include normally collected data 

·      significant change to teaching approaches/support strategies - which are in addition to what is normally provided for pupils.

·      generalising findings beyond the setting of the school

5.     Are there any activities which are not going to require ethical approval?

·      the evaluation of existing provision

·      performance reviews, and testing within normal educational requirements if there is no research question involved (used exclusively for assessment, management or improvement purposes).

·      inquiries based on the review of previously published research literature

·      research based solely on the researchers personal reflections and self-observation

6.     Can I rely on ‘equipoise’ and ‘informed consent’ to provide sufficient ethical protection for participants?

No - other matters to be considered include: transparency; the right to withdraw; potential harm from participation in the research; privacy and data storage

7.     Is the quality of the research undertaken an ethical issue?

Yes -  Gorard (2002).   No matter what ethical guidelines are put in place – if the research is poor quality – participants are being disturbed, schools are wasting their money, and teacher researchers will be wasting their time.  What’s more given the ease with which research findings can these days can be disseminated via social media or the plethora of conferences the research may lead to teachers and schools being misled, Coe (2008)

And finally

Ethical concerns are not limited to research but should also form an integral part of  discussions school improvement, quality improvement and evidence-based practice.  If you are only discussing ethical issues in the context of teacher or school-led research, then hopefully this blog post will give you pause for thought.  

References

BERA. (2018). Ethical Guidelines for  Educational Research (4th Edition). London. British Educational Research Association

Coe, R. (2008). Reflections on Ethics in Educational Research (Draft). University of Durham.

Gorard, S. (2002). Ethics and Equity: Pursuing the Perspective of Non- Participants. Social Research Update, 39 (Winter 2002). University of Surrey Social Science Research Update - Univeristy of Surrey. 39 Winter 2002.

McNiff, J. (2017). Action Research: All You Need to Know. London. SAGE

Robinson, V. and Lai, M. (2005). Practitioner Research for Educators: A Guide to Improving Classrooms and Schools. Thousand Oaks, CA. Corwin Press.

Are research active schools acting ethically?

A recent conference organised by the Institute for Effective Education - Improving outcomes for children: How do we know we’re succeeding? – covered a range of research and evaluation being carried out in schools. Everything from small-scale evaluations run by schools, where schools were testing innovations they had come up with, to large EEF trials, where schools were scaling up interventions with promise. It prompted me to think about the ethical framework within which these projects operate.  In one session, when I asked what ethical framework and processes the school used before and during the conduct of a randomised controlled trial, the response was that ‘equipoise’ was assumed, parents gave consent, but there was no formal approval process to look at the ethical issues before carrying out a trial.

Now I could go into a long discussion about equipoise and informed consent  But for want of a better phrase – for me equipoise is the research equivalent to a ‘get of jail card’.  In other words, we don’t know what works, which gives us an excuse to do pretty much what we like.  It’s useful for schools but does not provide sufficient ethical protection to pupils.  As for informed consent – this is quite rightly something that needs to be taken into account – but the latest British Educational Research Association ethical guidelines for educational research BERA (2018) identify a far greater range of ethical issues which need to be addressed.

Now to be fair – a comment made at the end of a session does not provide incontrovertible evidence to support my claim that research active schools need to give more thought to the ethical component of their work.    So with this in mind, it seem sensible look into the available research on school leaders views on research ethics.  Fortunately, I came across some recent research by Bryan and Burstow (2017) who engaged with 25 school leaders to explore how the ways in research-active schools were aware or using ethical guidance in their research practices. 

Bryan and Burstow undertook an initial online survey of 520 contacts – with 44 responses (8.5%) – and found some confirmation of their hypothesis that there was an issue around the  ethical awareness of school researchers

·      35.9 % of respondents indicating that they always take time to consider ethical issues

·      30.8% of respondents indicating they havent had research ethics at the forefron of their planning

·      20.5% of respondents considering themselves as part of an ethically correct organisaitons – sufficiently informed and demonstrating good practice.

·      12.8% of respondents indicating that they always take full considerations of any ethical issues and use published guidelines.

Bryan and Burstow then went onto interview 25 senior leaders – drawn from primary and secondary schools who managed professional learning within their schools and had research projects taking place and subsequenly identified six themes

1.     Ethics were not on the radar

2.     Schools were seen as moral high grounds – which were not going to do projects which potentially harmed children.

3.     Informed consent and right to withdraw – interestingly some respondents felt that children could not refuse to do something, with an expectation that everyone would take part.

4.     A rejection of anonymity – that staff teachers would get back to specific pupils if they felt there was a need to.

5.     Parental permission – a general expecation that parents would be ok with their children participating in reserch.

6.     Concerns about workload and how research might contribute to overloading teachers.

Bryan and Burstow then go onto to make three observations. First, none of the respondents from 25 schools involved in the interviews had engaged in the BERA guidelines, and the appeared to have limited understanding of the importance of quality, rigour and trustwortiness in research.  Second, there was a general dismissal of informed consents, of the right to withdraw or say no.  Third, teachers saw little difference between research and what they did in classrooms on a day to day basis.  Four, senior leaders in the schools had a rich understanding of how research within their schools was ethically situated – which went beyond a set of bureaucratic guidelines – with research practices aligned to pedagogic practices.

So where does this leave me and my claim that research active schools maybe acting unethically.  On reflection,– research active schools are probably not acting unethically, that said, my sense is that they are not acting in ways which could be considered as best-practice (see Stutchbury, K. and Fox, A. 2009). As Bryan and Burstow note – there is a need for a strategic approach to the issue of research ethics within schools.  The EER/IEE and Research Schools are well placed to do this – but I am not sure whether this is currently being done  That said, this is not about creating a bureaucratic approval process, instead it’s about getting school leaders to explicitly think about the ethical issues of any research undertaken.  This is particularly important, given what  Zhao (2018) has to say about the potential negative side-effects of educational interventions.

And finally, given the publication of this year’s OfSTED Chief Inspector’s Report and the comments made about off-rolling of pupils, unfortunately we cannot make the assumption that all schools are moral-high grounds and and that school leaders provide acceptable levels of ethical leadership and are appropriate environments, within which school-led research should be taking place.

References 

BERA (2018). Ethical Guidelines for Educational Research (Fourth Edition). London. British Educational Research Association

Bryan, H. and Burstow, B. (2017). Leaders’ Views on the Values of School-Based Research: Contemporary Themes and Issues. Professional Development in Education. 43. 5. 692-708.

Stutchbury, K. and Fox, A. (2009) Ethics in Educational Research: introducing a methodological tool for effective ethical analysis, Cambridge Journal of Education, 39 (4), 489-504

Zhao, Y. (2018). What Works May Hurt—Side Effects in Education. New York. Teachers College Press.

The school research lead: meta-analysis, effect sizes and the leaking ship

Over the last few weeks I have given considerable thought to the usefulness to school leaders of both effect sizes and meta-analyses, as they try to bring about improvement in their schools.  On the one hand, there is the view at effect sizes and meta-analysis have major limitations but remain the best we have, Coe (2018).  On the other hand, there is the view that effect sizes and meta-analysis are fundamentally wrong and do not represent the best that we have and there are viable alternatives, Simpson (2017) and Simpson (2018).

So what is the evidence-based school leader/teacher to do when the scientific literature they have at their disposal is possible either limited or just plain wrong.  Well a useful starting point is Otto Von Neurathe’s  simile comparing scientists with sailors on a rotting ship - We are like sailors who on the open sea must reconstruct their ship but are never able to start afresh from the bottom. Where a beam is taken away a new one must at once be put there, and for this the rest of the ship is used as support. In this way, by using the old beams and driftwood the ship can be shaped entirely anew, but only by gradual reconstruction.  This would suggest that if you think effect sizes and meta-analysis may be leaking planks on the goodship ‘evidence-based education’ – but remain the best we have - it would be foolish to rip them out before we have something to put in their place.  Alternatively, if you are of the view that effect sizes and meta-analysis are ‘wrong’ and that you cannot draw useful conclusions from them – if you keep these ‘leaky planks’  in place – this may well lead to the ‘ship’ taking on water and becoming unstable, going off in the wrong direction or even sinking.   If that is the case, then whoever is steering the ship needs to do a least three things. First, make adjustments in the direction the ship may ‘naturally’ travel by making appropriate changes in-course.  Second, redouble their efforts to find ‘new planks’ which can be used as replacements for the leaky planks. Three, find other materials which can help plug the leaks, whilst they look for new planks.

However, from the point of view of the evidence-based school leader, it probably does not matter which stance you take on effect sizes and meta-analysis, the actions you need to undertake the address the issues at hand will in large part remain the same.   First, as Kvernbekk (2016) states when looking at research studies it’s the causal claim and associated support factors that your are after – not the average effect size.  Properly conducted randomised controlled trials, which include an appropriate impact evaluation or qualitative element – may give you several clues as to what you need to do to make an intervention work in your setting.

Second, spend time making sure you are solving the right problem.  Solving the right problem will have a major impact on whether or not you are successful in bringing about favourable outcomes for pupils.  There is no point using high quality and trustworthy research studies, if they are being used to help you solve the wrong problem.  On the other hand, if they are helping you get a better understanding of the issues at hand, then that’s another matter.

Third, as an evidence-based school leaders you won’t just rely on the academic and scientific literature, you will also draw upon a range of different sources of evidence – practitioner expertise, stakeholder views and school/organisational data to help you come up with a solution which leads to a favourable outcome.  That does not mean that these other sources of evidence are without their own problems.  Nevertheless, in making a decision it might be better to use these sources of evidence –  and being aware of their limitations -  than not to use them at all, Barends and Rosseau (2018).

Four, given the ‘softness’ of the evidence available – even if you come up with a plausible solution, you will need to give a great deal of thought to the scale of implementation.  In all likelihood, small fast-moving iterative pilot studies within your school are more likely to lead to long-term success than school or multi-academy trust wide rollouts. Langley, Moen, et al. (2009) and Bryk, Gomez, et al. (2015)  provide useful guidance as to what to do given different levels of knowledge, resources and stakeholder commitment.

Five, as Pawson (2013) states,  it is important to attend extremely closely to the ‘quality of the reasoning in research reports rather than look only to the quality of the data.’ (p11)   Moreover, it is necessary to give thought and effort considering how you go about improving the quality of your own practical reasoning.  This could be done by making sure that before you make ‘evidence-based decisions’ you make sure your thinking is tested by individuals – who may well disagree with you.  You may also want to look at the work of Jenicek and Hitchcock (2005) who provide guidance on the nature of critical thinking and strategies that you can adopt to improve your own thinking skills.

And finally

This discussion should not be seen as an attempt to dismiss the usefulness of evidence-based practice.  Rather it should be seen as an attempt to outline what to do when the research evidence is a bit ‘squishy’.  Even if it wasn’t ‘squishy’ effect sizes and systematic reviews would only provide you with a small fraction of the evidence you need to make concerning decisions about what educational interventions to adopt or withdraw within your school, Kvernbekk (2016).

 References

Barends, E. and Rosseau, D. (2018). Evidence-Based Management: How to Use Evidence to Make Better Organizational Decisions. London. Kogan-Page.

Bryk, A. S., Gomez, L. M., Grunow, A. and LeMahieu, P. G. (2015). Learning to Improve: How America's Schools Can Get Better at Getting Better. Cambirdge, MA. Harvard Education Press.

Coe, R. (2018). What Should We Do About Meta-Analysis and Effect Size? CEM Blog. http://www.cem.org/blog/what-should-we-do-about-meta-analysis-and-effect-size/. 5 December, 2018.

Jenicek, M. and Hitchcock, D. (2005). Evidence-Based Practice: Logic and Critical Thinking in Medicine. United States of America. American Medical Association Press.

Jones, G. (2018). The Ongoing Debate About the Usefulness of Effect Sizes. GaryRJones.com. https://www.garyrjones.com/blog/.

Kvernbekk, T. (2016). Evidence-Based Practice in Education: Functions of Evidence and Causal Presuppositions. London. Routledge.

Langley, G. J., Moen, R., Nolan, K. M., Nolan, T. W., Norman, C. L. and Provost, L. P. (2009). The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco. John Wiley & Sons.

Pawson, R. (2013). The Science of Evaluation London. Sage Publications.

Popper, K. (1992). The Logic of Scientific Discovery (5th Edition). London. Routledge.

Simpson, A. (2017). The Misdirection of Public Policy : Comparing and Combining Standardised Effect Sizes. Journal of Education Policy. 32. 4. 450-466.

Simpson, A. (2018). Princesses Are Bigger Than Elephants: Effect Size as a Category Error in Evidence‐Based Education. British Educational Research Journal. 44. 5. 897-913.