The school research lead, testing interventions and better teaching and learning

March 30, 2019

As regular readers of this blog will know, I’m a great believer that those of us who are interested in evidence-based education can learn much from the experience of colleagues working in the field of evidence-based medicine. So I was delighted to have recently come across Evans et al’s (2011) book Testing Treatments: Better research for better healthcare which set out to explore how we ensure that medical treatments best meets the needs of patients. In particular it looks at how to ensure ‘that research is soundly based, properly done, able to distinguish harmful from helpful treatments, and designed to answer questions that matter to patients, the public, and health professionals.

What I like about the book was how it was upfront about the limitations of research i.e. all it can really do is help us become a little less uncertain about how to proceed. High quality research can indicate the probability that a particular treatment will lead benefits or harms for the patients, and which is equally true of research into interventions within education.

Helpfully, at the end of each chapter Evans et al come up with a list key points to think about, which for me, many of which equally applicable in education. Below is an edited list key points, which have been amended for educational settings.

Testing new teaching strategies/interventions is important because new teaching strategies are as likely to be worse as they are to be better than existing teaching strategies
Biased and unfair tests of interventions can leads to pupils losing out and having reduced life chances
Just because a major body recommends the approach, does not mean it might not harm pupils
The beneficial effects of interventions are often overplayed, and the harmful effects underplayed
Neither theory or professional opinion is a reliable guide to effective interventions
Just because an intervention or practice has been used for years does not mean that it is effective
Even if pupils are not ‘harmed’ by the intervention, using them is a waste of resources
More intensive use of an intervention is not necessarily beneficial, and can sometimes do more harm than good
Better diagnosis does not necessarily lead to better outcomes, sometimes it makes matters worse
Screening programmes should only be introduced on the basis of sound evidence of their effects
Dramatic effects of an intervention are rare
Uncertainties about the effects of an intervention are common
Fair tests of interventions are needed because we will not otherwise sometimes conclude that interventions are useful, when they are not, and vice versa
Comparisons are fundamental to all fair tests of treatments
A single study rarely provides enough evidence to guide intervention choices in education
Assessments of the relative merits of alternative interventions, should be based on systematic reviews of all the relevant, reliable evidence
New research should only proceed if an up-to-date review of earlier research shows that it is necessary
Much research is of poor quality and done for questionable reasons
Input from teachers, pupils and school stakeholders can lead to better research

Of course, this is not an exhaustive list of what needs to be done to improve the quality of and usefulness of educational research. On the other hand, will help you become more sceptical about some of the claims made about educational interventions. And by being sceptical, I’m not talking about someone who is a continual ‘naysayer’ but rather someone who withholds approval or disapproval until they have made appropriate and rigorous inquiries.

Reference

Evans, I., Thornton, H., Chalmers, I., & Glasziou, P. (2011). Testing treatments: better research for better healthcare. Pinter & Martin Publishers

Disciplined Inquiry as a panacea for performance management - Where's the evidence

March 23, 2019

This is the link to my session at researchED Blackpool in which I put forward the following argument

Colleagues in research schools (and wider) are showing an interest in disciplined inquiry
This is a product of three things
- Dylan William’s view that all teachers should seek to improve and should take part in 'disciplined inquiry’
- Bloggers writing about disciplined inquiry
- Widespread dissatisfaction with current models of performance management in schools
Disciplined inquiry is now being used in a number of schools as an integral part of school’s performance management processes and CPD activities
However, this is being done, with little or no reference to the research literature on what makes for effective performance management processes; the relationships between disciplined inquiry and teacher knowledge, skills, attitudes and behaviours and teacher outcomes; and different types of inquiry – such as action research
Ironically the promotion of disciplined inquiry as part of performance management is an example of what the evidence-based community is trying to avoid i.e. addressing problems with little reference to the research evidence-base and the adoption of practices promoted by gurus
Nevertheless, this does not mean we should not show interest in ‘disciplined inquiry’ as a way of addressing the problems associated with performance management in schools.
Although we should be upfront and say that while the adoption of DI seems a goods idea. there is little or no robust evidence about what works, where, for whom, to what extent, for how long - when undertaken as part of performance management

School Governance and Educational Improvement

March 16, 2019

The week before last saw the publication of the Education Endowment Foundation’s guidance on evidence-informed guidance. At the time, I was struck by the lack of any research evidence within the report, detailing how school governing bodies/trusts can bring about school improvement. Since then, with the aid of Terry Pearson @TPLTD, I have been able to locate a systematic literature review by Honingh, Ruite and van Thiel (2018) which looks at the relationship between school boards and educational quality.

Honingh et al having originally identified nearly 5000 articles on school boards and governing bodies in the USA, Netherlands, England and the Flemish Community of Belgium, eventually reduced to just 16 empirical articles from which they drew the following conclusions.

There is a lack of solid and robust empirical evidence on the relationship between school boards and governing bodies with educational quality. So it would seem that a theory of change which relates governing body activities with school improvement is not evidence-based. Governing bodies may have expectations placed upon them, which are just not supported by the evidence.
Within the 16 studies – there were differences between in the scope, scale and quality of the research, as well as country and regional differences. Claims and conclusions about the effectiveness of school boards/governing bodies are often based on personal experience, observations and opinions, as such the empirical base is mainly anecdotal
School boards and governing bodies are not homogenous. There are differences in how both a board is constituted – size, representation, role of school leader, primary and secondary – and the context in which the board is operating within, and the level of delegated authority. As such, this has major implications for practitioners when seeking to interpret and apply these findings.
The research on governing bodies has focussed on the outcomes of test results, rather than a broader definition of educational quality. As such, we know very little about the relationship between about the relationship between governing bodies and broader educational aims and objectives.

Some observations and implications

Just because since the 1980s governing bodies have become part and parcel of school life, that does not mean there is any evidence to support claims that they bring about school improvement. Indeed, this could probably be said about many activities within a school, for example, parents evenings (if you are communicating anything of importance at a parents’ evening this reflects a failure of day to day parent school communication)
Thought should be given to the development of a range of robust theories of change which links the work of the governing body with broader educational aims and objectives
Given pressures of senior leader and teacher work-load could a school be run without a governing body, with any functions deemed to be essential being undertaken by other individuals in the school, in a more time efficient and effective manner.
Social media and EduTwitter often gets a ‘bad-rap’. However, Twitter is full of people who are willing to make help, make suggestions and point out interesting research and is not just a playground for the ‘trads and the progs’ in their culture wars.

And finally

In coming weeks, future posts will be looking at how evidence-based practitioners can go about gaining feedback from stakeholders.

Reference

Marlies Honingh, Merel Ruiter & Sandra van Thiel (2018): Are school boards and educational quality related? Results of an international literature review, Educational Review, DOI: 10.1080/00131911.2018.1487387

The EEF and evidence-informed governance - where's the evidence

March 09, 2019

This week saw the publication of the Education Endowment Foundation’s guide to being an evidence-informed governor. However, what I found deeply ironic was the complete absence of any reference to research evidence about the relationship between school governance and school improvement. So to try and address this evidence gap I checked out the Ofsted (2019) overview of the research underpinning the new inspection framework. Unfortunately, a word search in the document – using the terms governor and governance – found not a single reference to either term, which I must admit I found surprising My next port of call was Google and Google Scholar – where I used the search terms ‘school governors and school improvement’ – and low and behold – the first article returned was Ransom et al (2005) paper ‘Does governance matter for school improvement? – which although behind a paywall, could also be found as freely available PDF. Using, Ransom et al as my ‘crib’ the rest of this post will:

· Articulate a theory of change for governance and school improvement

· Report of Ransom et al’s national study on the relationship between governance and school improvement in Wales

· Make some observations about a second major limitation of the EEF’s guidance on being an evidence-informed governor

A theory of change for governance and school improvement

Using Long et al’s (2018) PIT—B model for a theory of change as a prompt, I’ve come up with the following theory of change for governance and school improvement.

All schools need to be constantly seeking to improve. Our theory is that if governors reinforce the importance of instructional leadership, provide strategy, scrutinise practice, offer support and ensure accountability this will improve the effectiveness of leadership and management of the school. Increased effectiveness of school leadership will lead to an improvement in the environment of learning and teaching, which we believe will lead to increased standards of educational attainment and generate better results for pupils.

Researching school governance in Wales

Ransom et al undertook a research project into the governing bodies of 72 schools – both secondary and primary – during the period 1998-2002. Preliminary interviews were held with the headteacher and chair of governors, with questionnaire being administered to each member of the governing body. Thirty schools were chosen for more focused case-study research as they illustrated elements of good governance. Field work as undertaken with members of the school’s senior leadership team, along with observation of governing body and subcommittee meetings. School performance data appeared not to be made available by LEAs, although GCSE and Key Stage test scores were provided by the Wales Assembly Government. Of the 72 schools – 44 showed changes in performance over time – either improving, declining or being ‘stuck’ at a particular level of attainment.

Ransom et al analysed types of governance – deliberative forum, consultative sounding board, executive board and governing body (see end of post) – and found that there appeared to be a link between school performances and good governance. Practices which seems to be associated with the improvement of primary schools included in the study.

1. Governance and governors are valued: because they provide a different voice and perspective, because they bind the school to the wider community, strengthen the corporate nature of the school and the public, collective stature of its decisions.

2. Governance that represents the diversity of its parent communities: Including the participation and voice of different parents helps the school to understand the variety of learning needs as well as securing their commitment to supporting learning in the home.

3. Partnership between head and governors are of mutual support: The head values and supports the governing body in their roles, just as the governors seek to support the head and staff in the school. Heads do not seek to superintend, to take over the role of governance.

4. Clarity of roles: Heads are chief executives providing professional leadership and day-to-day management, while the governing body has oversight of the school: it is the publicly accountable body.

5. Organised as an executive board or governing body: exercising functions of scrutiny, strategy and accountability.

6. Scrutiny as the strategic function of the best primary school governing bodies, assuring the quality and standards of education in the school. This is achieved by:

· bringing high expectations to school;

· ensuring full deliberation and questioning of the policies, budgets, and practices of the school;

· putting in place systems for monitoring and reviewing the standards of achievement, financial plans and policy developments of the school.

1. Embodying the values and ethos of the school: The governors express the public values and purpose of justice and fairness as well as any particular denominational or language ethos.

2. Close attachment of governors to the life of the school through a system of links to curriculum areas and classroom visits in order to develop knowledge and understanding of the key practices of learning in the school.

3. Close ties with the community: Involving parents and the community is key to the success of the school and the governors have a key role in securing that partnership. P13-14)

Ransom et al go onto note that what is particularly distinctive about this set characteristics is the focus on ‘practices of scrutiny’ which are believed by headteachers and governors as being important in bringing about school improvement. As such, governing bodies as there appears to be a relationship between governance and school improvement.

How useful is the research for English schools in 2019?

First, it needs to be remembered that the research took place nearly 20 years ago in a different educational system. So it’s necessary to be careful in attempting to make any kind of read across from ‘there and then’ to ‘here and now’. Second, as Ransom et al acknowledge there are issues around the causal relationship between governance and school improvement. Does governance lead to improvements or does improvement generate better governance. That said, the research does provide some backing for a claim that school improvement could come about if governors challenge school leaders around practices identified in the EEF’s guidance: how well are your pupils doing; how effectively is the school spending its money; and, how does the school support effective teaching and learning.

An additional observation

Governing bodies differ from school to school. In some schools, school governors who wish to become more evidence-informed or want more evidence-informed decisions, may well be pushing against an ‘open-door’ and where there is a whole-school culture of evidence-use – (Coldwell et at 2017). On the other hand, the school may have a very weak evidence culture, with little or no use of research evidence by senior leaders. Indeed, there is some research by the Sutton Trust that suggests only 68% of headteachers and 45% of teachers in England cite using research evidence to inform decision-making, Governors who are governors in schools with little or no use of research evidence may find themselves quite isolated and subject to challenge, This may well be the case if the ‘evidence’ leads to the challenge of existing and long established practices. Unfotunately, the EEF guidance provides no assistance in how to address this issue.

And finally

It’s important for governors to remember that the Department for Education’s Competency Framework for Governance makes explicit reference to governors making decisions based on the best available evidence. In other words, making the best use of research evidence is not an option, instead it is necessary for competent governance – so even if the EEF’s guidance is not perfect, it is essential reading.

Update

Since this post was published, Terry Pearson @TPLTD, very kindly identified additional research on school governance, which is worthy of consideration.

Baxter, J. (2017). School governor regulation in England’s changing education landscape. Educational Management Administration & Leadership, 45(1), 20–39. https://doi.org/10.1177/1741143215587306

James, C., Brammer, S., Connolly, M., Fertig, M., James, J., & Jones, J. (2011). School Governing Bodies in England Under Pressure: The Effects of Socio-economic Context and School Performance. Educational Management Administration & Leadership, 39(4), 414–433. https://doi.org/10.1177/1741143211404258

References

Coldwell, M., Greany, T., Higgins, S., Brown, C., Maxwell, B., B, S., Stoll, L., Willis, B. & Burns, H. 2017. Evidence-informed teaching: an evaluation of progress in England Research report. London: Department for Education.

DfE. (2017). A Competency Framework for Governance the Knowledge, Skills and Behaviours Needed for Effective Governance in Maintained Schools, Academies and Multi-Academy Trusts : January 2017. London. Department for Education

EEF (2019) The EEF guide to becoming an evidence-informed school governor and trustee, London Education Endowment Foundation,

Long M, Macdonald A and Duncan T. (2018) Practical Tips for Developing and Using Theories of Change and Logic Models. 2018 Virginia AmeriCorps Annual Program Directors and Staff Meeting, Richmond, VA: ICF

Ofsted (2019) Education inspection framework: Overview of research, London, OFSTED.

Ranson, S., Farrell, C., Peim, N., & Smith, P. (2005). Does governance matter for school improvement?. School effectiveness and school improvement, 16(3), 305-325.

Note on different types of governance

Ransom et al identify four distinct types of governing body

Governance as a deliberative forum. Here governance constitutes largely a gathering of members, often parents, at which discussions of the school are determined and led by the headteacher as professional leader. Parents will not feel they can question the authority of the head though they may inquire about aspects of the school’s progress.

Governance as a consultative sounding board. Here governors define their role as providing a sounding board for the strategies and policies provided by the headteacher as principal professional.

Governance as an executive board. In these schools a partnership has developed between the governors and the school and, in particular, between the head and the chair with the former leading ‘‘primus inter pares’’. There may be a division of labour between them. The board assuming overall responsibility for the business aspects of the school: the budget, staffing, and the infrastructure of building. Their concern is with their legal responsibility and accountability for the school. The head assumes overall responsibility for curricular and pedagogic aspects of the school.

Governance as a governing body. In these schools, the governing body takes overarching responsibility for the conduct and direction of the school. The head will be a strong professional leader, but a member rather than leader of the governing body that acts as a corporate entity. The agenda and the meeting will be led by the chair. (p12 and 13)

The School Research Lead - and a PIT-B theory of change

February 24, 2019

Recently I’ve been giving a lot of thought to the related ideas of theories of change, theories of action and logic models. Fortunately, there are lots of really useful resources available online which provide practical advice on how to make these ideas ‘work’. In this post I am going to look at the work of (Long et al., 2018) and their PIT-B model for developing a theory of change (TOC). Full details of which can be found using the following link.

Let’s start by stating what is meant by the term ‘theory of change’. Put simply a theory of change explains ‘how and why’ a particular intervention will in a specific context bring about the desired or hoped for change.

Long et al recommend that when developing a theory of change, you begin with the objective or problem you are trying to solve or address. Second, you explain what aspect of the intervention/innovation you are proposing that allows it to solve the problem or achieve the objective. Next, describe how the intervention facilitates and carries out this causal mechanism. Finally, complete your causal chain by returning to the problem or objective you started with.

Long et al suggest that this can be done by using the PIT-B model

We can now use this model to help illustrate theories of change which could be developed for use in your work as a school research lead.

Example 1

Teachers in school do not see the practical relevance of educational research, when planning lessons and schemes of work, so it is not unexpected that teachers do not make use of educational research. Our theory is that if teachers have more opportunities to read and discuss research, they will begin to make greater use of educational research in their teaching. The school’s Journal Club promotes the use of educational research by providing setting for research literate colleagues to support colleagues think about how research could be applied in classroom setting. We believe that this encourage an increase in educational research, and increase its use in lesson plans and schemes of work.

Example 2

Teachers in school are often demotivated by existing schemes of performance management and objective setting for accountability purposes. Out theory is that teachers have opportunities to focus on ‘development’ objectives rather than ‘accountability’ objectives this will increase teacher motivation and engagement in performance management process. The school’s programme of ‘disciplined inquiry’ encourages teachers to think about how they can bring about improvements both teaching and pupil outcomes. We believe that this will increase the motivation of teachers, and eliminate the demotivating impact of accountability based performance management

However, it is important to realise that coming with a theory of change is the ‘easy-bit’ . In order to create a ‘good’ theory of change, Long et al recommend that the underpinning assumptions and hypotheses are stated. Second, the appropriate resources are available. Third, the language used is clear and unambiguous. Fourth, there should be agreement from all the relevant stakeholders. Last but not least, the focus should be on one particular intervention.

And finally

Developing a robust theory of change will not guarantee that you whatever intervention you are introducing will be a success. However, it will increase you chance of success as you will be able to articulate - the how and the why of what you are trying to achieve - this allows you to then focus on what you are going to do to make it happen - and this is another story.

Reference

Want to know more

If you are interested in finding out more about theories of change, theories of action and logic models, have a look at the following

https://www.betterevaluation.org/en/rainbow_framework/define/develop_programme_theory