Measuring the impact of Lesson Study in Schools
The end of 2017 saw a considerable amount of discussion about the impact of lesson study, which was in large part prompted by the
EEF Evaluation Report on lesson study - which suggested it had no impact on pupil learning outcomes. However, this raises all sorts of questions about how to go about measuring the impact of lesson study. So to help with this task - Sarah Seleznyov of the UCL IOE has written a guest post on how schools can go about measuring the impact of lesson study.
This is trickier to do than it might seem, since there is very little literature from Japanese authors on lesson study accessible to English-language speakers. Based on an extensive analysis of the literature, and ongoing dialogue with Tokyo Gakugei University professors, listed below are what I believe to be the critical components of the lesson study process:
1. Identify focus
Teachers compare long-term goals for student learning and development to current students’ current learning characteristics in order to identify a school-wide research theme
2. Planning
Teachers work in collaborative groups to carry out kyozai kenkyu (study of material relevant to the research theme). This study leads to the production of a collaboratively written plan for a research lesson. This detailed plan attempts to anticipate pupil responses, misconceptions and successes for the lesson.
3. Research lesson
The research lesson is taught by one teacher, who is a member of the collaborative planning group. Other members of the group act as silent observers, collecting evidence of pupil learning.
4. Post-lesson discussion
The collaborative group meet to discuss the evidence gathered. Their learning in relation to the research theme is identified and recorded. It is intended that this learning informs subsequent cycles of research.
5. Repeated cycles of research
Subsequent research lessons are planned and taught that draw on the findings from the post-lesson discussions.
6. Mobilising knowledge
Opportunities should be created for teachers working in one lesson study group to access and use the knowledge from other groups, either through observing other groups’ research lessons or through the publication of group findings.
7. Outside expertise
Where possible, there should be input from a koshi or ‘outside expert’ involved in the planning process and/or the research lesson. (Seleznyov, S. (forthcoming))
And what are the expected outcomes of teacher participation in lesson study? Lesson study aims to enable teachers to make positive changes to their day-to-day classroom practices so that improvements to pupil achievement are sustained beyond the lesson study process. It is not intended as a quick-fix problem solving approach to teaching and learning, nor a short term professional learning project.
In England, there is overwhelming pressure on schools and school leaders to provide evidence of the impact of any intervention that is intended to improve outcomes for pupils. This evidence of impact is expected to be within what is in research terms a very short time frame: a year, or two years at the most, and must be evidenced in terms of pupil outcomes. Lesson study experts, however, describe lesson study as a focus on the development of expertise over decades, not months.
In line with this focus on the long term development of teacher expertise, we advise teachers not to expect to see an impact on pupil learning after one, two or even three research lesson cycles. This would align with other literature on the development of teacher expertise, eg Hattie, who advises caution in judging ‘expert teachers’ using simplistic assessment measures such as tests, which can only measure improvements in shallow learning.
However, we can anticipate that qualitative changes to some of the ‘soft’ aspects of pupil learning (for example, engagement, resilience, perseverance) will lead to quantitative changes in attainment and progress in the long term (Gutman, L.M. and Schoon, I., 2013). Teachers engaged in lesson study can look for key aspects of the pupils’ learning that are likely to lead to the biggest changes in their progress over time and can see the research lessons as a vehicle to gain insight into these aspects.
When evaluating lesson study, school leaders should consider gathering evidence against the full logical chain that might enable changes to pupil outcomes. Our logical chain (Godrey et al., forthcoming) is based on the work of Guskey, who describes five different levels at which the effectiveness of professional development can be measured:
1. Teachers’ reactions
Teachers’ attitudes to and enjoyment of professional learning might improve. Did teachers like lesson study? How was it different to / better than the other professional development they experience?
2. Teachers’ professional learning
Teachers might report improved subject knowledge, pedagogical content knowledge and self-confidence. What did teachers learn? How did lesson study enable them to learn this?
3. The organisation’s professional development model
The structure, time, resourcing of the school’s professional learning programme might alter in order to accommodate lesson study. Cultural attitudes towards professional learning might shift in relation to: the role of peer-to-peer learning, teacher ownership of learning, lesson observation as learning not performance. Did we as leaders do enough to enable the lesson study to be of as high a quality as possible? Has teachers’ participation in lesson study made them think differently about eg working with other colleagues, lesson observations, etc?
4. Teacher use of new knowledge and skills
Teachers’ newly acquired confidence, subject knowledge and pedagogical content knowledge should lead to changes in practice. Have teachers made any changes to their day-to-day classroom practice based on what they learnt through lesson study? How substantial are these changes to practice?
5. Pupil learning outcomes
Changes in teachers’ practice should lead to improved attitudes to learning and ultimately progress for pupils, in terms of evidence from written work and assessment data. Have teachers noticed any changes to pupil learning, based on the changes they have made to their practice? What is their evidence for claiming this?
School leaders considering how best to evaluate the impact of their school’s lesson study projects should consider which of the above five levels are of most relevance to their own context and seek evidence against those priorities, rather than focusing solely on level five. And think also about how the school’s professional development model supports the effective implementation of lesson study:
• To what extent has your school invested in a long term approach to professional learning and does your proposed cost-impact evaluation for lesson study take this into account?
• How can your impact evaluation support teachers to focus less on the short term quantitative analysis of pupil assessment data and more on the longer term qualitative analysis of pupil learning?
• Can a process be put in place to gather long term evidence of impact on pupil learning?
For more information on our lesson study programmes, to join the UCL Institute of Education Lesson Study Network or to purchase the UCL Lesson Study Handbook, contact: s.seleznyov@ucl.ac.uk'
References:
Godfrey, D., Seleznyov, S., Wollaston, N., and Barrera-Pedamonte, F. (forthcoming). Target oriented lesson study (TOLS) Combining Lesson study with an integrated impact evaluation model.
Seleznyov, S. (forthcoming). Lesson study: an exploration of its translation beyond Japan
Guest post by Sarah Seleznyov - Measuring the impact of lesson study in schools
'If we are going to measure the impact of lesson study, we need first to be sure of what it is and what its expected outcomes are. Based on this understanding, we then need to decide what it is important to evaluate and measure.This is trickier to do than it might seem, since there is very little literature from Japanese authors on lesson study accessible to English-language speakers. Based on an extensive analysis of the literature, and ongoing dialogue with Tokyo Gakugei University professors, listed below are what I believe to be the critical components of the lesson study process:
1. Identify focus
Teachers compare long-term goals for student learning and development to current students’ current learning characteristics in order to identify a school-wide research theme
2. Planning
Teachers work in collaborative groups to carry out kyozai kenkyu (study of material relevant to the research theme). This study leads to the production of a collaboratively written plan for a research lesson. This detailed plan attempts to anticipate pupil responses, misconceptions and successes for the lesson.
3. Research lesson
The research lesson is taught by one teacher, who is a member of the collaborative planning group. Other members of the group act as silent observers, collecting evidence of pupil learning.
4. Post-lesson discussion
The collaborative group meet to discuss the evidence gathered. Their learning in relation to the research theme is identified and recorded. It is intended that this learning informs subsequent cycles of research.
5. Repeated cycles of research
Subsequent research lessons are planned and taught that draw on the findings from the post-lesson discussions.
6. Mobilising knowledge
Opportunities should be created for teachers working in one lesson study group to access and use the knowledge from other groups, either through observing other groups’ research lessons or through the publication of group findings.
7. Outside expertise
Where possible, there should be input from a koshi or ‘outside expert’ involved in the planning process and/or the research lesson. (Seleznyov, S. (forthcoming))
And what are the expected outcomes of teacher participation in lesson study? Lesson study aims to enable teachers to make positive changes to their day-to-day classroom practices so that improvements to pupil achievement are sustained beyond the lesson study process. It is not intended as a quick-fix problem solving approach to teaching and learning, nor a short term professional learning project.
In England, there is overwhelming pressure on schools and school leaders to provide evidence of the impact of any intervention that is intended to improve outcomes for pupils. This evidence of impact is expected to be within what is in research terms a very short time frame: a year, or two years at the most, and must be evidenced in terms of pupil outcomes. Lesson study experts, however, describe lesson study as a focus on the development of expertise over decades, not months.
In line with this focus on the long term development of teacher expertise, we advise teachers not to expect to see an impact on pupil learning after one, two or even three research lesson cycles. This would align with other literature on the development of teacher expertise, eg Hattie, who advises caution in judging ‘expert teachers’ using simplistic assessment measures such as tests, which can only measure improvements in shallow learning.
However, we can anticipate that qualitative changes to some of the ‘soft’ aspects of pupil learning (for example, engagement, resilience, perseverance) will lead to quantitative changes in attainment and progress in the long term (Gutman, L.M. and Schoon, I., 2013). Teachers engaged in lesson study can look for key aspects of the pupils’ learning that are likely to lead to the biggest changes in their progress over time and can see the research lessons as a vehicle to gain insight into these aspects.
When evaluating lesson study, school leaders should consider gathering evidence against the full logical chain that might enable changes to pupil outcomes. Our logical chain (Godrey et al., forthcoming) is based on the work of Guskey, who describes five different levels at which the effectiveness of professional development can be measured:
1. Teachers’ reactions
Teachers’ attitudes to and enjoyment of professional learning might improve. Did teachers like lesson study? How was it different to / better than the other professional development they experience?
2. Teachers’ professional learning
Teachers might report improved subject knowledge, pedagogical content knowledge and self-confidence. What did teachers learn? How did lesson study enable them to learn this?
3. The organisation’s professional development model
The structure, time, resourcing of the school’s professional learning programme might alter in order to accommodate lesson study. Cultural attitudes towards professional learning might shift in relation to: the role of peer-to-peer learning, teacher ownership of learning, lesson observation as learning not performance. Did we as leaders do enough to enable the lesson study to be of as high a quality as possible? Has teachers’ participation in lesson study made them think differently about eg working with other colleagues, lesson observations, etc?
4. Teacher use of new knowledge and skills
Teachers’ newly acquired confidence, subject knowledge and pedagogical content knowledge should lead to changes in practice. Have teachers made any changes to their day-to-day classroom practice based on what they learnt through lesson study? How substantial are these changes to practice?
5. Pupil learning outcomes
Changes in teachers’ practice should lead to improved attitudes to learning and ultimately progress for pupils, in terms of evidence from written work and assessment data. Have teachers noticed any changes to pupil learning, based on the changes they have made to their practice? What is their evidence for claiming this?
School leaders considering how best to evaluate the impact of their school’s lesson study projects should consider which of the above five levels are of most relevance to their own context and seek evidence against those priorities, rather than focusing solely on level five. And think also about how the school’s professional development model supports the effective implementation of lesson study:
• To what extent has your school invested in a long term approach to professional learning and does your proposed cost-impact evaluation for lesson study take this into account?
• How can your impact evaluation support teachers to focus less on the short term quantitative analysis of pupil assessment data and more on the longer term qualitative analysis of pupil learning?
• Can a process be put in place to gather long term evidence of impact on pupil learning?
For more information on our lesson study programmes, to join the UCL Institute of Education Lesson Study Network or to purchase the UCL Lesson Study Handbook, contact: s.seleznyov@ucl.ac.uk'
References:
Godfrey, D., Seleznyov, S., Wollaston, N., and Barrera-Pedamonte, F. (forthcoming). Target oriented lesson study (TOLS) Combining Lesson study with an integrated impact evaluation model.
Seleznyov, S. (forthcoming). Lesson study: an exploration of its translation beyond Japan