A major challenge for aspiring   evidence-informed teachers is knowing when to trust the experts.   It would be easy to assume that just because you have come across a particular interpretation of a concept or idea in a number different places - book, peer-reviewed article or blog - that it is correct.  Unfortunately, if you did this, you could well be making a mistake.   For example, in recent weeks I have come across three examples – Churches and Dommett (2016) , Firth (2018) and Ashman (2018)  - where the meaning of p-values and statistical significance would appear to have been misinterpreted.   Furthermore, as Gorard et al (2017) states this mistakes are not uncommon.  So to help aspiring   school research leads and evidence-informed teachers spot where p-values and statistical significance have been misinterpreted I will:

P values and statistical significance

When seeking to understand these terms there are  a number of major problems and as Greenland, et al. (2016) state: ‘There are no interpretations of these concepts, which are at once simple, intuitive, correct, and foolproof’ (p337). Greenland et al go onto illustrate their point by providing twenty-five examples of common misconceptions and interpretation of these terms, which even professional academics are prone.   Nevertheless, the American Statistical Association seek to informally  define a p-value as: the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.

The smaller the p value, the more unlikely are our results if the null hypothesis (and test assumptions) hold true.   Whereas, the larger the p value, the less surprising are our results, given the null hypothesis and (test assumptions) hold true.  In other words, as Greenland et al state: ‘The P value simply indicates the degree to which the data conform to the pattern predicted by the test hypothesis and all the other assumptions used in the test (the underlying statistical model). Thus P =  0.01 would indicate that the data are not very close to what the statistical model (including the null hypothesis) predicted they should be, while P =  0.40 would indicate that the data are much closer to the model prediction, allowing for chance variation’. p340

Statistical Significance

Put very simply a result is often deemed to be statistically significant if the p value is less than or equal to 0.05, although the level of statistical significance can be set lower levels, for example, p is less than or equal 0.01  

Interpreting p values and statistical significance – guidance from the American Statistical Association

Given difficulties in interpreting p values and statistical significance the American Statistical Association  - Wasserstein and Lazar (2016) – have provided some guidance on how to avoid some common mistakes.   This guidance is summarised in six principles

Some common misinterpretations – Churches, Dommett, Firth and Ashman

I will now look at how Churches and Dommett, Firth. and Ashman have - in my view-  all misinterpreted either p-values or statistical significance.  

Richard Churches and Eleanor Dommett - In their book Teacher-Led Research : Designing and implementing randomised controlled trials and other forms of experimental research – t include the following definitions within their glossary of terms

p-value – Probability value – that is the probability that the result may have occurred by chance (e.g p = 0.001 – a 1 in 1000 probability that the result may have happened by chance) Also known as the significance level.

Significance – The probability that a change in score may have occurred by chance. A threshold for significance (alpha) is set at the start of piece of research.  This is never less stringent than 0.05 ….

Unfortunately,  according to the ASA  both of these statements are incorrect. First, the p-value is a measure of the consistency of the results with a particular statistical model – with all the assumptions behind the model being maintained.   Second,  the p-value is not the probability that the data were produced by random chance alone as it also depends on the accuracy of the assumptions underpinning the statistical model  Third, the definition of significance conflates scientific significance, with statistical significance.

   

Jonathan Firth - Firth, J. (2018). The Application of Spacing and Interleaving Approaches in the Classroom Impact. 1. 2.

In a recent edition of Impact, Jonathan Firth uses p-values and statistical significance to the application of spacing an interleaving in the classroom where an opportunity sample of 31 school pupils between 16 and 17 years of age was used.

The mean percentages of correct answers on the end-of-task test for the interleaved and blocked conditions are shown in Figure 4. A between-subjects ANOVA was carried out. This analysis revealed a significant main effect of spacing (performance in the spaced condition being worse than the massed condition, with mean scores of 12.25 vs 9.45, p = .002), while interleaving did not have a significant main effect.  Importantly, there was also a significant (p = .009) interaction between the two variables (spacing vs interleaving), indicating that interleaving had a mediating or protective effect against the difficulties caused by spacing (see Figure 5). 

The findings demonstrated that spacing had a harmful effect on the immediate test, while the main effect of interleaving was neutral. The results fit with the idea that these are ‘desirable difficulties’, with the potential to impede learning in the short term. .   

Again, according to the ASA there are errors in both paragraphs.   Statistical significance does not demonstrate whether an  a scientifically or substantively important/significant relation has been detected.  Neither is statistical significance a property of the phenomenon being studied, but is a product of the consistency between the data and what would have been expected using the specified statistical model.   In other words, the map is not the territory.  

Greg Ashman - Ashman (2018) The Article That England’s Chartered College Will Not Print. Filling the Pail.

In a blogpost which criticises the EEF’s approach to both meta-cognition and meta-analysis, Greg also falls foul of the problems of interpreting p-values and statistical significance

If we focus only on the randomised controlled trials conducted by the EEF, the case for meta-cognition and self-regulation seems weak at best. Of the seven studies, only two appear to have statistically significant results. In three of the other studies, the results are not significant and in two more, significance was not even calculated. This matters because a test of statistical significance tells us how likely we would be to collect this particular set of data if there really was no effect from the intervention. If results are not statistically significant then they could well have arisen by chance.

Again using the ASA’s guidance there are a number of errors in this statement.     First, statistical significance – or rather the lack of it – does not tell us whether there was no effect from the intervention. It just tells us the data was inconsistent with the statistical model.   Second, even if the results are or are not statistically significant it does not mean the results have arisen by chance. It is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself.   In other words, it is a statement about the results of the study relative to a particular statistical model.

Where does this leave us?

First, p-values, significance and statistical are slippery concepts, which take time and effort to even begin to understand never alone master.   Indeed, you may need to forget what you have already learnt at university on under-graduate or post-graduate courses.

Second, misuse of p-values and statistical significance is not uncommon, so is something you have to watch out for when reading quantitative research reports.   So keep the ASA principles hand to see if they are being misapplied in research reports.   You don’t have to understand something and how it works, (though it helps) to be able to spot it misuse.

Third, just because you can come across something in a variety of formats – book, peer-reviewed article or blog and from a variety of authors – researchers, researchers at university, or school teachers - does not mean it is correct.

Fourth, I am not making personal comments about the personal integrity of any of the authors I have criticised.   These comments should be seen as ‘business not personal’ and are a genuine attempt to increase the research literacy of teachers and school leaders.   Being an evidence-informed teacher or school leaders is hard enough when you are using the right,  never mind the wrong, tools.

And finally,  it’s worth remembering the words of Greenland, et al. (2016) who state: ‘In closing, we note that no statistical method is immune to misinterpretation and misuse, but prudent users of statistics will avoid approaches especially prone to serious abuse. In this regard, we join others in singling out the degradation of P values into ‘‘significant’’ and ‘‘nonsignificant’’ as an especially pernicious statistical practice.’ p348.

References