Research - It should be called BS Repellent

I don't like BS.  I don't like it when people make bold, loud claims about different aspects of education, about what works and what doesn't.  What is even more troublesome for me than these controversial statements is when I don't have the research backing to quantitatively and qualitatively refute the BS.  After many negative experiences challenging people on their beliefs without backing of my own, I have determined that people cling strongly to these thoughts (whether they are BS or not) unless you can provide research to them that might change their mind.  And really, if you are questioning a set of beliefs with nothing more than your own beliefs, then you are a BSer just as much as the next person.

As a result, over the past several years, I have become a research junkie. I love reading studies about pretty much anything educational.  Homework--does it make a difference?  How about class size?  Failing a student?   But the one thing that I find with reading research is that it takes a great deal of time, and it seems that for each study you find that supports one side of an argument, there are two or three that support the other side of the argument: often, you end up walking away wondering which study to believe. 

However, I have found a new educational Bible for me.  It is called "Visible Learning", by John Hattie, and it is a compilation of more than 800 meta-analyses of different factors that influence student achievement.  And remembering that meta-analyses are the studies of all of the studies, they tend to remove some of the bias that may come from individual studies. And as far as I am concerned, this book represents what I am going to call educational BS Repellent, and it is something that I feel that all educators should read. 

We all have our beliefs about educational reform and about strategies that improve student achievement.  Some of these notions are based on experience, others are founded in research, and even more are based on what we tend to believe through our peers and those that we respect.  This is not to de-value the importance of what we have experienced, and in many cases, these experiences are in fact what is borne out by research.  But the question is, how do you know if a claim to the success or failure of an idea or initiative really is true, or simply BS? In times of the shrinking buying power of budgets and increased demands on public education, don't we think it might be important to do some research into the high-yield strategies that are research proven to work so that we can use those resources as efficiently as we possibly can?  And within that context, I believe that it is important to make sure that we quash some of the BS that tends to cloud our judgment when we are making critical decisions for educational reform and improving student achievement. 

Hattie's book defines a term called the "hinge point"--this is the point at which it can be said that a factor or innovation has a significant impact on student achievement, and is the average impact of all of the factors/innovations in the six categories of student, home, school, teacher, curricula, and teaching.  The numerical value of this "hinge point" is 0.4.  As a frame of reference, the factor with the highest impact on student had a value of 1.44, and the factor with the lowest (and in fact deleterious) effect was -0.34.

I encourage you to read this book and draw your own conclusions.  However, here is some of the research findings on six interesting educational thoughts that I had, and have now given me pause for thought.  Again, this is all from Hattie (2009), and all credit goes to him and the meta-analyses that he compiled in his book. These are simply a few of my reflections on his book.

1. Class Size

My initial thought: Decreasing Class Size from 25 to 15 could significantly improve student achievement.

The bold, loud claim I hear:  "Decreasing class sizes is a key to student success!"

What the research says:  Of the 138 factors of the meta-analyses done, this was ranked as number 106, and had a impact factor of 0.21, well below the hinge point of showing notable change.  This is based on studies of more than 40000 classes, and nearly 950000 students worldwide. Perhaps not surprisingly, "quality teaching" has nearly double the impact on student achievement than this factor.

My new thought:  Not the high-yield strategy that I believed.

2.  Retaining/failing students

My initial thought:  Holding a student back or failing them is not effective.

The loud, bold claim I hear: "Kids don't fail enough these days, they just get passed on.  They need to be held back if they don't have the skills!"

What the research says:  Of the 138 factors, this is ranked 136, and in fact has a negative effect on student achievement at -0.16.  This is based on 207 studies and more than 13000 students worldwide.  This effect gets worse over time, and retained students lose more and more ground on students as the years pass.  Students that are promoted do better on the same outcomes than those who are retained.  As well, those who are retained did poorer across the board, whether it was reading, language arts or math!

My new thought:  The same as my old one, retaining students is not effective, even in a sequentially building course such as math.

3. Ability grouping of students

My initial thought:  Very mixed. It might have some benefits for certain subject areas, maybe math?

The loud, bold claim that I hear:  "Kids need to be grouped by ability so the teacher can focus on their specific needs."

What the research says:  This factor ranks 121st out of 138, with 500 studies done over hundreds of schools.  It has an average effect of .14, and mostly for higher achieving students.  In specific subject areas, it has almost no impact: in English, .02, and in Math ZERO effect.  But it has "profoundly negative effects" on student's feelings of equity.

My new thought:  There is no benefit to 'streaming' students in schools, and with the changes in the current Math curriculum in British Columbia, the idea of creating adapted classes for students will have more cost than benefit.

4. The relationship between the teacher and the student

My initial thought:  This is a very important factor for student success.

What the research says:  Of the 138 factors, this ranked #11, with an impact factor of .72.  This is based on nearly 230 studies and more than 350000 students.  Some behaviours that are particularly important for teachers are empathy, warmth, and 'non-directivity': allowing students more student-initiated or student-directed activities for learning.

My new thought:  This is even more important than I initially believed (and I believed relationships to be tremendously important), and something that I want to emphasize at our school.

5.  The effect of Principals and School Leaders

My initial thought:  This is a very important factor for student success, especially transformational leadership--not because I am a Principal, but because I have read and heard so much about how important school leadership is supposed to be. 

What the research says:  This ranked 74th out of 138 factors, and came from nearly 500 studies spanning over 1.1 million students.  A quote from the book: "Instructional leadership refers to those principals who have their major focus on creating a learning environment free from disruption, a system of clear teaching objectives, and high teacher expectations for teachers and students.  Transformational leadership refers to those principals who engage with their teaching staff new in ways that inspire them to new levels of energy, commitment, and moral purpose to collaboratively overcome challenges and reach ambitious goals...the evidence supports the former (instructional) over the latter (transformational)." (Hattie, 2009).

My new thought: Not quite as important as I thought, and I need to spend more time and focus on setting clear teaching objectives and high standards for my staff and students.

6.  Formative Evaluation of programs

My initial thought:  Extremely important for teachers to adapt and change their methodologies in response to student learning. Using student data to guide instruction and reflection through collaboration with their peers is something that we have been focussing on in our school through our change in structures.

Loud, bold claim I hear:  "I know what works in my class!"

What the research says:  This ranks as #3 of 138, with an effect of 0.9 over nearly 4000 students and 38 studies.  Teachers being purposeful to innovations in that they are looking to see "what works" and "why it works" as well as looking for reasons why students do not do well lead to improvement in instruction and student achievement.

My new thought:  This is the high-yield strategy that can really make a difference at our school, and through the Professional Learning Community Model of providing time for teachers to collaborate and reflect on teaching practices, we have seen a marked increase in the success of our students.

Again, these are just a very few of the highlights of this very important book, and if you are interested in challenging your own beliefs about education with the research that is out there, I think Hattie's "Visible Learning" is a must-read.  Furthermore, if you are interested in finding out "what works" in terms of high-yield strategies to maximize your staff development, I think this is an excellent beginning. Or, if you just want to be able to call BS on people who make loud, bold statements about education without having done the research, the research here is your BS repellent.

And that is no BS.


Hattie, J., (2009). Visible Learning: A synthesis of over 800 Meta-Analyses relating to achievement. Routledge, Oxon.