Supporting Families
  • banner8
  • banner3
  • banner10
  • banner13
  • banner6
  • banner7
  • banner9
  • banner11
  • banner4
  • banner12
Sunday, 07 May 2017 23:16

Comment on Report claiming 'The 11-plus is a loaded dice'

A story on the BBC website features a Report that offers misleading and irrelevant data relating the Kent grammar school selection process, issued by Education DataLab (EDL). EDL has built this on information collected by the nebulous Kent Education Network (KEN), the link underlining the misuse of statistics by KEN in its passionate opposition to the existence of grammar schools in the county, so hardly an objective source of data. The title of the Report, ‘The 11-plus is a loaded dice - Analysis of Kent 11-plus data’, is itself highly pejorative based on the false claim in the document that there is an arbitrariness in who passes the Kent Test, although no doubt designed to capture headlines.

Education Datalab describes itself as a research organisation that produces independent, cutting-edge analysis of education policy and practice. Employing Joanne Bartley from Kent Education Network as one of the authors of the Report completely destroys any claim to independence or objectivity in this case.  

The Report purports to make proposals to increase the proportion of children on Free School Meals being identified as suitable for grammar school, although this should surely have been indicated in the title, and some of its ideas would have precisely the opposite result. The central proposal of change to achieve this, by changing the assessment process for suitability to grammar school, would actually introduce an additional and unnecessary unfairness for all children into the process, and remove the current requirement for children to reach an appropriate standard in English and maths.

Kent County Council produced a highly regarded Report last year prepared by a Select Committee of KCC on 'Social Mobility and Grammar Schools' (SMGS) published last year, which contains a range of excellent Recommendations that have been adopted in full by the Council and are currently being worked through. These Recommendations in my view offer a far more rational way forward than the mixed bag contained here, although oddly the EDL Report contains no reference to them. However, as I can see no chance of the EDL Report being adopted for the reasons below, I am disappointed it has gained such publicity.

The BBC article highlights the statement that 12% of Free School Meals children passed the Kent Test in 2015 when compared with 30% of their better off classmates. Cleverly, the Report omits the important qualification 'of those children who took the test', rather than using data for all children in the age group, which exaggerates the differential, for in the 2015 Test, the one being 'analysed', just 20.0% of all Kent children in the age group passed the test as my analysis at the time shows. It is a false, misleading and irrelevant comparison as it takes no account of the relative abilities of the two groups of children, nor of the make up and proportions of the sub-groups who chose to apply for the Test. Instead, I look at an appropriate measure of success at gaining grammar school places for children on Free School Meals (FSM) below, taken from SMGS, and comparing children with similar abilities, which is important OBJECTIVE evidence and so should surely have been quoted in the Report but was not.

The central recommendations of the Report are:
  • Kent should consider implementing an aggregate pass mark in its 11 plus test to reduce the proportion of pupils, just one mark away from passing/failing;
  • Kent should consider removing the headteacher assessment panel and correspondingly lowering the pass mark on the test.
Replacement of the Kent Test by a once only aggregate score assessment
The EDL analysis proposes the unexamined and flawed assumption that selection for grammar school should be based on a single test performance of children with the highest aggregate scores, without considering the disadvantages of this scheme for children. Both these recommendations remove some of the key strengths of the Kent process without serious analysis of the latter. The Kent Test currently requires children to reach a standard in each of the three elements of the test, Reasoning, English and Maths, together with an aggregate score, currently 320. This ensures that grammar school entrants have a basic standard in English and mathematics. According to the EDL Paper this is apparently a BAD THING, although I can see no rationale for this provided.

The Report makes great play about the unfairness of some children with an aggregate score of 320 not getting a grammar school place as they have failed to show grammar school ability in one or more elements. Quite correct; that is the point of setting a minimum standard in each subject to ensure grammar school entrants have an appropriate background across the board and so able to meet the wide academic demands of the education offered in grammar schools. The conclusion about the Kent Test that: “The test is identifying children who are (highly able) all-rounders, then, rather than children with particular aptitude in only one or two areas” is clearly meant as a criticism, but is indeed at least partially correct, although I have seen many children in the second category with basic skills across the board awarded grammar school places, so is false in that respect.  

Headteacher Assessment
What the two proposals achieve together is to turn the Test into a ‘one chance only on the day’ Test, a concept already often erroneously alleged by KEN to be a criticism of the  current set-up. This is to be achieved by abolishing the Headteacher Assessment (HTA) aspect of the Kent Assessment system, which selects a further 4% of children on top of the 21% found of grammar school ability by the Kent Test.

HTA takes into account: children’s work; a piece of written English completed at the same time as the Kent Test; a reference from the Primary Headteacher; and the marks in the Kent Test. Most importantly it can also be used for the school’s headteacher to explain personal circumstances that have played a part in underperformance, which can feature any element of disadvantage. It therefore allows a second chance for those who may have underperformed in the Kent Test and for disadvantaged children. Instead the Report proposes replacing this by lowering the pass mark, which of course simply has the effect of re-introducing this same rightly hated ‘one chance only’ effect for a different group of children.  

The EDL Report acknowledges that HTA is biased in favour of children on Free School Meals showing that it does work for disadvantaged children, but then criticises it because it delivers insufficient numbers, so it is unclear why it wishes to abolish it. Surely, on this basis, the way forward would be to extend HTA. The Report quite rightly criticises some primary school headteachers for failing to offer appropriate support to children who ought to be put forward for HTAs, perhaps like KEN because they do not believe in the selective system. However, it is clear that selection in Kent is here to stay for many years and headteachers have a responsibility for ensuring the best for their pupils. I have worked with too many families where primary headteachers have failed their pupils in this way to be blasé about this problem. The solution is surely to challenge such schools by those in authority to ensure they make the system works effectively, not undermine it. Indeed, the SMGS identifies work with primary schools as a priority. This includes amongst the 15 sensible recommendations: “Urge all Primary Headteachers to utilise Headteacher Assessment Panels within the Kent Test process to advocate for those most academically able children supported by the Pupil Premium”, along with powers to do so. You will find my own comments here 

Some children will be being passed or failed incorrectly’
This is the heading of one section of the Report, and the article appears to suggest there is precision and accuracy in defining children’s ability, by placing them in a line according to their suitability for grammar school, which is totally opposed to so many statements by KEN! Having proposed this false precision, the Report then criticises the Test for failing to deliver it: ‘In fact, no 11-plus test will ever sort children perfectly, even if we were to ask 10-year-olds to sit a test every day for a whole month’.  It then goes on to analyse the problems of the incorrect assessments it has artificially created, by exploring what it claims are the probabilities of pass/fail in individual tests. Yes, there is a band in which borderzone children can be passed or failed by the Test, but grading them solely on a single mark is grossly unfair and has no element of correctness. With the current system, it is the function of the HTA or grammar school appeals to look more closely at these situations. Both these processes rightly depend on judgments by independent panels, but there should be no correct or incorrectness in judgments. 

The section concludes with a proposed surrealistic outcome for parents, whereby instead of the straightforward pass/fail decision they are provided with at present, they will receive a report from KCC with: ‘alongside the letter stating whether the child had passed the 11-plus, parents are given an additional piece of information – the probability that they have been misclassified by the test’, still obsessing about correctness. I just don’t get this, although the Report talks about a concept called Classification accuracy. The information might include, as per the examples given: ‘one parent might be told their child had passed, and yet the probability she should, in fact, have failed was 39%. Another would be told their child has failed, but the probability he should have passed was 47%’. Quite what the point of this is I cannot see, I don’t believe in it in this context, and what parents are supposed to make of it is completely beyond me. Subsequent school appeals take many important factors into account, and I really cannot visualise this one being of any conceivable interest to Panels.

The Reasoning Element of the Kent Test and Tutoring
Back in 2014, in an analysis of Kent Test scores, I identified that unsurprisingly the Reasoning element of the Kent Test was producing the highest scores although results are nationally standardised, presumably as a consequence of tutoring which is easier for this Test. This Report identifies that children on FSM are likely to perform slightly better in Reasoning than in Mathematics, using comparative performance against non FSM children. This gives the headline figures that FSM pupils taking the Kent Test underperform other children by 7.7 points in the Reasoning Test as against 6.8 points in the maths (the figure for English is 3.9) which the Report considers highly significant. Another two tricks of statistics; firstly, these are figures comparing numbers greater than one hundred, so the difference between the two subjects is less than 1%. Secondly, for some reason, again presumably for headline purposes, it emphasises once again that FSM pupils taking the Kent Test achieve less well than non FSM pupils overall, which still proves nothing either way, as there is no comparison of relative abilities or of the way these groups self-select to take the Test.
 
 
Performance of Children on Free School Meals
As noted above, the statement that 12% of Free School Meals children passed the Kent Test in 2015 when compared with 30% of their better off classmates may well be true but it is a false, misleading and irrelevant comparison. The best comparison I can find is that is that 57.4 % of children on Free School Meals Ever (my preferred measure) who have achieved at least two Level 5’s in their KS2 SATs begin grammar school, against 78.7% of similar ability children not in this category. This of course comes from the SMGS Report, also looking at 2015 entry, when the Level 5 concept still applied. Yes, there is an unacceptable gap between these children of similar abilities, but KCC is working hard to close it in conjunction with many primary and grammar schools, as shown by examples already happening in the Report. Why no credit for these, or practical advice on how to do so?
 
Test Preparation in Primary Schools
There is one good recommendation in the Report that I would like to see KCC adopt, which is to allow all primary schools to provide up to 10 hours Test preparation for all children who wish it. This is also the level of practice recommended in the only piece of research I recall having seen into the effect of coaching, some years ago. I don’t believe it can take place in class, as it is only relevant for a proportion of children, who are self-selected, but could easily be offered after school as a voluntary activity and I can see no legal reason to stop this. This would help to reduce the disparity with those who are privately tutored or attend those private schools geared to secure success in the Kent Test.
 
The Gambling Analogy
Even a superficial reading of this report confirms that most children are appropriately identified by a combination of the two main routes of selection. The lead claim of the report is that ‘We say that getting into a grammar school in Kent is akin to rolling a dice because of the arbitrariness of who passes the test’. The use of the word arbitrary, claiming that the process is valueless to get a good headline, is completely wrong, foolish, and offensive to those children selected. The introduction of the concept of rolling a dice to emphasise that devaluation may well come form the the gambling industry, but it has no place in what claims to be an objective analysis. What it does do is to generate false headlines such as the one in The Times that reads: ‘Biased 11-plus is no reflection of ability’, which could of course have been its intent. Even a superficial analysis of this headline shows its error. Most definitions of bias refer to prejudice and a deliberate action to support a point of view. The Report itself contains no suggestion of intent in any unfairness in the system, so bias is an inappropriate word. ‘No reflection of ability’  is self-evidently just plain wrong!
 
Grammar School Provision
The Report claims that Kent has 11 super selective schools out of the 32 in the county, defined those ‘which make use of 11-plus test scores to prioritise applicants for admission, either ranking all applicants by score, or prioritising those who have scored above a given level’. By this definition, I can count just 8: Judd, Skinners and Tonbridge; Dartford Boys and Girls; Cranbrook; and Maidstone and Simon Langton Boys Grammars. Perhaps just a careless error, but again, one that becomes replicated in media that have not carried out an elementary check on accuracy.
 
Half of these - Dartford Girls, Judd, Skinners and Tonbridge Grammars - have introduced priority for a proportion of Pupil Premium or Free School Meals children, as have a number of other grammar schools, and I anticipate this will extend further for admission in 2019. 

It’s also worth noting that passing the 11-plus is not enough alone to gain entry to any grammar school of choice’. Of course it is not! The outcome with regard to places is no different for the rules for any oversubscribed school of any type, so why suggest it is different and of interest unless one is showing bias, but succeeding in further discrediting the Report.  

Meopham Grammar School?
Amongst the plethora of data (along with a multitude of follow-ups) that Joanne Bartley collects from Local Authorities and schools across the country regarding selection to grammar schools, there is an important question to which I also would like to know the answer although it is irrelevant to this article. Back last September, the Swale Academy Trust proposed changing the status of Meopham School in Gravesham to become a grammar school when and if the rules changed and instituted a Consultation. She has vigorously  pursued the results of this Consultation and indeed was promised a Report back on January 18th. Since then the Trust has illegally ignored her further requests for the information. What are they trying to hide? Perhaps the ridiculous nature of the initial proposal as demonstrated in my article written at the time. 
 
Conclusion
I have only reported on parts of a very lengthy Report that attempts to blind readers by the complexity of analysis. Many of the illustrations I have given above make no sense, illustrate false conclusions and appear counter-productive in terms of narrowing the gap for FSM children looking for places at grammar schools. If you have read this far, please consult the SMGS Report that contains many excellent and realistic proposals for closing the gap, from  a Local Authority committed to do so, some of which are coming to fruition. Sadly, it is fair to note that there are a number of institutions both at primary and secondary level, who have no interest in promoting these, and for whom there appears no sanction to force them to do so.

Given that the New Grammar School policy now looks likely to come to fruition with a Conservative victory in the forthcoming election (I have grave reservations about the policy), I presume this analysis is ostensibly designed to advise the new institutions on the way forward. However, it is clear they will be of a different character to our present grammar schools with a built-in commitment to supporting disadvantaged children, so little here will apply. Kent is a county with grammar schools operating a single base system of admission, although there are a variety of add-on differences. This is not transferable to individual institutions with individual selection processes operating to the new rules within a comprehensive school set-up. However, it may serve to warn those institutions of the perils of its recommendations. Finally, I consider the analysis is far too flawed, with too many false conclusions and errors to be useful. In practice it is clear there is a different agenda. 

I can see that it has whipped up considerable anti-grammar school feeling in parts of the media which may well have been its aim. Typical is the report in The Times, which claims amongst other misinterpretations that: ‘Only children doing exceptionally well in all three papers will be given a place at grammar schools’, which is clearly a nonsense.  I make no judgment about the rights or wrongs of the selective system here, I have simply looked at the facts.

I operate alone and part-time, producing this in short time, in order to respond to the issues raised, but I accept to late to influence them. As a result, I also accept that I may well have made errors myself in this analysis, and if so am happy to correct them.

Last modified on Saturday, 20 May 2017 08:43

4 comments

  • Comment Link Saturday, 20 May 2017 09:14 posted by David Miham

    Dear Peter

    Thank you for your detailed article on the 11 plus selection process in Kent.

    As a governor of two primary schools serving areas of East Kent with high levels of deprivation - both schools have over 55% pupil premium - where over recent years a number of pupil premium children achieving KS2 SATS results in the top 25% nationally have not transferred to Grammar Schools. Whilst the Headteachers do everything they can to encourage parents to apply for their child to be entered for the Kent test if the parents do not engage with the school or do not want their child to take the test for fear of the damage failure to "pass" will have or think their child will not fit into what they think is an elitist school. The school cannot force the parent to apply for the child to take the test however able the child or how suitable the child is for a Grammar school education.

    The timing of the test immediately after the summer holidays also puts children from poor backgrounds at a disadvantage - the coaching industry in Kent is such that the majority of middle class parents pay for coaching months or even years before they take the test and certainly during the summer holidays. Low income families cannot afford the cost and Kent schools are not allowed to offer coaching. Whilst KCC claim the test is coaching proof there is no evidence to substantiate - the fact that the child has continued to work during the holidays is likely to be of benefit.

    The Kent test is not 11 plus - children born June to August are only just 10 when they take the test. The Headteacher Appeal system works well to secure places for children who miss the mark in one of the papers where there school work demonstrates they have the ability to benefit from a selective school education.

    One possible option would be for a limited number of places at Grammar schools be held open for those children that perform well in their KS2 SATS and where Headteachers can show good progress and attainment in the child's work in year 6 to be offered a place in July. This would enable those Pupil premium children who were not entered for the Kent test to progress to a Grammar school.

    Kind regards

    David PETER: Thank you for this, which makes some excellent points. However, this article is primarily about a shoddy and tainted piece of research, which has received undue attention without challenge elsewhere. I agree that the majority of headteachers, including those of your two schools, whether or not they believe in selection, are working to reduce the gap for this is the system we have in Kent, but it is unrealistic for the reasons you quote to assume it can be closed. Nevertheless, the work on minimising it can and should be continued. One of the biggest challenges remains to change the approach in those schools that will do nothing, or in some cases actively harm the the opportunities and life chances of these children.

  • Comment Link Monday, 08 May 2017 23:29 posted by James Rogers

    You refer to the 'Gambling Analogy' in your brilliant exposé. Did you know that Joanne Bartley was previously the Marketing Director of an Online Gambling Company according to Linkedin? Perhaps that is where she got the idea.

  • Comment Link Monday, 08 May 2017 22:00 posted by Joanne Bartley

    Dear Peter,

    As you pointed out I was listed as one of three authors on the Education Datalab report. My role had two parts.

    1) To provide the data - as you noted I make FOI requests for this. We two are probably the only people in Kent keeping any sort of check on test data. KCC do not look at it as a matter of course.

    2) To give information on the test process in Kent, which is complex and they wanted to understand it. The only factual inaccuracy with the process you noted was based on '11 super-selectives'. We used KCC's definition of super-selective for this, based on the report that you rate so highly.

    The author of the entire text of the report was the lead for this study, Dr.Becky Allen. She was assisted by other statisticians at Education Datalab. https://educationdatalab.org.uk/team/

    I had no say on any outcomes, conclusions or headlines. As you pointed out some thoughts here are not in line with KEN's views. That's because the researchers made their own minds up.

    The Education Datalab research organisation is highly respected, which is why their work gained so much press. Although of course you are entitled to dispute their findings and disagree with their opinions of Kent's test process.

    I urge people to read the report in full and judge it themselves. I think the graphs are particularly telling, especially the one on the oddness of who gets through HTA and who does not. It surprised me anyway.

    In May 2016 Cllr Whittle said that she believed 600 additional pupil premium children would gain access to grammar schools based on the SMGS report's recommendations. So far this hasn't happened, and we shall see for ourselves in a year or two how much worth there is to the council's report. Personally I found the recommendations rather weak.

    KEN has made some simple recommendations to KCC based on greater openness and transparency around the Kent Test. We would like annual reports on the number of disadvantaged pupils passing the test, and simple reporting of test accuracy based on pupil's eventual GCSE outcomes.

    KCC should have nothing to fear by publishing such information publicly, and we hope it will allow us to see the accuracy of the test, and watch the % of disadvantaged pupils in grammars increase significantly.

    Regards,

    Joanne Bartley
    PETER: Just two comments. Jenny Whittle's 600 FSM pupils are across the whole secondary range, so some 80 per year. These would not be additional, but part of the 25% currently offered places changing the admission profile and will take more than your couple of years I am afraid, as 2018 admission priorities are already set. We are seeing a promising start to this aim with Judd, Skinners, Tonbridge, Weald and Dartford Girls all reserving some places for FSM or Pupil Premium children either now or for 2018. So there is certainly progress in West Kent, which might have been regarded as the most difficult area. This pattern is also being replicated in other parts of the county. .Secondly, I do agree with you about publication of this data, and will certainly follow it through myself on this website.

  • Comment Link Monday, 08 May 2017 18:14 posted by Fair Play

    Thanks for your demolition of an appalling Report. How can two professional and supposedly independent researchers produce such a flawed and biased document. Answer: Joanne Bartley, Answer: Joanne Bartley the third and, from the nature of the Report, dominant author. What on earth was Education DataLab doing letting her become joint author, although from the tone of the document, she was actually the main writer. Utterly destroys their claim to be objective.

Leave a comment

Make sure you enter the (*) required information where indicated.
Basic HTML code is allowed.