Part of the arrangements to be put in place for the summer exam series in 2022 is to give AS and A level students advance notice of some of the question topics in their exams.
An anonymous 'source' claimed in an article in the inews on 7 January 2022 that:
"The emphasis is going to be on higher tariff questions. The questions that earn the most marks and therefore differentiate at the higher level are the ones where the biggest attention is going to be given, so that’s where they’re going to give some sense of what kind of questions will be asked and what kind of topics. The trouble with doing that is the kids at the lower end who may well have been most disadvantaged by all of this, actually again could be doubly disadvantaged."
There is an assertion that releasing information about the topics of only the high-tariff questions will ‘doubly disadvantage’ lower attaining students, but no reasons are given, apart from a suggestion that it follows by implication from the claim that higher tariff questions ‘differentiate at the higher level’.
In this blog we look at some data to see whether this claim is correct. As a measure of attainment we take the total score on an exam paper. A plot of question score (smoothed) against total score can help us to consider (for that particular question) what the effect of releasing information about the topic might be.
Figure 1 below shows the plots of the easiest and hardest six-mark questions on a paper that consisted mostly of short-answer questions worth 1-3 marks, along with a few longer questions worth 6-8 marks. The red dashed line shows the harder question and the blue solid line shows the easier one. The steepness of the curve shows where the question is ‘differentiating’ the most, and for both these questions we can see that they have similar slopes that indicate similar differentiation across most of the mark range.
Figure 1: Relationship between question mark and total mark for the easiest and hardest 6-mark questions on an exam paper.
Figure 2: Relationship between question mark and total mark for the easiest and hardest 1-mark questions on an exam paper.
Compare Figure 1 with Figure 2, which plots the equivalent curves for the easiest and hardest 1-mark questions on the same paper. The curves are a long way apart, and the steep part of the curve is in a very different part of the mark range. The hardest question differentiates most at the higher scores (as we’d expect) and the easiest one differentiates most at the lower scores.
The effect of making any question easier (e.g. by giving advance notification of the topic) is to shift the curve to the left by a certain amount. The students who would benefit the most from this are the students whose attainment corresponds to the steepest part of the curve, wherever that happens to be. Making a very easy question easier most benefits the very lowest attaining students (because the others are already getting the marks on it). Making a very hard question easier most benefits the highest attaining students (because it is still too difficult for most other students to gain marks on).
The steeper the slope, the narrower the range of attainment in which there will be a benefit. (The steepness of the slope reflects a property of questions known as ‘discrimination’; the location of the curve to the left or the right reflects its ‘difficulty’).
Exam questions differ in both difficulty and discrimination. Even among questions with the same tariff (number of marks) there can be variation in what these plots look like. However, there is usually less variation in the plots for higher tariff questions. This can be thought of in terms of a simple averaging process – if we imagine a 6-mark question being made up of six 1-mark questions, it is possible that the hardest and easiest of those 1-mark questions could look like the curves in Figure 2. However, combining them will tend to produce something more like the curves in Figure 1.
If the higher attaining students are mostly getting full marks on a particular question, then making it easier (e.g. by giving students advance warning that its topic will be on the paper) can only benefit the lower attaining students. Therefore, the claim in the inews article would only be valid about the marks gained by students if it were also true that all or most higher tariff questions were also the most difficult.
If the test scores in our example are broken into two subsets – the score on the low-tariff questions and the score on the high-tariff questions, we get the plot in Figure 3. This shows that in this particular case the higher tariff questions were indeed slightly more difficult on the whole (because the red line is to the right of the blue line). However, the red line has a fairly constant slope over most of the mark range - so there would be no particular advantage to higher attaining students of making these questions easier.
Figure 3: Relationship between part-test mark and total mark for the low tariff and high tariff questions on an exam paper.
Furthermore, even if it were the case that it became relatively easier for higher attaining students to gain marks on some questions, the way that grade boundaries are determined would compensate for this. Grade boundaries are not constrained all to move by the same amount or in the same direction compared with previous exam papers. It is quite possible (for example) for a lower grade boundary to remain at the same mark while a higher grade boundary rises by a mark or two, thus compensating for any extra easiness at the higher-attaining end of the distribution.
There may of course be reasons why higher attaining students might be advantaged by advance notice of topics, for example if certain topics are only taught, or only taught well, to the higher attaining students. But even in this case, it seems more likely that advance notice would benefit lower attaining students. One can imagine in normal times a situation where a difficult topic that rarely appears on an exam paper does not get taught thoroughly to the lower attaining students – but the certainty that the topic will appear might actually ensure that these students are better prepared for it.
To conclude, giving advance information about higher tariff questions will not ‘doubly disadvantage’ lower attaining students – at least not purely on the basis of having a higher tariff.