Tom Benton

Benton, T., Gill. T., Hughes, S., & Leech. T. (2022). A summary of OCR’s pilots of the use of Comparative Judgement in setting grade boundaries. Research Matters: A Cambridge University Press & Assessment publication, 33, 10–30.

The rationale for the use of comparative judgement (CJ) to help set grade boundaries is to provide a way of using expert judgement to identify and uphold certain minimum standards of performance rather than relying purely on statistical approaches such as comparable outcomes. This article summarises the results of recent trials of using CJ for this purpose in terms of how much difference it might have made to the positions of grade boundaries, the reported precision of estimates and the amount of time that was required from expert judges.

The results show that estimated grade boundaries from a CJ approach tend to be fairly close to those that were set (using other forms of evidence) in practice. However, occasionally, CJ results displayed small but significant differences with existing boundary locations. This implies that adopting a CJ approach to awarding would have a noticeable impact on awarding decisions but not such a large one as to be implausible. This article also demonstrates that implementing CJ using simplified methods (described by Benton, Cunningham et al, 2020) achieves the same precision as alternative CJ approaches, but in less time. On average, each CJ exercise required roughly 30 judge-hours across all judges.

Benton, T. (2021). Item response theory, computer adaptive testing and the risk of self-deception. Research Matters: A Cambridge University Press & Assessment publication, 32, 82-100.

Computer adaptive testing is intended to make assessment more reliable by tailoring the difficulty of the questions a student has to answer to their level of ability. Most commonly, this benefit is used to justify the length of tests being shortened whilst retaining the reliability of a longer, non-adaptive test.

Improvements due to adaptive testing are often estimated using reliability coefficients based on item response theory (IRT). However, these coefficients assume that the underlying IRT model completely fits the data. This article takes a different approach, based on comparing the predictive value of shortened versions of real assessments based on adaptive and non-adaptive approaches. The results show that, when explored in this way, the benefits from adaptive testing may not always be quite a large as hoped.

Benton, T. (2021). On using generosity to combat unreliability. Research Matters: A Cambridge Assessment publication, 31, 22-41.

Assessment reliability can be affected by various types of unforeseen events. In any such circumstances where a concern is raised that the reliability of assessment is lower than usual, our natural inclination is to allow extra leniency in grading to reduce the chances of students missing out on a grade they deserve. This article shows how, by focusing on the risk for individual students, we might logically approach this situation in deciding exactly how much additional generosity is required. In particular, it shows how making progress with this problem requires an acceptance that no assessment system is perfect and transparency about the level of reliability that is achievable. Having developed an approach, this article also shows how this may lead to different outcomes than the competing desire to maintain assessment standards so that the group of students in question are not unfairly advantaged relative to previous and future cohorts.

Benton, T. (2020). How useful is comparative judgement of item difficulty for standard maintaining? Research Matters: A Cambridge Assessment publication, 29, 27-35.

This article reviews the evidence on the extent to which experts’ perceptions of item difficulties, captured using comparative judgement, can predict empirical item difficulties. This evidence is drawn from existing published studies on this topic and also from statistical analysis of data held by Cambridge Assessment. Having reviewed the evidence, the article then proposes a simple mechanism by which such judgements can be used to equate different tests, and evaluates the likely accuracy of the method. 

Benton, T. (2019). Which is better: one experienced marker or many inexperienced markers? Research Matters: A Cambridge Assessment publication, 28, 2-10.

For many practical purposes, it is often assumed that the quality of a marker is directly related to their seniority. At its extreme, the assumption is that the most senior marker (the Principal Examiner) is always right even in cases where large numbers of junior markers have a collectively different opinion regarding the mark that should be awarded to a given script. To investigate this assumption, this article compares the predictive value of marks provided by the most senior marker to those made by simply taking the mean mark awarded by many junior markers. Predictive value was estimated via the correlation of the scores assigned to scripts with the overall achievement of the same candidates in other exam papers taken within the same month. By looking at the relative predictive value of the two different sources of marks, we can begin to make some inferences about the extent to which senior markers are genuinely more accurate than their junior colleagues.

Benton, T. and Gallacher, T. (2018). Is comparative judgement just a quick form of multiple marking? Research Matters: A Cambridge Assessment publication, 26, 22-28.

This article describes analysis of GCSE English essays that have both been scored using comparative judgement and marked multiple times. The different methods of scoring are compared in terms of the accuracy with which the resulting scores can predict achievement on a separate set of assessments. This results show that the predictive value of marking increases if multiple marking is used and (perhaps more interestingly) if statistical scaling is applied to the marks. More importantly, the evidence in this article suggests that any advantage of comparative judgement over traditional marking can be explained in terms of the number of judgements that are made for each essay and by the use of a complex statistical model to combine these. In other words, it is the quantity of data that is collected about each essay and how this data is analysed that is important. The physical act of placing two essays next to each other and deciding which is better does not appear to produce judgements that are in themselves any more valid than from getting the same individual to simply mark a set of essays.

Benton, T. (2018). How many students will get straight grade 9s in reformed GCSEs? Research Matters: A Cambridge Assessment publication, 25, 28-36.

This article describes an attempt to predict the number of students that will achieve straight grade 9s in reformed GCSEs. The prediction is based upon an analysis of unreformed GCSE and international GCSEs from the cohort completing Key Stage 4 in 2016. Several methods are applied and evaluated against the number of candidates known to have achieved straight grade 9s in the three subjects that were reformed before examination in summer 2017. The results suggest that, of those candidates taking at least 8 GCSEs, between 200 and 900 will achieve straight grade 9s. Furthermore, we predict that more than 2,000 students will achieve a perfect score in their Attainment 8 accountability measure.

Benton, T. (2017). How much do I need to write to get top marks? Research Matters: A Cambridge Assessment publication, 24, 37-40.

This article looks at the relationship between how much candidates write and the grade they are awarded in an English Literature GCSE examination. Although such analyses are common within computer-based testing, far less had been written about this relationship for traditional exams taken with a pen and paper. This article briefly describes how we estimated word counts based on images of exam scripts, validates the method against a short answer question from a Biology examination, and then uses the method to examine how the length of candidates’ English Literature essays in an exam relate to the grade they were awarded. It shows that candidates awarded a grade A* wrote around 700 words on average in a 45-minute exam - an average rate of 15-words per minute across the period. In contrast, grade E candidates who produced around 450 words - an average rate of 10-words per minute. Whilst it cannot be emphasised strongly enough that performance in GCSEs is judged by what students write and not how much, the results of this research may help students facing examinations have a reasonable idea of the kind of length that is generally expected.

Benton, T. (2017).  The clue in the dot of the ‘i’: Experiments in quick methods for verifying identity via handwriting. Research Matters: A Cambridge Assessment publication, 23, 10-16.

This article demonstrates some simple and quick techniques for comparing the style of handwriting between two exams. This could potentially be a useful way of checking that the same person has taken all of the different components leading to a qualification and form one part of the effort to ensure qualifications are only awarded to those candidates that have personally completed the necessary assessments. The advantage of this form of identity checking is that it is based upon data (in the form of images) that is already routinely stored as part of the process of on-screen marking. This article shows that some simple metrics can quickly identify candidates whose handwriting shows a suspicious degree of change between occasions. However, close scrutiny of some of these scripts provides some reasons for caution in assuming that all cases of changing handwriting represent the presence of imposters. Some cases of apparently different handwriting also include aspects that indicate they may come from the same author. In other cases, the style of handwriting may change even within the same examination response.

Benton, T. (2016). Paper presented at the AEA-Europe annual conference, Limassol, Cyprus, 3-5 November 2016

Dunn, K., Darlington, E. and Benton, T. (2016). Revisiting the topics taught as part of an OCR History qualification. Research Matters: A Cambridge Assessment publication, 22, 2-8.

Given the introduction of a broader range of options in OCR's new A level history specification, this article follows on from a previous analysis of A level History options based on the previous specification for OCR History (Specification A). That research relied on OCR History centres responding to requests for participation in an online survey. However, OCR’s introduction of an online ‘specification creator’ tool for centres has provided quantitative information about the topics which schools intend to teach their students as part of their A level.  As with the previous study, we sought to establish what the common topic choices and combinations are.

Benton, T. (2016). On the impact of aligning the difficulty of GCSE subjects on aggregated measures of pupil and school performance. Research Matters: A Cambridge Assessment publication, 22, 27-30.

It is empirically demonstrated that adjusting aggregated measures of either student or school performance to account for the relative difficulty of General Certificate of Secondary Education (GCSE) subjects makes essentially no difference. For either students or schools, the correlation between unadjusted and adjusted measures of performance exceeds 0.998. This indicates that suggested variations in the difficulty of different GCSE subjects do not cause any serious problems either for school accountability, or for summarising the achievement of students at GCSE.

Zanini, T. & Benton, T. (2015). The roles of teaching styles and curriculum in Mathematics achievement: Analysis of TIMSS 2011. Reserach Matters: A Cambridge Assessment publication, 20, 35-44.

This article provides empirical evidence about the link between Mathematics achievement, curriculum, teaching methods and resources used in the classroom. More specifically, this research explores common teaching styles and topics taught across countries with respect to their Mathematics achievement. In order to do so, we make use of the fifth TIMSS survey, which provides a rich set of information regarding aspects of the curriculum (e.g., the emphasis on problem solving and interpreting data sets), resources used by teachers in the classroom (e.g., calculators and textbooks) and teaching styles (e.g., how often students are asked to take written tests, to work out problems individually rather than with teachers' guidance), along with measures of achievement in Mathematics gathered in 2011. Although TIMSS is administered to students and their teachers in both Grades 4 and 8 (Years 5 and 9 respectively, within England), analysis in this research is restricted to the Grade 8 students (aged 14). When analysing data aggregated at jurisdictional level, this allows us to explore relationships in the Mathematics achievement of 15 year-olds as measured by PISA 2012.

Benton, T. and Bramley, T. (2015). Cambridge Assessment Research Report. Cambridge, UK: Cambridge Assessment.

Benton, T. (2014). Examining the impact of tiered examinations on the aspirations of young people. Research Matters: A Cambridge Assessment publication, 17, 42-46.

Tiered examinations are commonly employed within GCSE examinations in the UK. They are intended to ensure that the difficulties of exam papers are correctly tailored to the ability of the candidates taking them; this should ensure more accurate measurements and also a better experience for candidates as they do not spend time addressing questions that are either too easy or too difficult given their level of skill. However, tiered examinations have also been criticised for potentially damaging the aspirations of young people entered for lower tier examinations by placing a limit on the grades they can achieve. This article explores the extent of the link between GCSE entry tier and aspirations and also investigates the extent to which this link can be explained by differences in achievement and background characteristics of pupils.

The research makes use of data available from the Longitudinal Study of Young People in England (LSYPE) linked to information available from the National Pupil Database (NPD) regarding the qualifications achieved by pupils and also their entry tier at GCSE. Analysis was completed using a combination of multilevel modelling and propensity score matching and showed that differences in aspirations between pupils entering different tiers can almost entirely be explained by differences in background characteristics.

Benton, T. (2013) Paper presented at the 78th Annual Meeting of the Psychometric Society, Arnhem, The Netherlands, 22-26 July 2013