28 March 2022
New research from Cambridge University Press & Assessment reports how UK exam board, OCR, used comparative judgement for the first time to help set grade boundaries for several GCSEs and A Levels in autumn 2020. This novel approach to setting grade boundaries required senior examiners to systematically compare the quality of student work between different exam series over time. OCR is now considering what further testing is required before deciding whether to adopt the approach in future exam series.
The potential for comparative judgement to replace conventional exam marking has been the subject of increasing sector interest, but it also has the potential to help maintain a test standard over time – so it is no harder or easier to achieve a given grade in one year than another. The potential benefit of using comparative judgement for capturing expert judgement for the purpose of standard maintaining is an area that Ofqual, the exams regulator for England, has previously described as ‘very promising’. The main benefit of comparative judgement is the systematic comparison of the content and quality of student scripts from different exam series to determine the position of grade boundaries, focusing on the quality of responses rather than other sources of information, such as previous grade boundaries or statistical data that provide more of a confirmatory approach to their determination.
OCR used comparative judgement as the primary source of evidence to set grade boundaries in three qualifications taken in the autumn 2020 exam series – GCSE English Language, and A Levels in English Literature and Psychology. It is believed this was the first time comparative judgement has been used in this way. These qualifications were chosen because the usual statistical sources of evidence for setting grade boundaries were not available and because previous evidence had suggested comparative judgement could provide an effective approach to maintaining standards. A write-up of the approach, published in the latest edition of Research Matters, shows that grade boundaries identified using comparative judgement in other circumstances can be shown to be similar to those identified using existing techniques. It also finds that a ‘simplified’ comparative judgement approach can achieve similar or better accuracy in less time than previous ways of conducting comparative judgement. That said, the authors conclude that more research is needed to confirm the benefits of its use, particularly in more objectively marked subjects, like maths or science.
Dr Frances Wilson, Principal Researcher, OCR, said: "Our research shows that there is great potential for comparative judgement to play a significant role in awarding GCSEs, A Levels and other qualifications in future. We will continue to move cautiously, undertaking more research to confirm the benefits or otherwise of comparative judgement over established approaches to maintaining standards in qualifications."