Tim Gill

Gill, T. (2022). The concurrent validity of comparative judgement outcomes compared with marks. Research Matters: A Cambridge University Press & Assessment publication, 33, 68–79.

In Comparative Judgement (CJ) exercises, examiners are asked to look at a selection of candidate scripts (with marks removed) and order them in terms of which they believe display the best quality. By including scripts from different examination sessions, the results of these exercises can be used to help with maintaining standards.

Results from previous CJ studies have demonstrated that the method appears to be valid and reliable in many contexts. However, it is not entirely clear whether CJ works as well as it does because of the physical and judgemental processes involved (i.e., placing two scripts next to each other and deciding which is better based on an intuitive, holistic, and relative judgement), or because CJ exercises capture a lot of individual paired comparison decisions quickly. This article adds to the research on this question by re-analysing data from previous CJ studies and comparing the concurrent validity of the outcomes of individual CJ paired comparisons with the concurrent validity of outcomes based on the original marks given to scripts.

The results show that for 16 out of the 20 data sets analysed, mark-based decisions had higher concurrent validity than CJ-based decisions. Two possible reasons for this finding are: CJ decisions reward different skills to marks; or individual CJ decisions are of lower quality than individual decisions based on marks. Either way, the implication is that the CJ method works because many individual paired comparison decisions are captured quickly, rather than because of the physical and psychological processes involved in making holistic judgements.

Benton, T., Gill. T., Hughes, S., & Leech. T. (2022). A summary of OCR’s pilots of the use of Comparative Judgement in setting grade boundaries. Research Matters: A Cambridge University Press & Assessment publication, 33, 10–30.

The rationale for the use of comparative judgement (CJ) to help set grade boundaries is to provide a way of using expert judgement to identify and uphold certain minimum standards of performance rather than relying purely on statistical approaches such as comparable outcomes. This article summarises the results of recent trials of using CJ for this purpose in terms of how much difference it might have made to the positions of grade boundaries, the reported precision of estimates and the amount of time that was required from expert judges.

The results show that estimated grade boundaries from a CJ approach tend to be fairly close to those that were set (using other forms of evidence) in practice. However, occasionally, CJ results displayed small but significant differences with existing boundary locations. This implies that adopting a CJ approach to awarding would have a noticeable impact on awarding decisions but not such a large one as to be implausible. This article also demonstrates that implementing CJ using simplified methods (described by Benton, Cunningham et al, 2020) achieves the same precision as alternative CJ approaches, but in less time. On average, each CJ exercise required roughly 30 judge-hours across all judges.

Gill, T. (2019). Methods used by teachers to predict final A Level grades for their students. Research Matters: A Cambridge Assessment publication, 28, 33-42.

This research used a survey to investigate how Chemistry, English Literature and Psychology teachers go about the process of estimating their students’ A level grades. There are a variety of different sources of information available to help teachers, including statistically based predictions (e.g., ALIS), performance in previous assessments (e.g., GCSEs) or in-class assessments, and their own judgements of students’ motivation, interest and resilience.  Teachers were also asked to provide grade estimates for their current A level students and these were then compared with actual grades to provide an indication of accuracy. Follow up interviews were undertaken to elicit more detail about the process of making estimates, and to ask teachers about specific students who either under or over-performed compared to their estimate.  

The results will be discussed in the context of recent reforms to A levels (e.g., de-coupling of AS levels), which are likely to have had an impact on how teachers make their estimates and how accurate they are.  

Gill, T. (2018). How have students and schools performed on the Progress 8 performance measure? Research Matters: A Cambridge Assessment publication, 26, 28-36.

The new league table measures (Attainment 8 and Progress 8) are based on performance in a student’s best eight subjects at GCSE (or equivalent). One criticism of the previous measures was that they penalised schools with a low-attaining intake. As Progress 8 is a value-added measure, it already accounts for the prior attainment of the student and should in theory no longer penalise these schools. The purpose of this research was to delve deeper into the relationship between Progress 8 scores and various student and school level factors. In particular, multilevel regression modelling was undertaken to infer which factors were most important in determining scores at student level. The results showed that various groups of students were predicted higher Progress 8 scores including girls, less deprived students, students without SEN and students in schools with a higher performing intake.   At the school level, higher Progress 8 scores were found amongst schools with higher-performing intakes. This suggests that one of the main aims of the new measures (levelling the playing field) has not been completely achieved.

Gill, T. (2018). How have students and schools performed on the Progress 8 performance measure? Presented at the annual conference of the British Educational Research Association, Northumbria University, Newcastle, UK, 11-13 September 2018.

Video link: https://www.youtube.com/watch?v=0qEtOfJAiMs

Gill, T. (2017). An analysis of the effect of taking the EPQ on performance in other Level 3 qualifications. Research Matters: A Cambridge Assessment publication, 23, 27-34.

The Extended Project Qualification (EPQ) is a stand-alone qualification taken by sixth form students. It involves undertaking a substantial project, where the outcome can range from writing a dissertation or report to putting on a performance.  It is possible that some of the skills learnt by students whilst undertaking their project (e.g. independent research, problem-solving) could help them in other qualifications taken at the same time. Two separate investigations were undertaken: firstly, the performance of individual students was analysed, using a multilevel regression model to compare EPQ and non-EPQ students. The results showed that there was a small, but statistically significant effect, with those taking EPQ achieving better results on average in their A levels. The second investigation analysed performance at school level, using a regression to model the effect of increasing the percentage of students in a school taking EPQ. The results showed a significant and positive effect of increasing the percentage of students taking EPQ. However, the effect was very small.

Gill, T. (2016). Assessing the equivalencies of the UCAS tariff for different qualifications. Research Matters: A Cambridge Assessment publication, 21, 16-23.

In the United Kingdom (UK) the Universities and College Admissions Service (UCAS) provides the application process for most universities. The UCAS tariff points system is used by universities to help them select students for their courses. Each grade in eligible qualifications is allocated a points score,which can then be summed in order to provide an overall points score for each student. The allocation of points is such that,in theory,students with the same overall points score gained from different qualifications can be considered to be of equivalent ability or potential. The purpose of this article is to test whether this assumption works in practice, by calculating empirical equivalencies of the UCAS tariff for different qualification.

Gill, T. (2015). Using generalised boosting models to evaluate the UCAS tariff. Research Matters: A Cambridge Assessment publication, 20, 2-6.

The Universities and Colleges Admissions Service (UCAS) is a UK-based organisation providing the application process for almost all British universities. The UCAS tariff points system is used by universities to help select students for entry to their courses. Each grade in a qualification has a certain number of UCAS points allocated to it, which are then summed to provide an overall tariff points score for each student. The assumption made is that two students with the same UCAS tariff gained from different qualifications are of the same ability, or have the same potential to achieve at university. This article uses a statistical technique known as generalised boosting models (GBMs) to evaluate the use of the UCAS tariff as a predictor of degree outcome.

Gill, T. (2014) Statistics Report Series No. 56 (revised)

Emery, J., Gill, T., Grayson, R. and Vidal Rodeiro, C. L. (2012). Cambridge Assessment Statistics Reports: Recent highlights. Research Matters: A Cambridge Assessment publication, 14, 45-50.

The Research Division publishes a number of Statistics Reports each year based on the latest national examinations data. These are statistical summaries of various aspects of the English examination system, covering topics such as subject provision and uptake, popular subject combinations, trends over time in the uptake of particular subjects and the examination attainment of different groups of candidates. The National Pupil Database (NPD) is the source of most of these reports. This is a very large longitudinal database, owned by the Department for Education, which tracks the examination attainment of all pupils within schools in England from their early years up to Key Stage 5 (A level or equivalent). Another database, the Pupil Level Annual School Census (PLASC), can be requested matched to the NPD. This contains background information on candidates such as deprivation indicators, language, ethnicity and special educational needs. Other sources of data used to produce the Statistics Reports include the Inter-Awarding Body Statistics produced by the Joint Council for Qualifications (JCQ). This article highlights some of the most recent Statistics Reports, published between 2010 and 2011.

Black, B. and Gill, T. (2011). Does doing a critical thinking A level confer any advantage for candidates in their performance on other A levels? Research Matters: A Cambridge Assessment publication, 11, 20-24.

Critical Thinking AS level was introduced in schools in 2001. There is much research that shows that the teaching of Critical Thinking (CT) does improve critical thinking skills. However, there is less research which shows whether CT skills can be profitably transferred to other subject domains. The purpose of this research was to investigate whether taking an AS level in CT improves performance in other A levels. Using national examination data from 2005 and 2006 we compared the performance of CT students with those not taking CT. The results of both a basic comparison of mean A level performance and a regression analysis (which accounts for prior attainment) showed a significant and positive effect of taking CT, compared with not doing so. According to the regression results, the difference was equivalent to about one tenth of a grade per A level, although the difference was greater for those achieving a higher CT grade. 

Raikes, N., Fidler, J. and Gill, T. (2010). Must examiners meet in order to standardise their marking? An experiment with new and experienced examiners of GCE AS Psychology. Research Matters: A Cambridge Assessment publication, 10, 21-27.

When high-stakes examinations are marked by a panel of examiners, the examiners must be standardised so that candidates are not advantaged or disadvantaged according to which examiner marks their work.

It is common practice for Awarding Bodies’ standardisation processes to include a “Standardisation” or “Co-ordination” meeting, where all examiners meet to be briefed by the Principal Examiner and to discuss the application of the mark scheme in relation to specific examples of candidates’ work.  Research into the effectiveness of standardisation meetings has cast doubt on their usefulness, however, at least for experienced examiners.  

In the present study we addressed the following research questions:

1. What is the effect on marking accuracy of including a face-to-face meeting as part of an examiner standardisation process?
2. How does the effect on marking accuracy of a face-to-face meeting vary with the type of question being marked (short-answer or essay) and the level of experience of the examiners?
3. To what extent do examiners carry forward standardisation on one set of questions to a different but very similar set of questions?

Sykes, E., Novakovic, N., Greatorex, J., Bell, J., Nadas, R. and Gill, T. (2009). How effective is fast and automated feedback to examiners in tackling the size of marking errors? Research Matters: A Cambridge Assessment publication, 8, 8-15.

Reliability is important in national assessment systems. Therefore, there is a good deal of research about examiners’ marking reliability. However, some questions remain unanswered due to the changing context of e-marking, particularly the opportunity for fast and automated feedback to examiners on their marking. Some of these questions are:

•    will iterative feedback result in greater marking accuracy than only one feedback session?
•    will encouraging examiners to be consistent (rather than more accurate) result in greater marking accuracy?
•    will encouraging examiners to be more accurate (rather than more consistent) result in greater marking accuracy?

Thirty three examiners were matched into four experimental groups based on severity of their marking. All examiners marked the same 100 candidate responses, in the same short time scale. Group 1 received one session of feedback about their accuracy. Group 2 received three iterative sessions of feedback about the accuracy of their marking. Group 3 received one session of feedback about their consistency. Group 4 received three iterative sessions of feedback about the consistency of their marking. Absolute differences between examiners’ marking and a reference mark were analysed using a general linear model. The results of the present analysis pointed towards the answer to all the research questions being “no”. The results presented in this article are not intended to be used to evaluate current marking practices. Rather the article is intended to contribute to answering the research questions, and developing an evidence base for the principles that should be used to design and improve marking practices.

Elliott, G., Curcin, M., Bramley, T., Ireland, J., Gill, T. and Black, B. (2009). Assessment instruments over time. Research Matters: A Cambridge Assessment publication, 7, 23-25.

As Cambridge Assessment celebrated its 150th anniversary in 2008 members of the Evaluation and Psychometrics Team looked back at question papers over the years. Details of the question papers and examples of questions were used to illustrate the development of seven subjects: Mathematics, Physics, Geography, Art, French, Cookery and English Literature. Two clear themes emerged from the work across most subjects - an increasing emphasis on real-world contexts in more recent years and an increasing choice of topic areas and question/component options available to candidates.

Gill, T. and Bramley, T. (2008). Using simulated data to model the effect of inter-marker correlation on classification consistency. Research Matters: A Cambridge Assessment publication, 5, 29-36.

The marking of exam papers is never going to be 100% reliable unless all exams consist entirely of multiple-choice or other completely objective questions. Different opinions on the quality of the work or different interpretations of the mark schemes create the potential for candidates to receive a different mark depending on which examiner marks their paper. Of more concern for candidates is the potential for candidates to receive a different grade from a different examiner. The purpose of this study was to use simulated data to estimate the extent to which examinees might get a different grade for: i) different levels of correlation between markers and ii) for different grade bandwidths.