Martin Johnson

Johnson, M. and Shaw, S. D. (2019). What Is Computer-Based Testing Washback, How Can It Be Evaluated, And How Can This Support Practitioner Research? Journal of Further and Higher Education. 43(9), 1255-1270.

Johnson, M. (2018). Articulation Work: How do senior examiners construct feedback to encourage both examiner alignment and examiner development? Research Matters: A Cambridge Assessment publication, 26, 9-14.

This is a study of the marking feedback given to a group of examiners by their Team Leaders (more senior examiners who oversee and monitor the quality of examiner marking in their team). This feedback has an important quality assurance function but also has a developmental dimension, allowing less senior examiners to gain insights into the thinking of more senior examiners. When looked at from this perspective, marking feedback supports a form of examiner professional learning.

This study set out to look at this area of examiner practice in detail. To do this, I captured and analysed a set of feedback interactions involving 30 examiners across three Advanced level General Certificate of Education subjects. For my analysis, I used a mixture of learning theory and sociological theory to explore how the feedback was being used and how it attained its dual goals of examiner monitoring and examiner development.

Greatorex, J., Johnson, M. and Coleman, V. (2017). A review of instruments for assessing complex vocational competence. Research Matters: A Cambridge Assessment publication, 23, 35-42.

The aim of the research was to explore the measurement qualities of checklists and Global Rating Scales [GRS] in the context of assessing complex competence. Firstly, we reviewed the literature about the affordances of human judgement and the mechanical combination of human judgements. Secondly, we reviewed examples of checklists and GRS which are used to assess complex competence in highly regarded professions. These examples served to contextualise and elucidate assessment matters. Thirdly, we compiled research evidence from the outcomes of systematic reviews which compared advantages and disadvantages of checklists and GRS. Together the evidence provides a nuanced and firm basis for conclusions. Overall, literature shows that mechanical combination can outperform the human integration of evidence when assessing complex competence, and that therefore a good use of human judgements is in making decisions about individual traits, which are then mechanically combined. The weight of evidence suggests that GRS generally achieve better reliability and validity than checklists, but that a high quality checklist is better than a poor quality GRS. The review is a reminder that including assessors in designing assessment instruments processes can helps to maximise manageability.

Crisp, V., Johnson, M. and Constantinou, F. (2018) A question of quality: Conceptualisations of quality in the context of educational test questions. Research in Education (ahead of print).

Johnson, M. (2016). All in good time: Influences on team leaders’ communication choices when giving feedback to examiners. Research Matters: A Cambridge Assessment publication, 21, 28-33.

During the standardisation and live marking phases of an examination session it is increasingly common for team leaders and examiners to interact via a digital marking environment. This environment supports a number of quality assurance functions. Team leaders can see examiners’ real time scripts and mark submissions, and they can compare examiners’ marks with preordained definitive marks on special monitoring scripts to check marking accuracy. The digital marking system also allows team leaders to give examiners feedback on their marking. This article focuses on feedback practices, using observation, interview and survey data from 22 team leaders and six examiners to explore the rationales and communication choices involved in such practices. The analyses suggest that the objective of giving feedback is to construct messages that allow examiners insights into a team leader’s thinking, and this interaction is central to the development of an examiner’s understanding of how to interpret and apply mark schemes in accordance with their team leader’s views. This article discusses how the concept of synchrony underpins the feedback practices of team leaders and examiners, ensuring that the participants develop a shared focus so that both feedback message intention and reception are aligned.

Johnson, M. (2013) Paper presented at the European Conference on Computer Supported Cooperative Work, 21-25 September 2013.

Johnson, M. (2013). Cambridge Assessment Qualitative Research Methods Reading Group. Research Matters: A Cambridge Assessment publication, 15, 38.

This article is an update on the status of a research-based reading group that was formed to with the intention of sharing methods-related expertise within the Cambridge Assessement group. Since 2011 a series of Research Division-based reading groups have been organised. The remit of the group was initially to bring together researchers from across the Cambridge Assessment group to look at a variety of different qualitative research methods. The initiative was considered to be a useful way of sharing expertise amongst colleagues as well as being an important opportunity to raise awareness of the ways of using qualitative research methods in Cambridge Assessment’s own research.

Shaw, S., Johnson, M. and Warwick, P. (2013). Assessment for Learning in International Contexts (ALIC): understanding values and practices across diverse contexts Research Matters: A Cambridge Assessment publication, 15, 17-28.

Johnson, M. and Mehta, S. (2011). Evaluating the CRAS framework: Development and recommendations. Research Matters: A Cambridge Assessment publication, 12, 27-33.

This article reviews conceptual issues surrounding comparisons of demand through a critical evaluation of the CRAS (Complexity-Resources-Abstractness-Strategy) framework (Pollitt, Hughes, Ahmed, Fisher-Hoch and Bramley, 1998).

The article outlines the origins of the CRAS framework in the scale of cognitive demand (Edwards and Dall’Alba, 1981). The characteristics of the CRAS framework are then outlined, with attention being drawn to the assumptions that underlie these characteristic features. The article culminates in a set of recommendations and guidance that are relevant for potential users of the CRAS framework.

Shiell, H., Johnson, M., Hopkin, R., Nadas, R. and Bell, J. (2011). Extended essay marking on screen: Does marking mode influence marking outcomes and processes? Research Matters: A Cambridge Assessment publication, 11, 2-7

Research into comparisons between how people read texts on paper and computer screen suggests that the medium in which a text is read might influence the way that a reader comprehends that text. This is because some of the reading behaviours that support comprehension building, such as seamless navigation and annotation of text, are not easily replicated on screen.

Additional research also suggests that reading long texts can be more cognitively demanding on screen, and that this extra demand can have a detrimental effect on how readers comprehend longer texts. In the context of examination marking, there might be concerns that such a mode-related effect might lead to essays being marked less accurately when marked on screen compared with when they are marked on paper.

To investigate further the potential links between marking mode and the outcomes and processes of extended essay marking, the current project replicated an earlier study (Johnson and Nádas, 2009), replacing GCSE essays with longer Advanced GCE essays. The current project considered three broad areas of enquiry, exploring mode-related influences on (i) marking outcomes, (ii) manual marking processes and (iii) cognitive marking processes.

Johnson, M. and Shaw, S. (2010). Towards an understanding of the impact of annotations on returned exam scripts. Research Matters: A Cambridge Assessment publication, 10, 16-21.

There is little empirical study into practices around scripts returned to centres. Returned scripts often include information from examiners about the performance being assessed. As well as the total score given for the performance, additional information is carried in the form of the annotations left on the script by the marking examiner.

Examiners’ annotations have been the subject of a number of research studies (Crisp and Johnson, 2007; Johnson and Shaw, 2008; Johnson and Nádas, 2009) but as far as we know there has been no research into how this information is used by centres or candidates and whether it has any influence on future teaching and learning. This study set out to look at how teachers and students interact with examiners’ annotations  on scripts.

This study used survey and interview methods to explore:
 
1. How do teachers and centres use annotations?
2. What is the scale of such use?
3. What importance is attached to the annotations?
4. What factors might influence the interpretation of the annotations?

Johnson, M. and Nadas, R. (2009). An investigation into marker reliability and some qualitative aspects of on-screen marking. Research Matters: A Cambridge Assessment publication, 8, 2-7.

There is a growing body of research literature that considers how the mode of assessment, either computer- or paper-based, might affect candidates’ performances (Paek, 2005). Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor consistency when dealing with extended textual answers in different modes.

This study involved 12 examiners marking 90 GCSE English Literature essays on paper and on screen and considered 6 questions: 1. Does mode affect marker reliability? 2. Construct validity – do examiners consider different features of the essays when marking in different modes? 3. Is mental workload greater for marking on screen? 4. Is spatial encoding influenced by mode? 5. Is navigation influenced by mode? 6. Is ‘active reading’ influenced by mode?

Johnson, M. and Shaw, S. (2008). Annotating to comprehend: a marginalised activity? Research Matters: A Cambridge Assessment publication, 6, 19-24.

One of the important premises underlying this article is that the cognitive processes involved in reading can play a significant role in assessment judgements. Although we acknowledge that not all assessments of performance rely on assessors appraising written texts, many tests use written evidence as an indicator of performance. As a result, it is important to consider the role of assessors’ comprehension building when reading candidates’ textual responses, particularly where candidates are offered a greater freedom in determining the form and scope of their responses.

This paper brings together literature about linguistics and annotation practices, both empirical and theoretical, and suggests that a critical link exists between annotating and reading activities. Through making the different functions of annotation explicit the intention of this paper is to primarily amplify the importance of the impact of annotating on assessor comprehension.

Johnson, M. (2008). '3Rs' of assessment research: Respect, Relationships and Responsibility – what do they have to do with research methods? Research Matters: A Cambridge Assessment publication, 6, 2-4.

This article focuses on the merits and challenges of using qualitative research methods, and how these can contribute positively to the study of assessment. Based on a presentation from a methods-related research seminar, this paper appeals for research engagement with those areas where assessment affects the lives of others. This appeal means not only asking the difficult questions but also having the appropriate methodologies to try to answer them. The paper goes on to champion the strengths of mixed methods approaches that allow for triangulation, complementarity (where findings gained through one method offer insights into other findings) and expansion (of the breadth and scope of the research beyond initial findings).

Johnson, M. (2008). Holistic judgement of a borderline vocationally-related portfolio: a study of some influencing factors. Research Matters: A Cambridge Assessment publication, 6, 16-19.

The assessment of a large portfolio of mainly textual evidence demands an assessor to accommodate a great deal of information. This comprehension process is influenced by the linear nature of the reading process which leads to the gradual construction of a mental representation of the text in the head of the reader (Johnson and Laird, 1983).

Understanding how assessors work with portfolios also requires us to consider how assessors integrate and combine different aspects of an holistic performance into a final judgement. Sanderson (2001) suggests that the social context of the assessor is important to consider since it recognises their participation in a community of practice (Wenger, 1998) and constitutes an ‘outer frame’ for their activity.

This study sought to explore issues of consistent assessor judgement by gathering data about individual assessors’ cognitive activity as well as the socio-contextual features in which their practices were undertaken. It focused on an OCR Nationals unit in Health and Social Care (Level 2). Six assessors were asked to ‘think aloud’ whilst they judged the unit. This commentary was then transcribed into a verbal protocol and analysed with qualitative text analysis software. A modified Kelly’s Repertory Grid (KRG) interview technique was also used to gather data about different assessors’ perceptions of constructs within the same assessment criteria.

Johnson, M. and Burdett, N. (2008). School-based assessment in international practice. Research Matters: A Cambridge Assessment publication, 5, 24-29.

The term ‘school-based assessment’ (SBA) can conjure up diverse and not necessarily synonymous meanings which often include forms of ongoing and continual classroom assessment of a formative nature. This article attempts to clarify how, and why, SBA has been successfully introduced in various contexts and the importance of the context in its success or otherwise. It reviews SBA research literature and lists some of the advantages of SBA, some of the reservations about using SBA, and some principles about when and why SBA should be used.

Johnson, M. (2007). Is passing just enough? Some issues to consider in grading competence-based assessments. Research Matters: A Cambridge Assessment publication, 3, 27-30.

Competence-based assessment involves judgements about whether candidates are competent or not. For a variety of historical reasons, competency-based assessment has had an ambivalent relationship with grading (i.e., identifying different levels of competence), although it is accepted by some that ‘grading is a reality’ (Thomson, Saunders and Foyster, 2001, p.4). The question of grading in competence-based qualifications is particularly important in the light of recent national and international moves towards developing unified frameworks for linking qualifications. This article uses validity as a basis for discussing some of the issues that surround the grading of competence-based assessments and is structured around 10 key points.

Crisp, V. and Johnson, M. (2006). Examiners annotations: practice and purpose. Research Matters: A Cambridge Assessment publication, 2, 11-14.

The processes of reading and writing are recognised to be inextricably intertwined. Writing helps to support cognitive demands made upon the reader whilst processing a text (e.g., O’Hara, 1996; Benson, 2001). Examiners annotate scripts whilst marking (e.g., underlining, circling, using abbreviations or making comments) and this may reflect the cognitive support for comprehension building that annotations can provide. There is also some existing evidence that annotations might act as a communicative device in relation to accountability and that annotating might have a positive influence on markers’ perceptions and affect their feelings of efficacy.

This research investigated the use of annotations during marking and the role that annotations might be playing in the marking process. Six mathematics GCSE examiners and six business studies GCSE examiners who had previously been standardised to mark the paper were recruited. Examiners initially marked ten scripts which were then reviewed by their Team Leader. Examiners then marked a further 46 (Business Studies) or 40 (Mathematics) scripts.

The examiners later attended individual meetings with researchers. The session began with each examiner marking a small number of new scripts to re-familiarise themselves with the examination paper and mark scheme. A researcher then observed each examiner as they continued marking a few further scripts. Each examiner was interviewed about their use of annotations.

The findings portray a clear sense that markers in both subjects believed that annotating performed two distinct functions. The first appeared to be justificatory, communicating the reasons for their marking decisions to others. This mirrors the statutory requirements for awarding bodies to establish transparent, accountable procedures which ensure quality, consistency, accuracy and fairness. The second purpose was to support their thinking and marking decisions. In addition to helping markers with administrative aspects of marking (for example, keeping a running tally of marks), there are claims that annotations also support higher order reading comprehension processes.