Martin Johnson

Martin Johnson

I joined Cambridge Assessment in 2002, and prior to that I was a primary teacher for 10 years, during which time I completed an MA in Primary Education at the University of Reading.

My work at Cambridge Assessment has focussed on a diverse set of issues. These include studies into students’ experiences and perceptions of assessment (across primary, secondary and tertiary phases of education); advisory work on international test and curriculum development at primary and secondary level; investigation into how shifts from paper-based to computer-based assessment modes influence examiner practices; and the consideration of specific issues that relate to vocational assessment and learning contexts.

I completed a PhD at the University of Cambridge looking at how examiners communicate with each other when developing their common mark scheme understanding. I’ve also recently completed two observational studies exploring how question writers develop exam questions. I am a member of the Editorial Committee of the Journal of Vocational Education and Training, and I’m also a reviewer for seven other academic journals.

In my spare time I like to travel with my family, play guitar, and listen to loud music.

Publications

2018

A review of instruments for assessing complex vocational competence

Greatorex, J., Johnson, M. & Coleman, V. (2017). A review of instruments for assessing complex vocational competence. Research Matters: A Cambridge Assessment publication, 23, 35-42.

The aim of the research was to explore the measurement qualities of checklists and Global Rating Scales [GRS] in the context of assessing complex competence. Firstly, we reviewed the literature about the affordances of human judgement and the mechanical combination of human judgements. Secondly, we reviewed examples of checklists and GRS which are used to assess complex competence in highly regarded professions. These examples served to contextualise and elucidate assessment matters. Thirdly, we compiled research evidence from the outcomes of systematic reviews which compared advantages and disadvantages of checklists and GRS. Together the evidence provides a nuanced and firm basis for conclusions. Overall, literature shows that mechanical combination can outperform the human integration of evidence when assessing complex competence, and that therefore a good use of human judgements is in making decisions about individual traits, which are then mechanically combined. The weight of evidence suggests that GRS generally achieve better reliability and validity than checklists, but that a high quality checklist is better than a poor quality GRS. The review is a reminder that including assessors in designing assessment instruments processes can helps to maximise manageability.

A question of quality: Conceptualisations of quality in the context of educational test questions

Crisp, V., Johnson, M. and Constantinou, F. (2018) A question of quality: Conceptualisations of quality in the context of educational test questions. Research in Education (ahead of print).

2017

A culture of question writing: How do question writers compose examination questions in an examination paper?
Johnson, M. and Rushton, N. (2017). Presented at the 18th annual AEA Europe conference, Prague, 9-11 November 2017.
What is effective feedback in a professional learning context? A study of how examination markers feedback to each other on their marking performance
Johnson, M. (2017). Presented at the annual conference of the British Educational Research Association, University of Sussex, Brighton, UK, 5-7 September 2017.
More like work or more like school? Insights into learning cultures from a study of skatepark users
Johnson, M. and Oates, T. (2017). Presented at the Journal of Vocational Education and Training International Conference, University of Oxford, UK, 7-9 July 2017
Multiple voices in tests: towards a macro theory of test writing
Constantinou, F., Crisp, V., and Johnson, M. (2017).  Multiple voices in tests: towards a macro theory of test writing.  Cambridge Journal of Education (ahead of print).
How do question writers compose external examination questions? Question writing as a socio-cognitive process
Johnson, M., Constantinou, F. and Crisp, V. (2017). How do question writers compose external examination questions? Question writing as a socio-cognitive process. British Educational Research Journal (BERJ). 43(4), 700-719.

2016

Making Sense of a Learning Space: How Freestyle Scooter-riders Learn in a Skate Park
Johnson, M. and Oates, T. (2016). Informal Learning Review, 140, 17-21.
The challenges of researching digital technology use: examples from an assessment context
Johnson, M. (2016). International Journal of e-Assessment, 1(2), 1-10.
Reading between the lines: exploring methods for analysing professional examiner feedback discourse
Johnson, M. (2017). Reading between the lines: Exploring Methods for Analysing Professioanl Examiner Feedback Discourse. International Journal of Research & Method in Education, 40(5), 456-470.
Feedback effectiveness in professional learning contexts
Johnson, M. (2016). Review of Education, 4(2), 195-229.
How do question writers compose examination questions? Question writing as a socio-cognitive process
Johnson, M., Constantinou, F. and Crisp, V. (2016). Paper presented at the AEA-Europe annual conference, Limassol, Cyprus, 3-5 November 2016
'Question quality': The concept of quality in the context of exam questions
Crisp, V., Johnson, M. and Constantinou, F. (2016). Paper presented at the AEA-Europe annual conference, Limassol, Cyprus,3-5 November 2016
Writing questions for examination papers: a creative process?
Constantinou, F., Crisp, V. and Johnson, M. (2016). Paper presented at the 8th Biennial Conference of the European Association for Research in Learning and Instruction (EARLI) SIG 1 - Assessment and Evaluation, Munich, Germany, 24-26 August 2016
Researching effective feedback in a professional learning context
Johnson, M. (2016). Paper presented at the 7th Nordic Conference on Cultural and Activity Research, Helsingør, Denmark, 16-18 June 2016
All in good time: Influences on team leaders’ communication choices when giving feedback to examiners

Johnson, M. (2016). All in good time: Influences on team leaders’ communication choices when giving feedback to examiners. Research Matters: A Cambridge Assessment publication, 21, 28-33.

During the standardisation and live marking phases of an examination session it is increasingly common for team leaders and examiners to interact via a digital marking environment. This environment supports a number of quality assurance functions. Team leaders can see examiners’ real time scripts and mark submissions, and they can compare examiners’ marks with preordained definitive marks on special monitoring scripts to check marking accuracy. The digital marking system also allows team leaders to give examiners feedback on their marking. This article focuses on feedback practices, using observation, interview and survey data from 22 team leaders and six examiners to explore the rationales and communication choices involved in such practices. The analyses suggest that the objective of giving feedback is to construct messages that allow examiners insights into a team leader’s thinking, and this interaction is central to the development of an examiner’s understanding of how to interpret and apply mark schemes in accordance with their team leader’s views. This article discusses how the concept of synchrony underpins the feedback practices of team leaders and examiners, ensuring that the participants develop a shared focus so that both feedback message intention and reception are aligned.

2015

Finding the common ground: Teachers' and employers' representations of English in an assessment context
Child, S., Johnson, M., Mehta, S. and Charles, A. (2015).  English in Education, 49(2), 150-166.
Reading between the lines: exploring the characteristics of feedback that support examiners’ professional knowledge building
Johnson, M. (2015) Paper presented at the British Educational Research Association (BERA) conference, Belfast, 15-17 September 2015
Articulation work: Insights into examiners' expertise from their remote feedback sessions
Johnson, M. (2015). Communication & Language at work. 1(4), 28-52
Assessment, aim and actuality: insights from teachers in England about the validity of a new language assessment model
Johnson, M., Mehta, S. and Rushton, N. (2015). Pedagogies: An International Journal. 10(2), 128-148.

2014

Assessment for Learning in International Contexts: exploring shared and divergent dimensions in teacher values and practices
Warwick P., Shaw, S. D. and Johnson, M. (2014). Assessment for Learning in International Contexts: exploring shared and divergent dimensions in teacher values and practices. The Curriculum Journal, 25(4), 1-31.
Insights into contextualised learning
Johnson, M (2014). Insights into contextualised learning: how do professional examiners construct shared understanding through feedback? E-Learning and Digital Media, 11(4), 363-378.
A case study of inter-examiner feedback from a UK context
Johnson, M (2014). A case study of inter-examiner feedback from a UK context: Mixing research methods to gain insights into situated learning interactions. Formation et pratiques d’enseignement en questions, 17, 67-88.

2013

'Seeing what they say' - Mapping the characteristics of effective remote feedback

Johnson, M. (2013) Paper presented at the European Conference on Computer Supported Cooperative Work, 21-25 September 2013.

Can you dig it?
Johnson, M and Lewis, C. (2013). Developing an approach to validly assessing diverse skills in an archaeological context. Journal of Vocational Education and Training. 65(2), 177-192
Cambridge Assessment Qualitative Research Methods Reading Group

Johnson, M. (2013). Cambridge Assessment Qualitative Research Methods Reading Group. Research Matters: A Cambridge Assessment publication, 15, 38.

This article is an update on the status of a research-based reading group that was formed to with the intention of sharing methods-related expertise within the Cambridge Assessement group. Since 2011 a series of Research Division-based reading groups have been organised. The remit of the group was initially to bring together researchers from across the Cambridge Assessment group to look at a variety of different qualitative research methods. The initiative was considered to be a useful way of sharing expertise amongst colleagues as well as being an important opportunity to raise awareness of the ways of using qualitative research methods in Cambridge Assessment’s own research.

Assessment for Learning in International Contexts (ALIC): understanding values and practices across diverse contexts

Shaw, S., Johnson, M. and Warwick, P. (2013). Assessment for Learning in International Contexts (ALIC): understanding values and practices across diverse contexts Research Matters: A Cambridge Assessment publication, 15, 17-28.

The Assessment for Learning in International Contexts (ALIC) project sought to extend knowledge around teachers’ understandings of Assessment for Learning (AfL). Using a modified version of a survey devised by James and Pedder for use with teachers in England, evidence was gathered about the assessment practices that were highly valued by teachers across international contexts (Argentina, India, Indonesia, Saudi Arabia, and Nigeria). The extent of congruence between these values and teachers’ reported classroom practices was then explored.

In very broad terms, the items most valued by the teachers in this study demonstrate the considerable value placed upon practices linked positively to formative assessment principles and strategies. Certainly it seems that teachers have a particular concern with learning more about student learning and with promoting the development of pupil agency in assessment and learning.

2012

The Assessment for Learning in International Contexts (ALIC) Research Project
Shaw, S., Johnson, M. and Warwick, P. (2012) Research Intelligence 119, 14-15
Interpreting examiners’ annotations on examination scripts: a sociocultural analysis
Johnson, M. and Shaw, S. (2012) Irish Educational Studies 31(4), 467-485
Feedback as scaffolding: Senior Examiner monitoring processes and their effects on examiner marking
Johnson, M. and Black, B. (2012) Research in Post-Compulsory Education 17(4), 391-407
Essay Marking on Screen: Factsheet 3
Cognitive workload is greater on screen
Essay Marking on Screen: Factsheet 2
Marking behaviours differ on screen
Essay Marking on Screen: Factsheet 1
Essay marking accuracy is reliable across modes
The effects of features of examination questions on the performance of students with dyslexia
Crisp, V., Johnson, M. and Novakovic, N. (2012) British Educational Research Journal, 38(5), 813-839.
What's going on? Analysing visual data to understand context-based decision-making processes
Johnson, M. and Black, B. (2012) International Journal of Research & Method in Education, 35(3), 243-250
Technologically mediated communication: methods for exploring examiners’ real-time feedback interactions
Johnson, M. and Black, B. (2012) EARLI Special Interest Group 17 (Qualitative and Quantitative Approaches to Learning and Instruction) conference on Mixed Methods in Educational Interactions, Saxion University, Deventer, September 2012
Feedback as scaffolding: Senior Examiner monitoring processes and their effects on examiner marking
Johnson, M. and Black, B. (2012) British Educational Research Association Annual Conference, University of Manchester, September 2012
A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies
Johnson, N. and Nádas, R. (2012) Educational Research and Evaluation, 18(5) 425-440
Extended essay marking on screen: is examiner marking accuracy influenced by marking mode?
Johnson, M., Hopkin, R., Shiell, H. and Bell, J.F. (2012) Educational Research and Evaluation: An International Journal on Theory and Practice, 18, 2, 107-124
Marking extended essays on screen: exploring the link between marking processes and comprehension.
Johnson, M., Hopkin, R, and Shiell, H. (2012) E-Learning and Digital Media

2011

Extended essay marking on screen: does marking mode influence marking outcomes and processes?
Johnson, M., Hopkin, R., Shiell, H. and Bell, J. F. (2011). Paper presented at the British Educational Research Association annual conference, University of London Institute of Education, September 2011.
Can you dig it?': developing an approach to validly assessing diverse skills in an archaeological context
Johnson, M. and Lewis, C. (2011). Paper presented at the Journal of Vocational Education and Training International conference, Oxford, July 2011.
Evaluating the CRAS framework: Development and recommendations

Johnson, M. and Mehta, S. (2011). Evaluating the CRAS framework: Development and recommendations. Research Matters: A Cambridge Assessment publication, 12, 27-33.

This article reviews conceptual issues surrounding comparisons of demand through a critical evaluation of the CRAS (Complexity-Resources-Abstractness-Strategy) framework (Pollitt, Hughes, Ahmed, Fisher-Hoch and Bramley, 1998).

The article outlines the origins of the CRAS framework in the scale of cognitive demand (Edwards and Dall’Alba, 1981). The characteristics of the CRAS framework are then outlined, with attention being drawn to the assumptions that underlie these characteristic features. The article culminates in a set of recommendations and guidance that are relevant for potential users of the CRAS framework.

Extended essay marking on screen: Does marking mode influence marking outcomes and processes?

Shiell, H., Johnson, M., Hopkin, R., Nadas, R. and Bell, J. (2011). Extended essay marking on screen: Does marking mode influence marking outcomes and processes? Research Matters: A Cambridge Assessment publication, 11, 2-7

Research into comparisons between how people read texts on paper and computer screen suggests that the medium in which a text is read might influence the way that a reader comprehends that text. This is because some of the reading behaviours that support comprehension building, such as seamless navigation and annotation of text, are not easily replicated on screen.

Additional research also suggests that reading long texts can be more cognitively demanding on screen, and that this extra demand can have a detrimental effect on how readers comprehend longer texts. In the context of examination marking, there might be concerns that such a mode-related effect might lead to essays being marked less accurately when marked on screen compared with when they are marked on paper.

To investigate further the potential links between marking mode and the outcomes and processes of extended essay marking, the current project replicated an earlier study (Johnson and Nádas, 2009), replacing GCSE essays with longer Advanced GCE essays. The current project considered three broad areas of enquiry, exploring mode-related influences on (i) marking outcomes, (ii) manual marking processes and (iii) cognitive marking processes.

Extended essay marking: Does the transition from paper to screen influence examiners' cognitive workload?
Johnson, M., Hopkin, R. and Shiell, H. (2011) The International Journal of e-assessment, 1, 1, 1-25

2010

Marking essays on screen: an investigation into the reliability of marking extended subjective texts
Johnson, M., Nadas, R. and Bell, J.F. (2010) British Journal of Educational Technology, 41, 5, 814-826
Towards an understanding of the impact of annotations on returned exam scripts

Johnson, M. and Shaw, S. (2010). Towards an understanding of the impact of annotations on returned exam scripts. Research Matters: A Cambridge Assessment publication, 10, 16-21.

There is little empirical study into practices around scripts returned to centres. Returned scripts often include information from examiners about the performance being assessed. As well as the total score given for the performance, additional information is carried in the form of the annotations left on the script by the marking examiner.

Examiners’ annotations have been the subject of a number of research studies (Crisp and Johnson, 2007; Johnson and Shaw, 2008; Johnson and Nádas, 2009) but as far as we know there has been no research into how this information is used by centres or candidates and whether it has any influence on future teaching and learning. This study set out to look at how teachers and students interact with examiners’ annotations  on scripts.

This study used survey and interview methods to explore:
 
1. How do teachers and centres use annotations?
2. What is the scale of such use?
3. What importance is attached to the annotations?
4. What factors might influence the interpretation of the annotations?

School Based Assessment in International Practice
Johnson, M. and Burdett, N. (2010) Problems in Modern Education, 4, 64-73
Marking essays on screen and on paper
Johnson, M., Nadas, R. and Green, S. (2010) Education Journal, 121, 39-41
Intention, interpretation and implementation: some paradoxes of Assessment for Learning across educational contexts
Johnson, M. and Burdett, N. (2010) Research in Comparative and International Education, 5, 2, 122-130

2009

Marginalised behaviour: digital annotations, spatial encoding and the implications for reading comprehension
Johnson, M. and Nadas, R. (2009) Learning, Media and Technology, 34, 4, 323-336
An exploration of the effect of pre-release examination materials on classroom practice in the UK
Johnson, M. and Crisp, V. (2009) Research in Education, 82, 47-59
An investigation into marker reliability and other qualitative aspects of on-screen essay marking
Johnson, M., Nadas, R. and Shiell, H. (2009) British Educational Research Association (BERA) Annual Conference
An investigation into marker reliability and some qualitative aspects of on-screen marking

Johnson, M. and Nadas, R. (2009). An investigation into marker reliability and some qualitative aspects of on-screen marking. Research Matters: A Cambridge Assessment publication, 8, 2-7.

There is a growing body of research literature that considers how the mode of assessment, either computer- or paper-based, might affect candidates’ performances (Paek, 2005). Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor consistency when dealing with extended textual answers in different modes.

This study involved 12 examiners marking 90 GCSE English Literature essays on paper and on screen and considered 6 questions: 1. Does mode affect marker reliability? 2. Construct validity – do examiners consider different features of the essays when marking in different modes? 3. Is mental workload greater for marking on screen? 4. Is spatial encoding influenced by mode? 5. Is navigation influenced by mode? 6. Is ‘active reading’ influenced by mode?

2008

A case of positive washback: an exploration of the effect of pre-release examination materials on classroom practice: ECER abstract
Johnson, M. & Crisp, V. (2008) European Conference on Educational Research (ECER), Gothenburg
Annotating to comprehend: a marginalised activity?

Johnson, M. and Shaw, S. (2008). Annotating to comprehend: a marginalised activity? Research Matters: A Cambridge Assessment publication, 6, 19-24.

One of the important premises underlying this article is that the cognitive processes involved in reading can play a significant role in assessment judgements. Although we acknowledge that not all assessments of performance rely on assessors appraising written texts, many tests use written evidence as an indicator of performance. As a result, it is important to consider the role of assessors’ comprehension building when reading candidates’ textual responses, particularly where candidates are offered a greater freedom in determining the form and scope of their responses.

This paper brings together literature about linguistics and annotation practices, both empirical and theoretical, and suggests that a critical link exists between annotating and reading activities. Through making the different functions of annotation explicit the intention of this paper is to primarily amplify the importance of the impact of annotating on assessor comprehension.

'3Rs' of assessment research: Respect, Relationships and Responsibility – what do they have to do with research methods?

Johnson, M. (2008). '3Rs' of assessment research: Respect, Relationships and Responsibility – what do they have to do with research methods? Research Matters: A Cambridge Assessment publication, 6, 2-4.

This article focuses on the merits and challenges of using qualitative research methods, and how these can contribute positively to the study of assessment. Based on a presentation from a methods-related research seminar, this paper appeals for research engagement with those areas where assessment affects the lives of others. This appeal means not only asking the difficult questions but also having the appropriate methodologies to try to answer them. The paper goes on to champion the strengths of mixed methods approaches that allow for triangulation, complementarity (where findings gained through one method offer insights into other findings) and expansion (of the breadth and scope of the research beyond initial findings).

Holistic judgement of a borderline vocationally-related portfolio: a study of some influencing factors

Johnson, M. (2008). Holistic judgement of a borderline vocationally-related portfolio: a study of some influencing factors. Research Matters: A Cambridge Assessment publication, 6, 16-19.

The assessment of a large portfolio of mainly textual evidence demands an assessor to accommodate a great deal of information. This comprehension process is influenced by the linear nature of the reading process which leads to the gradual construction of a mental representation of the text in the head of the reader (Johnson and Laird, 1983).

Understanding how assessors work with portfolios also requires us to consider how assessors integrate and combine different aspects of an holistic performance into a final judgement. Sanderson (2001) suggests that the social context of the assessor is important to consider since it recognises their participation in a community of practice (Wenger, 1998) and constitutes an ‘outer frame’ for their activity.

This study sought to explore issues of consistent assessor judgement by gathering data about individual assessors’ cognitive activity as well as the socio-contextual features in which their practices were undertaken. It focused on an OCR Nationals unit in Health and Social Care (Level 2). Six assessors were asked to ‘think aloud’ whilst they judged the unit. This commentary was then transcribed into a verbal protocol and analysed with qualitative text analysis software. A modified Kelly’s Repertory Grid (KRG) interview technique was also used to gather data about different assessors’ perceptions of constructs within the same assessment criteria.

Exploring assessor consistency in a Health and Social Care qualification using a sociocultural perspective
Johnson, M. (2008) Journal of Vocational Education and Training, 60, 2, 173-187
Grading in competence-based qualifications – is it desirable and how might it affect validity?
Johnson, M. (2008) Journal of Further and Higher Education, 32, 2, 175-184
School-based assessment in international practice

Johnson, M. and Burdett, N. (2008). School-based assessment in international practice. Research Matters: A Cambridge Assessment publication, 5, 24-29.

The term ‘school-based assessment’ (SBA) can conjure up diverse and not necessarily synonymous meanings which often include forms of ongoing and continual classroom assessment of a formative nature. This article attempts to clarify how, and why, SBA has been successfully introduced in various contexts and the importance of the context in its success or otherwise. It reviews SBA research literature and lists some of the advantages of SBA, some of the reservations about using SBA, and some principles about when and why SBA should be used.

Judging Text Presented on Screen: implications for validity
Johnson, M. and Greatorex, J. (2008) E-Learning, 5, 1, 40-50

2007

The use of annotations in examination marking: opening a window into markers’ minds
Crisp, V. and Johnson, M. (2007) British Educational Research Journal, 33(6), 943–961
Assessors’ holistic judgements about borderline performances: some influencing factors
Johnson, M. and Greatorex, J. (2007) British Educational Research Association (BERA) Annual Conference
The effects of features of GCSE questions on the performance of students with dyslexia
Crisp, V., Johnson, M. and Novakovic, N. (2007) British Educational Research Association (BERA) Annual Conference
Does the anticipation of a merit grade motivate vocational test-takers?
Johnson, M (2007) Research in Post-Compulsory Education, 12, 2, 159-179
Grading, motivation and vocational assessment
Johnson, M. (2007) The Journal of Vocational Education and Training International Conference, University of Oxford

2006

Vocational review 2002-2006: Issues of validity, reliability and accessibility
Johnson, M. (2006) British Educational Research Association (BERA) Annual Conference
A review of vocational research in the UK 2002-2006: measurement and accessibility issues.
Johnson, M. (2006) International Journal of Training Research, 4, 2, 48-71
On-line Mathematics Assessment: The Impact of Mode on Performance and Question Answering Strategies
Johnson, M. and Green, S. (2006) The Journal of Technology, Learning and Assessment, 4, 5

2005

Judging learners’ work on screen. How valid and fair are assessment judgements?
Johnson, M. and Greatorex, J. (2005) British Educational Research Association (BERA) Annual Conference
The use of annotations in examination marking: opening a window into markers' minds
Crisp, V. and Johnson, M (2005) British Educational Research Association (BERA) Annual Conference
Concepts of Difficulty: A Child's Eye View
Johnson, M. (2005) In: M. Pandis, A. Ward and Mathews, S. R. (Eds), Reading, Writing, Thinking. 199-207. Newark, DE: International Reading Association

2004

On-line assessment: the impact of mode on students’ strategies, perceptions and behaviours
Johnson, M. and Green, S. (2004) British Educational Research Association (BERA) Annual Conference
On-line assessment: the impact of mode on student performance
Johnson, M. and Green, S. (2004) British Educational Research Association (BERA) Annual Conference

2003

Concepts of difficulty - a child's eye view
Johnson, M. (2003) British Educational Research Association (BERA) Annual Conference
Changes in Key Stage Two Writing from 1995 to 2002
Green, S., Johnson, M., O’Donovan, N. and Sutton, P. (2003) United Kingdom Reading Association Conference

2002

What makes a good writing stimulus?
Johnson, M. (2002) British Educational Research Association (BERA) Annual Conference
Comparability Study of Pupils' Writing from Different Key Stages
Green, S., Pollitt, A., Johnson, M. and Sutton, P. (2002) British Educational Research Association (BERA) Annual Conference

Research Matters

Research Matters

Research Matters is our free biannual publication which allows us to share our assessment research, in a range of fields, with the wider assessment community.