Irenka Suto

Suto, I., Greatorex, J., Vitello, S., and Child, S. (2020). A way of using taxonomies to demonstrate that applied qualifications and curricula cover multiple domains of knowledge. Research Matters: A Cambridge Assessment publication, 30, 26-34.

Educational taxonomies are classification schemes which provide the terminology that educationalists need to describe and work with different areas of knowledge. It is good practice to use taxonomies to formulate and review curricula, learning objectives, and associated assessments. Demonstrating sufficient coverage of each of an adequate range of knowledge domains is critical for authenticity, for assessment reliability, and for transparency surrounding what students are learning.

In this study we explored whether any educational taxonomies that were designed for general educational contexts (sometimes called ‘academic’ contexts) could be utilised in applied educational contexts (often called ‘vocationally-related’ in England) . To do this, we identified nine published taxonomies with sufficient potential, and selected and combined the most appropriate. This process led us to develop a new model of demand.  We then applied the selected taxonomies experimentally to existing curricula in a range of applied subjects which are taught at secondary and tertiary level in England. We also used the selected taxonomies to develop a tool for writing educational objectives. This article ends with suggestions for applying the selected taxonomies in other areas of assessment.

Elliott, G., Suto, I. and Walland, E. (2019). What makes researchers anxious? It’s Time to Talk about talking about research. Research Matters: A Cambridge Assessment publication, 27, 26-32.

Increasingly, workplaces provide accessible places and events for employees to discuss mental health issues. In the course of discussions in our organisation, researchers raised the issue of workplace activities that cause them anxiety – top of the list being giving presentations. This article describes a workshop held with researchers at Cambridge Assessment to encourage the discussion to develop more widely. We reflect upon themes about workplace anxiety identified by the workshop participants, relate the themes to the literature, and consider practical techniques for reducing the impact of anxiety. Our aim is that this study will be helpful in enabling other researchers to become more open in discussing the aspects of their role which cause anxiety. In turn, this will facilitate finding ameliorative solutions to the issues raised, to the mutual benefit of individuals and organisations.

Shaw, S., Kuvalja, M. and Suto, I. (2018). An exploration of the nature and assessment of student reflection. Research Matters: A Cambridge Assessment publication, 25, 2-8.

Reflection is often considered to be one of the so-called ‘21st century’ or ‘transversal’ skills, or ‘life competencies’. Many societies value people who can reflect upon their own beliefs and experiences in the classroom and beyond, and learn from them. It is also important to be able to contemplate the work of others at a deep level. In this article, we review some of the academic literature on reflection and explore ways in which it is assessed in educational contexts. The Advanced Subsidiary (AS) level qualification in Global Perspectives and Research offered by Cambridge Assessment International Education serves as a case study for how reflection can be assessed as part of a taught curriculum.

Suto, I., Mehta, S., Child, S., Wilson, F. and Jeffrey, E. (2013). A level reform: Is the Government in tune with its stakeholders? Research Matters: A Cambridge Assessment publication, 16, 9-14.

Reformed GCE A levels are on the educational horizon for many students and their teachers. Awarding bodies are in the process of redeveloping their courses and from September 2015, the new syllabuses will be taught in sixth forms across England. In this article, we give a chronological account of the recent developments in Government policy which have fed into these plans. Alongside this account, we describe five studies that we have undertaken within our Higher Education (HE) Engagement research programme. An overarching aim of our research has been to ascertain the views and experiences of stakeholders in schools, colleges and universities on multiple aspects of A level reform.

Suto, I. (2013). 21st Century Skills: Ancient, ubiquitous, enigmatic? Research Matters: A Cambridge Assessment publication, 15, 2-8.

The understanding and skills needed to compete in today’s global economy are arguably quite different to those upon which 19th and 20th century education systems have traditionally focussed. Life has become much more international, multicultural and inter-connected. Seismic advances have occurred in ICT and in access to it. These have enabled the economies of developed countries, including the UK’s, to shift from a basis of material goods and services to one of information and knowledge. The aim of this article is to explore some of the benefits and risks of building pedagogies and curricula around 21st Century skills. I begin by outlining some conceptualisations of 21st Century skills. I then address the question of how their development in young people can best be supported; I describe recent examples of alternative approaches used in the UK and internationally, including extended projects for sixth-form students. I also start to consider the value placed by stakeholders on the summative assessment of 21st Century skills, and finally, the feasibility of such assessment for test developers.

Mehta, S., Suto, I., Elliott, G. and Rushton, N. (2013). Independent research at A level: Students’ and teachers’ experiences. Research Matters: A Cambridge Assessment publication, 15, 9-16. 

Our aims were to explore teachers’ and students’ experiences and perspectives of independent research at A level. The study focused Economics, French and Mathematics. It investigated: (i) the extent to which teachers think research and investigative skills can be developed at A level; (ii) the resources and guidance that students use; and (iii) whether subject-specific differences arise. A questionnaire and follow-on interview methodology was used. 47 Mathematics teachers, 24 Economics teachers and 15 French teachers participated. Additionally, 299 Mathematics students, 228 Economics students and 136 French students took part.

About half of the French and Economics teachers were found to assign investigative/research tasks to their students at least once a fortnight. On the other hand, about half of the Mathematics teachers set such tasks less often and a further 40% never set them at all. The frequency with which the teachers set investigation/research tasks as homework/private study showed the same subject-specific differences as the classroom context. The internet was the most frequently listed source that students across all three subjects consulted while engaging in independent research. The interview data shed further light on general and specific internet usage. Overall, the findings explain some of the variation in preparedness of new undergraduates for independent study and research-related tasks at university.

Suto, I. and Nadas, R. (2012). Starting them young: research and project management opportunities for 16 to 19 year olds. Research Matters: A Cambridge Assessment publication, 13, 27-31.

Several educational routes have been developed which entail project work with a specific focus on independent learning and research. In this article we outline some of the options that exist at Level 3, primarily for 16 to 19 year olds (Years 12 and 13) in the UK and internationally. We then conduct a more detailed comparison of two routes: the Extended Project Qualification, and the International Baccalaureate Extended Essay. Many stakeholders may be unaware of the differences in the aims, structure, and scope of these routes. It is important for students and teachers to be conscious of the differences so that they can make informed decisions about what is most suitable for them. End-users such as higher education admissions tutors and employers also need to understand the differences in order to weigh up the experiences and achievements of applicants fairly.

Suto, I. and Shaw, S.  (2010). A tricky task for teachers: assessing pre-university students' research reports. Research Matters: A Cambridge Assessment publication, 10, 10-16.

In the UK and internationally, many students preparing for university are given the challenge of conducting independent research and writing up a report of around 4,000 or 5,000 words. Such research activities provide students with opportunities to investigate a specialist area of study in greater depth, to cross boundaries with an inter-disciplinary enquiry, or to explore a novel non-school subject such as archaeology, cosmology or anthropology. In this study, we explored the feasibility of applying a single mark scheme to research reports covering diverse topics in order to reward generic research skills. Our aim was to investigate the reliability with which teachers can mark diverse research reports, using four different generic assessment objectives. We also investigated teachers’ views in applying generic mark schemes, particularly when marking reports on unfamiliar topics. Our analyses indicated that marking reliability was good, though like almost all qualifications, imperfect. Possible reasons and explanations for marking difficulty related to subject knowledge, the clarity of student thought, and the overall level of student performance. 

King, P., Novakovic, N. and Suto, I. (2009). Capturing expert judgement in grading: an examiner's perspective. Research Matters: A Cambridge Assessment publication, 8, 32-33.

There exist several methods of capturing expert judgement which have been used, or could potentially be used, in the process of determining grade boundaries for examinations. In a recent study, we sought to explore the judgements entailed in three such methods: (i) rank ordering, (ii) traditional awarding, and (iii) Thurstone pairs. A key aim was to identify the features of candidates’ scripts that affect the judgements made in each of the three methods. To achieve this, sixty experienced examiners participated in the study. Each made judgements about overall script quality, using each method on a different batch of scripts. Additionally, each examiner completed a research task in which he or she was asked to rate a fourth batch of scripts for a series of features, using rating scales devised by the researchers. Subsequent data analysis entailed relating the judgemental data on script quality to the script feature data. Immediately after taking part in the study, one examiner recorded and offered the Research Division his views and experiences of participation. His perspective is the focus of this article.

Suto, I., Greatorex, J. and Nadas, R. (2009). Thinking about making the right mark: Using cognitive strategy research to explore examiner training. Research Matters: A Cambridge Assessment publication, 8, 23-32.

In this article, we draw together research on examiner training and on the nature of the judgements entailed in the marking process. We report new analyses of data from two recent empirical studies, Greatorex and Bell (2008) and Suto and Nadas (2008a), exploring possible relationships between the efficacy of training and the complexity of the cognitive marking strategies apparently needed to mark the examination questions under consideration. In the first study reported in this article, we considered the benefits of three different training procedures for experienced examiners marking AS-level biology questions. In the second study reported here, we explored the effects of a single training procedure on experienced and inexperienced (graduate) examiners marking GCSE mathematics and physics questions. In both studies, it was found that: (i) marking accuracy was better after training than beforehand; and (ii) the effect of training on change in marking accuracy varied across all individual questions. Our hypothesis that training would be more beneficial for apparently more complex cognitive marking strategy questions than for apparently simple cognitive marking strategy questions was upheld for both subjects in Study 2, but not in Study 1.

Nadas, R. and Suto, I. (2008). An exploration of self-confidence and insight into marking accuracy among GCSE maths and physics markers. Research Matters: A Cambridge Assessment publication, 5, 9-15.

A considerable volume of literature in education and occupational research investigates issues in self-confidence and insight. However, GCSE markers’ perceptions of their marking performance have not, to our knowledge, been examined. Exploring markers’ perceptions is important for several reasons. First, if markers’ estimates of their own performance prove to be accurate, then this information could be used by Awarding Bodies in standardisation procedures to identify and discuss examination questions that markers have difficulties with. If, however, markers’ insight proves to be unreliable and unrelated to their actual marking accuracy, then their feedback on ‘problem areas’ could be misleading: for example, when conducting standardisation procedures, Principal Examiners might find themselves focussing on the ‘wrong’ questions. Secondly, investigating whether self-confidence and insight change or become more accurate with more marking practice or more feedback could inform marker training practices.

In this article, we present research which explored GCSE markers’ perceptions of their own marking accuracy. Markers’ levels of self-confidence and insight, and possible changes in these measures over the course of the marking process, were investigated. The term ‘self-confidence’ here denotes markers’ post-marking estimates of how accurately they thought they had marked a sample of questions; ‘insight’ refers to the relationship between markers’ actual marking accuracy and estimated accuracy, indicating how precise their estimates were. Overall, we found very different patterns of self-confidence and insight for maths and physics markers.

Suto, I., Crisp, V. and Greatorex, J. (2008). Investigating the judgemental marking process: an overview of our recent research. Research Matters: A Cambridge Assessment publication, 5, 6-9.

This article gives an overview of a number of linked studies which explored the process of marking GCSE and A-level examination questions from a number of different angles. Key aims of these studies were to provide insights into how examiner training and marking accuracy could be improved, as well as reasoned justifications for how item types could be assigned to different groups of examiners in the future. The research studies combined several approaches, exploring both the information that people attend to when marking items and the sequences of mental operations involved. Examples include studies that used the think-aloud method to identify the cognitive marking strategies entailed in marking student responses, or to explore the broader socio-cognitive influences on the marking process. Other examples explored the relationship between cognitive marking strategy complexity and marking accuracy.

This article brings together the findings from these various related studies to summarise the influences and processes that have been identified as important to the marking process from the research conducted so far.

Suto, I. and Nadas, R. (2007). The ‘Marking Expertise’ projects: Empirical investigations of some popular assumptions. Research Matters: A Cambridge Assessment publication, 4, 2-5.

Recent transformations in professional marking practice, including moves to mark some examination papers on screen, have raised important questions about the demands and expertise that the marking process entails. What makes some questions harder to mark accurately than others, and how much does marking accuracy vary among individuals
with different backgrounds and experiences? We are conducting a series of interrelated studies, exploring variations in accuracy and expertise in GCSE examination marking.

In our first two linked studies, collectively known as Marking Expertise Project 1, we investigated marking on selected GCSE maths and physics questions from OCR’s June 2005 examination papers. Our next two linked studies, which comprise Marking Expertise Project 2, are currently underway and involve both CIE and OCR examinations. This time we are
focussing on International (I) GCSE biology questions from November 2005 and GCSE business studies questions from June 2006. All four studies sit within a conceptual framework in which we have proposed a number of factors that might contribute to accurate marking. For any particular GCSE examination question, accuracy can be maximised through increasing the marker’s personal expertise and/or through decreasing the demands of the marking task, and most relevant factors can be grouped according to which of these two routes they contribute to. In this article, we present a summary of some key aspects and findings of the two studies comprising our first project. We end by looking ahead to our second  project on marking expertise, which is currently in progress.

Suto, I. and Greatorex, J. (2006). A cognitive psychological exploration of the GCSE marking process. Research Matters: A Cambridge Assessment publication, 2, 7-11.

GCSEs play a crucial role in secondary education throughout England and Wales, and the process of marking them, which entails extensive human judgement, is a key determinant in the futures of many sixteen-year-olds. The aims of our study were to investigate the cognitive strategies used when marking GCSEs and to interpret them within the context of psychological theories of human judgement.

Two GCSE examinations were considered: an intermediate tier Mathematics paper, which used a ‘points-based’ marking scheme, and a foundation tier Business Studies paper, which used a ‘levels-based’ scheme. For each subject, a group of six experienced examiners marked four identical script samples each. The first three of these samples were marked silently. Whilst marking the fourth sample, the examiners were asked to ‘think aloud’ concurrently. Using a semi-structured interview schedule, the examiners were later questioned about their marking experiences retrospectively.

A qualitative analysis of the verbal protocol data enabled us to propose a tentative model of marking, which includes five distinct cognitive marking strategies: matching, scanning, evaluating, scrutinising, and no response. These strategies were broadly validated not only in the retrospective interviews with the participating examiners, but also by other senior mathematics and business studies examiners.

Suto, I. and Green, S. (2006).  International perspectives on vocational education: What can we learn from each other? Research Matters: A Cambridge Assessment publication, 2, 2-7.

The broad aim of Vocational Education and Training (VET) is to provide students with the technical skills and knowledge needed to enter the workforce. It exists for an extensive range of subject areas and may be delivered through many different kinds of training institutions and enterprises. Over the past four years, this huge area of education and assessment has received close scrutiny from the Government and others, as part of a broader review of education and skills among 14 to 19 year olds (Tomlinson, 2004; Department for Education and Skills, 2005). Given that this process has resulted in proposals for considerable reform of VET for this age group, we deemed it important to know more about the international context within which they are set. Who does VET affect globally, and what might we learn from the experiences of other countries? The aims of this project, therefore, were to identify and examine two main types of data: (i) on the extent of participation in VET and its associated assessment worldwide; and (ii) relating to key differences in the VET systems of different countries.

There were three stages to the project:
1. A review of the quantitative data available.
2. A review of other key literature.
3. A discussion group at an international conference.

In this report, we summarise some of the main findings from each stage. We conclude that, in general, there is a paucity of internationally comparable quantitative data relating to vocational education and training.