One of the fundamental elements of assessment is the science behind good question writing.
Without a question that enshrines the key principles of validity, reliability and fairness, the answers you receive may not be reflective of what you’re trying to assess.
Brooke Wyatt is the Senior Assessment Advisor in the Assessment Projects Group in Cambridge Assessment International Education. She will be collaborating with assessment consultant, Nancy Sneddon for our forthcoming workshop series on Question writing for effective assessment this November.
In this Q&A, we discuss with Brooke her assessment background both as a science teacher and as Senior Assessment Training Manager at Cambridge Assessment Network, and how these experiences have informed her approach to assessment, especially in the context of question writing.
How has your background as a Science Teacher informed your assessment practice?
“I was a science teacher in a secondary school setting for 15 years. In the first few years, I was heavily reliant on using assessments that were provided with the schemes of work that my school bought. At this point, I didn’t put too much consideration into how good those assessments were at fulfilling their purpose.
In lessons, I used a range of tasks, taking advantage of the classroom environment and the time and space we had for a practical subject. Early on in my career, writing new tasks was challenging and I can give examples where, looking back, I was assessing my classes’ creative writing skills, or something that was not intended instead of the skills I was targeting.
In all subjects, you need questions that assess the range of assessment objectives. So not just questions that assess the knowledge, but how that knowledge is applied to an unfamiliar context. The challenging part of this is to find unique contexts without overcomplicating them or avoiding introducing reading skills that are not supposed to be assessed."
Over time I gained more confidence using a range of assessments and even writing my own."
What inspired your transition from teaching to specialising in assessment, particularly in the principles of question writing?
“I wish I had a better story for this question. After so many years in one profession, I felt I needed a change and a fresh start where I could use my teaching expertise. I was fortunate to find a role at Cambridge that involved primary and lower secondary science and I absolutely loved it.
I will say that it was very eye-opening to see that the process of writing a question paper took so long!
Having since developed my understanding of the assessment cycle, it now makes sense why each stage of the process is there and how valuable it is to ensure questions and the entire assessment are valid, reliable and fair.”
What do you consider the most common challenges that individuals, especially those new to question writing, encounter in creating effective assessment items?
“The biggest challenge is that there is so much to consider when writing good questions! You must be creative to think of new contexts and come up with inventive ways to cover the same learning objectives, but you also need to think about the language used in the question and ensure it is unambiguous.
In all subjects, you need questions that assess the range of assessment objectives. So not just questions that assess the knowledge, but how that knowledge is applied to an unfamiliar context. The challenging part of this is to find unique contexts without overcomplicating them or avoiding introducing reading skills that are not supposed to be assessed.
Another big challenge is finding the time. Writing questions is time-consuming and will require more than one person, someone needs to edit the work.”
What key principles do you emphasise when teaching individuals how to write assessment questions? How do these principles contribute to the overall quality of assessments?
“The key principles we include in our assessment training are validity, reliability, and fairness.
A valid assessment measures what it claims to measure. For example, a science exam should test the knowledge, skills and understanding of the science, and not something else, such as reading comprehension.
Validity also refers to how the assessment outcomes are used, and the inferences made. Often, assessment outcomes are used for a range of purposes, some of which were not intended in the assessment design.
The more you understand about the assessment purposes and the limitations of the results, the better. A large part of validity is fairness. There should not be anything in a question that keeps a learner from being able to access and answer it. A reliable assessment is consistent and repeatable, and the same results should be the outcome.”
Could you provide insights into how question writers can ensure accessibility in their assessments? What considerations are crucial for making questions inclusive?
“Accessibility considerations are so important when writing assessments. You want to make sure the language used is clear, with simple vocabulary and sentence structure. Clear instructions must be used for each question to avoid ambiguity - which can be a barrier to understanding the intentions of the question.
Questions must be free from bias, stereotypes, or culturally specific references that might disadvantage any group.
In addition, images can introduce misconceptions or confusion. When using images in a question you must consider its value and if it is completely necessary.”
In your years of experience, how have you seen question writing practices evolve, and what trends do you anticipate in the future?
“I have been fortunate with my experience working at Cambridge University Press & Assessment as I have worked with teams with extensive experience.
We rely on processes informed by years of research, and we are always looking for ways to improve.
Accessibility considerations are so important when writing assessments. You want to make sure the language used is clear, with simple vocabulary and sentence structure. Clear instructions must be used for each question to avoid ambiguity - which can be a barrier to understanding the intentions of the question."
The obvious trend will be towards more digital assessments. The questions and tasks presented digitally still need to consider all the same principles as on paper.
However, there are new considerations in terms of validity when the subjects are taught using paper in the classroom but assessed digitally, and of course, ensuring that accessibility is still vital. A digital assessment must have all the same accessibility considerations as on paper, but also have the benefit of tools that may even allow for enhanced access arrangements.
A huge benefit of digital assessment is auto-marking which results in timely feedback to the learners. This can be built into the question writing process as the mark scheme may need to include reasons for incorrect answers as well as an indication of the correct answer."
What advice would you give Brooke Wyatt the teacher on question writing and assessment?
“In my experience, teachers are not given nearly enough training in assessment. I would encourage my former self to learn more about the principles of assessment to reflect on the quality and effectiveness of the assessments I used.
This would also help with understanding the limitations of some assessments and not to make inferences that were not valid.
Finally, I’d tell my former self that good assessments take a long time to get right. It can be a team effort as reviewing each other’s work can save time while improving the quality.”
Beyond the workshop series, what advice would you give to question writers for continuous improvement in their craft? Are there specific resources or practices you recommend for ongoing learning?
“As there is so much to consider when writing questions, it is useful to develop a checklist. This can include generic things to look for when reviewing your own or others’ questions, but also some subject-specific things to look out for.
This will reduce errors and ensure that you learn from the mistakes you will make.
Again, I can recommend working as a part of a team. You will learn from each other, share the workload and act as a critical friend to each other to improve the questions that you have written.”
If you want to develop your question-writing skillset, join Brooke on her upcoming workshop series on the topic.