Validity is the key principle that should be at the centre of all good assessment.
Classic definitions of assessment validity describe it as the indicator of whether an assessment measures what it is intended to measure.
However, the same assessment could be seen as more valid in one context and less valid in others.
For example, if you were designing an assessment to understand how well an individual can play football, the tasks and observations that you might set for an adult would look very different compared to if you assessed a child.
This is a simple example, but it shows that there are thousands of ways that validity can be shaped by the context underpinning the assessment itself.
For this reason, many assessment professionals argue that we shouldn’t just be thinking about how well an assessment targets the knowledge, skills, or competencies we might be assessing, but also if the assessment allows us to confidently draw inferences about a learner based on their outcomes (e.g. their score or grade).
So rather than asking whether a given test is valid or not, we should really be asking whether there is evidence to support the conclusions that we might want to draw from the assessment.
What is assessment validation?
Assessment validation is the technical process of collecting and analysing various forms of data to arrive at a professional judgement on the overall quality (validity) of assessments.
It’s one of the most important aspects of educational assessment, because it’s an essential process to ensure confidence and trust in the assessment outcomes.
Awarding organisations spend a lot of time and resources validating their assessments. At Cambridge University Press & Assessment we have dedicated teams that conduct in-depth research, analyse various forms of data, and apply complex statistical techniques so that we can learn more about how well our assessments have worked.
The scale and long-term nature of these investigations means it can be difficult to know where to start with validation, particularly for busy practitioners who don’t have too much time to check the quality of their assessments.
However, with careful understanding of:
- the purposes of your assessments,
- the claims you want to make about them,
- the right methods to investigate them,
Validation will become a rewarding and essential process in making assessments work for you and your learners.
You can read more about this topic in a blog by Dr Simon Child comparing validation to your favourite courtroom drama.
The research behind validity
Cambridge University Press & Assessment has an extensive research history on validity and validation.
One key resource to read is the Research Matters paper: An approach to validation: Developing and applying an approach for the validation of general qualifications.
This research explores how validity should be understood and evidenced in practice.
Simply not as a result of a test, but as a structured argument about whether the judgments and decisions made from assessment results are defensible.
The paper explores questions like:
- Do the tasks actually reflect the knowledge and skills the assessment is supposed to measure?
- Are the scores dependable and consistent?
- Can performance on the assessment be generalised beyond the specific tasks students see on the day?
A key message from this work is that validation is not a one-off event. Every time you conduct an assessment, you need to consider the claims you want to make about students and how you can defend those claims to others.
Confidence in assessment outcomes is built over time, by continually gathering evidence, reviewing how assessments function in practice, and reflecting on how results are used.
Threats to validity
An essential focus of validity is identifying elements of your assessments that might present a barrier to assessing as precisely or fairly as you’d like to.
For example, if you gave someone an assessment that had questions and tasks that were not reasonably expected for the assessment, then it would be an unfair ‘threat’ to its validity.
In recent years, Cambridge has focused on how we can design assessments to open up maximum opportunities for learners to showcase what they know and can do, for example, by deepening our understanding of how candidates interpret the questions we set.
This has become even more important as new threats to assessment validity emerge in the world we work in. For example, assessment professionals are increasingly seeking to ensure assessments are inclusive and accessible, moving towards a universal design approach to ensure validity. Furthermore, the evolving landscape around technology and advancements with AI require us to be informed and adaptable in terms of our assessment design if we are to mitigate these risks.
Find out more
If you’re interested in learning more about validity and the other key principles of assessment you may be interested in A101: Introducing the Principles of Assessment or Dr Simon Child’s popular workshop on validation.
References
Shaw, S. & Crisp, V., 2020. An approach to validation: Developing and applying an approach for the validation of general qualifications. Research Matters: A Cambridge Assessment publication, Special Issue 3, pp.3–44.