Graeme Arnott is an Assessor for SECTT (the Scottish Electrical Charitable Training Trust) which is the managing agent for the delivery of the electrical installation apprenticeship in Scotland. In this member spotlight Graeme tells us about the 'rich complexity of assessment' and offers insights on nervousness as a construct irrelevant variance.
Reflections on assessment in electrical apprenticeships
"Once upon a time I had a superficial view of assessment. Becoming involved this year in a review of the Scottish Joint Industry Board's assessment of apprentice electricians and joining The Assessment Network at Cambridge has enabled me to explore in depth, the dazzlingly rich complexity of assessment.
Approximately nine hundred electrical installation apprentices are taken on each year in Scotland and their apprenticeship lasts up to five years. My daily work involves two assessments that define an electrician's status: the Final Integrated Competence Assessment (FICA) and the Approved Competence Assessment (ACA).
FICA is the summative, high-stakes, synoptic assessment that concludes the Scottish Government approved Modern Apprenticeship and is the gateway to qualification and regrading as an electrician. ACA is a voluntary assessment taken by electricians who want to develop their role in the industry by becoming approved electricians."
Understanding FICA
"FICA is rightly a rigorous assessment. It takes place over two days and has six distinctly separate sections: Assessment of Safe Working Practices, Composite Installation, Inspection & Testing, Safe Isolation, Fault Diagnosis and Rectification, and Underpinning Knowledge. It’s often compared to a driving test.
The apprentices that are assessed are employed by a wide range of companies from small-family businesses to large multinationals and they are all involved in a myriad different aspects of electrical installation work. The sheer variety of workplace settings means that FICA aims to assess what is common to all these different possibilities.
FICA isn’t, in this sense, an attempt to assess whether the apprentice is good at their job but instead whether they can demonstrate that they have gained the fundamental capabilities to work in any of the jobs available to an electrician. It might seem paradoxical for a vocational assessment, but it is FICA’s simulated nature, its inauthenticity that contributes so powerfully to the quality of apprentice training, and thereby the quality of electrical work carried out in the real-world."
Nervousness as a 'Construct Irrelevant Variance'
"When I was an electrician assessing the performance of my apprentice in the workplace, ‘good enough’ simply wasn’t good enough. Poor quality electrical work can lead to dangerous situations that can result in a threat to life and the destruction of property. So, I expected a lot from my apprentices.
FICA, in its simulated nature, serves a different yet equally critical purpose: ensuring every electrician possesses the fundamental capabilities to ensure safety and quality across all potential work environments. It takes place in an unfamiliar setting with unfamiliar people. The contrast with their usual work environment can be unnerving. One candidate tellingly described the centre as "eerily quiet."
All that pressure, whilst being observed, and in an unusual place not surprisingly gives rise to a lot of nervousness. And that nervousness sometimes gets in the way of success."
I became interested in nervousness as a construct irrelevant variance (CIV) not only because it threatens the validity of the assessment but also because it lowers the pass rate. Only recently we had a candidate faint (thankfully I caught him) and another candidate needed prescribed medicine to help him cope."
Practical approaches to overcoming nervousness
"One of the maddening features of nervousness is that the more you try to convince yourself not to be nervous, the more nervous you become. I’ve been looking at three ways that might help overcome the problem:
- Control: Being properly prepared is the best way to avoid nervousness. Educators play a key role in training apprentices in deliberate practice.
- Forgiveness in Assessment: Inspired by UX design principles, I’ve been exploring how forgiveness might be built into the marking scheme or moderation process without compromising integrity.
- Mastering the Moment: Developing mechanisms for coping with test-anxiety and harnessing stress energy.
Nervousness as a CIV exceeds expected nervousness, so it’s about mitigation in advance, recognition in the marking, and self-management in the moment. All easier said than done, of course!"
Less distress, more success
"It’s a sheer pleasure to observe a candidate who knows what they’re doing and is aware that they know that they’re performing well. It’s quite another thing to look on as a panicking candidate stares wildly around themselves looking for an escape route.
I started to explore how cognitive load theory might help these candidates make sense of their surroundings. Simple acts like finger-pointing and looking to the start and end of a process can establish neurological and visual pathways that reduce cognitive load and increase confidence."
When learners master their craft
"I left school when I was fifteen, and I’ve worked in this industry for over forty years. I started to become interested in training and assessment not long after I’d qualified as an electrician when I was entrusted with an apprentice.
More recently when I’ve described what I’ve been doing on A101: Introducing the Principles of Assessment, I’ve been asked if that is specialist work or had it described as 'a bit niche'. It’s been strange to find that many people regard assessment as something slightly separate from education. And it’s been even stranger, though something of a relief, to find out that I’m not the only one who has had that experience.
What I find most rewarding about working in assessment is witnessing the transformation in individuals as they demonstrate that they’ve mastered their craft." I hope that through rigorous, yet thoughtful, evaluation, we contribute to maintaining the high standards that keep our industry, and the public, safe.
Even after so much time in this industry there’s still something new to learn, some new complexity to unpick. The biggest takeaway from the A101 course for me has been that some of that complexity has been elucidated. Every week on A101 has raised an issue that I’d like to explore further. I wish I’d done it years ago."