Master thesis at DTU, developed in cooperation with Novo Nordisk
Novo Nordisk was investigating Augmented Reality (Microsoft HoloLens2) as a complementary tool for training shop-floor operators, alongside their existing peer-to-peer "4P" training process. The problem: no evaluation framework existed, in the literature or inside the company, to determine whether AR training actually improves training and production outcomes in a pharmaceutical, GMP-regulated setting, or how it compares to the training methods already in use.
My task was to develop and validate a Protocol, the UX Performance Evaluation Protocol (UXPEP), that lets Novo Nordisk's process supporters measure the UX performance of AR training in a structured, repeatable, and compliant way. The work was framed by two research questions: what factors shape an operator's experience of AR training in this context, and how can effectiveness, efficiency, and satisfaction be measured in practice.
The project followed the Double Diamond design process across Discover, Define, Develop, and Deliver. The Discover and Define phases combined a literature review with stakeholder mapping, contextual interviews, and user journey mapping to identify who the Protocol had to serve: operators being trained, the process supporters who would run the evaluation, the Augmented Instruction authors, and the XR experts responsible for scaling AR at Novo Nordisk.
Every design decision after that point was tested against a mixed-method approach: quantitative data from timing, error rates, and standardized questionnaires (SUS, NASA-TLX, NPS), combined with qualitative data from observation, think-aloud sessions, and interviews.
The Double Diamond framework structured the project from open-ended research through to a delivered, field-verified Protocol.
The Protocol couldn't be designed from a desk. It was built and revised through repeated field testing at a Novo Nordisk production site in Måløv, run directly in the clean room where the trained process actually takes place.
The first round of field tests, in November 2023, put the AR training itself under observation: six operators, ranging from AR-curious newcomers to skeptical senior 4P trainers, performed a real Augmented Instruction while I ran direct observation combined with the think-aloud method, recording sessions through the HoloLens2 and following up with a post-session questionnaire. This surfaced concrete usability issues, physical strain and screen flicker from the headset, older operators needing more guidance to operate the device, younger operators exploring more freely, that fed directly into the Protocol's design requirements and into personas built from the demographic and behavioral data collected.
Later iterations shifted the field test's focus from the AR training to the Protocol itself: Novo Nordisk's XR experts applied draft versions of the UXPEP as process supporters, on real training sessions, so its feasibility in a live, time-constrained GMP environment could be assessed rather than assumed.
More experienced operators consistently rated the AR training's potential to reduce their own training time lower than less experienced operators did, and reported higher perceived workload. Demographic and experience data had to become a required input to the Protocol, not an afterthought.
Process supporters running the UXPEP under real time pressure couldn't reliably follow a full SOP document mid-session. A concise, printable step-by-step Cheat Sheet was added and field-verified as the difference between a Protocol that works on paper and one that works on the floor.
Recording sessions on the process supporter's own work phone meant they couldn't be reached by phone during a session. The Protocol was revised to remove this and similar dependencies, so one missing device or signal couldn't stall the entire evaluation.
The final Protocol, delivered as an SOP, Excel tracking file, and Post-Session Questionnaire, structured around what to do before, during, and after each training session.
The final UXPEP identifies three components that shape an operator's experience of AR training, user characteristics, the system (HoloLens2 and Augmented Instruction), and the task context, and provides Novo Nordisk's process supporters with a compliant, SOP-formatted way to evaluate all three, before, during, and after a training session.
In its last field-verification round, the expert who ran the Protocol as process supporter rated it 87.5 on the System Usability Scale and described the balance between the effort required to run it and the effort required from the trainee as "realistic to use in a production setting for all involved." That result validated not just the AR training being evaluated, but the evaluation tool itself, giving Novo Nordisk a repeatable way to compare AR training against its existing peer-to-peer process using the production metrics it already tracks.
This case study covers Novo Nordisk internal process detail and is protected.
Incorrect password. Try again.