QUIZZING
The inability to test trainee comprehension resulted in frustrating workarounds and low adoption rates in the training workspace. Our solution resulted in $1.4 million in expansion revenue, and a 84% adoption rate among existing users.
the problem.
Federal and international standards mandate that personnel in medical device companies demonstrate competence through documented training programs.
Documenting proof of compliance in our system was a lengthy and needlessly cumbersome process— downloading a quiz, completing it, and re-uploading it, at which point it would be scored by a training manager. This proved to be a major barrier to adoption.
the solution.
Hypothesis: Providing auto-graded, multiple-choice quizzing in-app will replace existing workarounds, providing a baseline solution upon which we can expand to include additional methods of capturing training efficacy.
We tested this hypothesis through a combination of customer interviews, existing feedback analysis, and early-stage user testing.
93% of customers validated our initial hypothesis.
As part of our discovery, I recorded, tagged, and analyzed over 65 insights. Some major themes were:
Question Definition
should be tied to specific documents. When a document changes, it might be issued in a stand-alone training. If questions were defined on the requirement level, trainees would lose context.
Transparency
should encourage “open book” policies, and correct answer choices should be revealed on failure.
Analytics
to evaluate quality.
Continued Relevance
of questions should be determined when a document goes through a change order. Managers should be able to determine whether a change warrants retraining and revising of questions.
After gathering this data, I created a workflow model and hosted several structured sessions with internal team members, including PMs, CSMs, and other designers. The goal was to identify areas where we were making assumptions, understand whether those assumptions had been rigorously tested, and understand what interconnectivity and dependencies existed elsewhere in the system.
“Remember, always, that everything you know, and everything everyone knows, is only a model. Get your model out there where it can be viewed. Invite others to challenge your assumptions and add their own.”
Key
Management at Requirement Level
Bidirectional Linking
Re-evaluation upon document change
Email Notifications
Multiple Reviewers and Edit Permissions
Trainee Submission
Document-based quizzing
Question Duplication
Automatic grading
Features
We then launched an unmoderated usability test of our prototype, gathering over 30 responses. We analyzed screen recordings and voiceovers, and generated heat maps to identify pain points and further validate our assumptions.
Among other insights, we found that 81% of respondents struggled to understand the distinction between quizzes for each document. This aligned with assumptions from internal stakeholders that our customers’ mental models tie training evaluation to individual documents rather than sets. As a result, we separated out quizzes by document, which served the secondary benefit of allowing us to display just one question per screen, reducing cognitive load.
the challenges.
Balancing Compliance and Transparency
1
One of the overarching themes we heard during discovery was the desire for transparency. Training managers wanted their trainees to feel empowered to refer back to the training material when taking a quiz.
Of course, we also wanted to ensure that our design followed the spirit of the quality regulations. In essence, we didn’t want to make the quiz too easy, but rather guide users in the right direction.
“I'm all about transparency. We're not here to fail anybody. We want people to be able to confidently go and find the answer or ask what the answer is. And, the harder you make training for somebody, the less inclined they're going to do it. And we already know that it's a hard thing to maintain 100% compliance on.”
— Discovery Participant
To strike this balance, we linked the related document alongside each question. This ensured that users had easy access to the relevant information, but encouraged them to sift through the material to find the right answer. This way, if they’re wondering about a specific procedure while on the job, they’ll know exactly where to find it.
Ensuring Continued Question Relevance
2
Training managers also expressed the need to create and edit quizzes before a training document is actually published. They wanted the ability to manipulate question and answer choices based on changes made to the document so that the training could go into effect immediately upon publishing the new version of the document.
Since there isn’t always a draft state, we needed a way to simultaneously alert users that a new version of the document existed in draft, and give them a way to edit the quiz related to the new version without altering the current version’s quiz. We solved this by introducing an alert banner that described the draft/published status and included a button that allowed users to switch between the versions.
This allowed us to use descriptive copy that would be clear for those new to the system, yet unobtrusive for experienced users. Also, showing the banner selectively based on the state of the document allowed us to reduce clutter where no draft version exists.
the results.
$1.4 million in expansion revenue
63% increase in unique users
51% reduction in time-to-task for trainees
A short video demo walking customers through the expected functionality of the new feature to prepare them for its release.
the lessons.
The central challenge of striking the appropriate balance between ease-of-use and regulatory compliance was a thin line to walk, but we were able to achieve a feature design that satisfied all end-users, from trainees, to training managers, to auditors. Quizzing was an excellent exercise in interpreting compliance standards in a way that felt natural for end-users.
We followed up with feature enhancements such as:
empowering managers to access quiz history
the ability to copy questions across requirements
optionality for moderated manager review alongside automatic grading
With more time, I would have loved to implement generative analytics to allow managers to evaluate question quality. We heard during discovery that metrics like failure rate could be useful in helping determine whether a question should be revisited on document revision. Empowering managers to make data-driven decisions about the quality of their training program could make it easier for medical device manufacturers to abide by the spirit of regulatory law and drive not just compliance, but true quality.