Inspera Observed User Testing (Original)

This guide explains what Observed User Testing (OUT) is; why you should conduct it for certain assessments; how to prepare for an OUT session in your school and how to run an OUT session in your school.

What is Observed User Testing?

Purpose of Observed User Testing

The purpose of OUT is to review the assessment as students will experience it online.

It is usually conducted for heavily weighted or hurdle assessments, such as end-of-semester exams. For ways to review other kinds of assessments in Inspera, please see Reviewing Assessment in Inspera.

An OUT session is where one or more assessments are tested by two people. The “tester” takes the role of the student and should be someone who has NOT authored the assessment. The “observer” should be the course coordinator who can authorise changes to be made if necessary.

The data you are interested in collecting is how “students” use and navigate their way through the assessment in Inspera. OUT may be able to identify problems that can be rectified before students sit their assessments, but if no problems are identified, that is useful data too.

OUT is not peer review. Peer review of the assessment should have been completed prior to the OUT Session. OUT is more focused on ensuring that tasks and questions in the assessment ‘work’ as intended and the students’ experience on laptop screens is satisfactory. It may be that things are noticed in the Inspera platform that won’t have been picked up in prior reviews, particularly if questions have only been reviewed on large desktop monitors, on paper, or in PDF/Word formats. Testing the assessment involves checking the following:

Tasks and questions in the assessment 'work' as intended
Links, videos and audio tracks are functional
Images display properly
Information is laid out well on a laptop-size screen for students
It is possible to complete the question (e.g., labels that need to be dragged are not out of reach at the top or bottom of the screen)
Navigation instructions (e.g., scroll all the way to the bottom) or formatting instructions (e.g., the answer is not case sensitive) are provided where necessary
Everything students need to know to be able to fairly complete the questions has been included (e.g., is pinpoint accuracy important in hotspot questions?).

With respect to making decisions about what, if anything, needs to be changed, the point of Observed User Testing is not to prove or disprove something. It is to inform your judgment. There is no such thing as the “Average User”.

Observed User Testing Philosophy

Keep user testing simple so you do enough of it.
Usability testing usually produces useful results, no matter how poorly it is conducted.
Problems that are serious enough to do something about, are hard to miss.

Why should you conduct Observed User Testing?

If a link is broken, a video doesn’t play, or an image cannot be viewed, the assessment cannot be completed.

If any of these things happen in a time-limited assessment, this can be extremely stressful for students and technical assistance staff
At best, you may need to exclude those tasks or questions that could not be completed from the assessment
You may need to give extensions, which will probably inconvenience students, tutors who are marking, and yourself
At worst, you may need to organise a re-sit/deferred assessment. If these are final year students or this was a major hurdle piece of assessment, it could be an extremely stressful and inconvenient experience for all.

From a staff perspective

Depending on how many assessments you have that should be tested, you may want to run a ‘bulk’ OUT session to complete testing efficiently
OUT provides a date from which to work backwards to plan review dates and then author deadlines
An OUT session “primes” people to see errors that they may not see if they were in their offices by themselves glazing over as they click through endless numbers of assessments
OUT sessions are also a good opportunity for those who may wish to use Inspera going forward to see and complete an assessment in the platform; and to share what has been done in Inspera so far with colleagues.

How to prepare for an OUT session in your school?

By the time you do OUT, the assessment tasks and questions should have been written, vetted, peer-reviewed, and populated in Inspera (as a Question Set). After that, the following can be done:

Pick a date (if you haven’t already) when you’ll hold the OUT session and book a venue
Invite course coordinators and other relevant staff (e.g., other teaching team members, support staff or your school/faculty Learning Designer)
Remind people that their assessment has to be authored and reviewed one week before the OUT date. This is to allow time for the assessments to be set up in the Delivery module as Tests.

To set up the assessments to be tested

The OUT instance of the assessment should be set up using one-time users.

Switch off any random pull settings in the Question Set so that all questions can be tested in the OUT session
If you are testing an exam that will ultimately be delivered via Inspera Exam Portal, you need not test the exam in Inspera Exam Portal during your OUT session
Please see separate resources for assistance with assessment settings.
List the assessments to be tested and put testers and observers next to each one in a table
Remind attendees to bring their laptops and power cables for testing the assessment
Export your Question Set as a PDF (see the Exporting a Question Set as PDF guide for instructions), ensuring all questions are included (i.e. no random pull settings enabled at the time of export). Ensure the Question Set PDF is stored securely. You can annotate this document with any changes identified during OUT.
If you would prefer to annotate a paper copy of the Question Set, print your Question Set PDF and ensure this is stored securely. Don’t forget to bring pens to OUT!
If you intend to use one-time user accounts for testing, provide login details to testers (see example below)
You may wish to provide copies of/links to the Things to look for in review document to assist testers and observers during the OUT session.

Example login instruction card for testing using one-time user accounts

How to run an OUT session in your school?

Bring the information for testers and observers (e.g., on a PowerPoint slide or handout) that tells them:
- Log in to https://uqi.inspera.com
- The method they’ll use to access the assessment (TestCode or one-time user account)
- The name of the assessment they will be testing/observing
If observers are to annotate on paper, bring the paper copies of the Question Sets and pens
Attendees should bring their laptops
Testers log in to https://uqi.inspera.com
Observers open their PDF Question Set for annotating or have the paper copy of the Question Sets ready
The tester should complete the assessment like a student and talk out loud, describing any thought processes or mentioning anything that is unclear
The observer should make a note of any errors, ambiguities, or technical difficulties experienced by the ‘student’ which need to be fixed. The observer may wish to refer to the Things to look for in review guide for prompts
The observer may also learn useful information from looking carefully at the navigation behaviour of the ‘student’, especially if they are talking out loud about what they are thinking and doing. The observer may prompt for information, for example:
- What are you doing at this moment?
- You skipped that question - did you have a reason for that?
- You paused before you clicked on that button - why was that?
- What are you thinking right now?
- You’ve navigated backwards - why is that?
Provide a deadline for when amendments found in the OUT session must be made
Ensure any copies of the assessment (printed or electronic) are stored securely.