Version: Jul 7, 2026
Step 1: Preparations and recruitment 3
Step 3: Perform and document test 6
Example: how a test situation might look like 7
Instructions to the test leader - how to facilitate the test 7
Step 4: Gather results and analyse 8
Recommendation for the analysis 8
User tests are one of the main components of accessibility testing and capture the potential accessibility issues that real users may encounter. Use this guide to ensure user tests are performed in a structured, cohesive and repeatable manner within the Joomla Accessibility Programme.
If you follow this guide you will not only get qualitative insights where you are in the process, you will also be able to track progress over time.
The guide contains the complete list of relevant steps to perform user tests, if you already have certain steps done or need to scale down, feel free to adapt this process to your current situation.
Below are links to the resources you need to set up and perform user tests. In the descriptions of the steps in this process you will find more information about how to use them.
 Document type |
Link |
|---|---|
| User test result, documentation template | Spreadsheet, editable |
| Manuscript, instructions and test scenarios template | Google document, editable |
| Accessible Usability Scale (AUS) | Accessible Usability Scale (AUS) - Fable |
Before setting up and performing user tests you need to take some initial steps. Some of these are already in place for the first time of user testing.
Define product or parts of the product that are to be tested
Define the user group that you need feedback from. For instance, in need of feedback on cognitive accessibility, aim to recruit user testing participants who have cognitive disabilities.
Arrange for suitable compensation for the participants. We at Axess lab give participants a gift card at the end.
Recruit and book participants, either remotely or on site. Preferably book participants at least 2-3 weeks in advance, and make room for other preparations in the time leading up to the tests.
Be mindful of how you invite people. You need to give them clear information about the situation such as: what is the task? Why are they doing it? Where is it? When is it and how long? Who is participating in the meeting? What does the person get out of it? Do they need to bring something?
Tip: Add images of things that prepare the person for the situation, such as a picture of the people leading the test, so they know who they are meeting. Or perhaps of the entrance to the building where the test takes place.
If possible, add an observer in the booking. So the test leader, perhaps you who read this, get support in taking notes and listening.
After the bookings are done and you know what you are testing and with who you are testing with. Now we need to set up the actual tests.
Prepare the manuscript for the tests. We have provided you with a template to start from (Manuscript, instructions to user and scenarios (template)). Prepare a manuscript that is to be used in all user tests.
In that manuscript we have two sections:
Instructions about how the test is done. You can adapt this information to the current context. One of the main goals for this section is to communicate clearly that it is not them that is to be tested, they can’t do anything wrong. If they do not succeed with a task for example, that is only really good feedback for the system’s usability or accessibility, not a failure on their part. Another part to emphasize is that they need to talk about what they think about while testing.
Scenarios and tasks. Usually it’s good to start off with a scenario, so the participant can imagine that situation and act from it. You can add tasks and follow up questions to these scenarios.
Tip: It’s good to prepare the main scenarios and tasks that you want to test, but try to keep them down to a reasonable amount to have time for them.
Update a copy of this template for the documentation: Results user tests, assistive tech in Joomla (template).
Add the correct number of users in the overview with relevant
information about the participant’s situation (you can complete what you
don’t know in the user test, for example if they use any assistive
tech).
Adjust the tabs for each user documentation to fit the user
journeys and tasks that are covered in these. In our version used for
the august 2026 user tests, we have all user journeys that were provided
already there, but we changed the order to make sure we have time for
the most prioritised.
You may decide if you wish to use the time parameter. Depending on how you perform the test it is useful, for example if you observe the participant with little intervene it is more reliable than if you discuss experience and the user for example goes back to summarise experiences. In these tests with screen reader users, we find that it often results in the latter, for the test leader to understand the screen reader experience, hence the time parameter gets a bit unreliable to use.
Add a tab for each user containing these pre-filled user journeys you test. You also need to update the formula on the result summary page. This is due to the calculations on the results summary referring to the name of each user test tab, so if you change the number of tabs for the user tests and the names of these, make sure to update the calculation on the assessments in the summary accordingly.
Here is the summary page pointing out the number of tasks completed
independently reported on all user tests. The formula counts the
occurrences of “Completed independently” in column D on each user’s
sheet and adds them together.
When you have all the documents in place for the tests to begin, it’s getting close to the actual tests. But before that we know that many people, especially participants that might have cognitive disabilities, need a reminder.
The day before the test, write to the participant a reminder about the upcoming test. If you are doing the tests on zoom or Teams, include instructions on where they can call you if they run into technical difficulties. If you are doing them on site, it is still wise to include a number they can call if they need to for other reasons. Usually text messages as SMS is preferred for people, but adapt to how they wish to be contacted.
These are preferred roles on each test:
Greet the participant, presentations of all in the room
Test begins: test leader read through instructions not just
verbally, share screen or have a printed paper with these to read. After
that the participant gets ready to share screen or position so the test
leader see their screen in the room.
If test is to be recorded, start this now.
Scenarios. The test leader goes through scenarios and listens to
the person reflect and observe actions. Meanwhile, an observer document.
The test leader might take simple notes to add to the main documentation
later.
The test leader keep track of time and when there is
approximately 10 minutes left its time for wrap up.
AUS: you can use different wrap up questions. We suggest using the Accessible Usability Scale (AUS) - Fable since that both gives us the participants overall thoughts of their experience and a quantitative number that helps comparison over time.
Stop recording.
Thank you and information about gift card.
If you are not used to leading user tests, don’t worry! It is not rocket science. What’s most important is that you are present in the moment, listen to the person and curious. And probably most importantly make sure to signal that this is a relaxed and safe environment, that makes room for peoples organic reactions and thoughts.
Here are more advice beyond that foundation:
After each test we suggest planning time directly after note down observations in the main documentation. It makes the final result summary easier, if as much information from the tests gets summarised directly, when memory of the test is still good.
So when all the results are collected in the documentation file you should already have a quite nice summary of the general outcome when it comes to the rate of success or failure of the user journeys.
You can choose to analyse the results in different ways. Here are some perspectives and ways to tackle it, for example:
We suggest not spending too much time on over-documenting - that is a common pitfall, that eats too much time, making the user tests more difficult to run as often as they should. So, you already have a good overview of the results in the main spreadsheet, you don’t also need to write pages and pages with analysis that few or no one reads. We suggest that you put your analysis directly in a format such as a presentation, powerpoint. In that way you put your analysis in a format that is ready to also present to colleagues and stakeholders. This presentation is an important part for impact and sharing knowledge from these tests to other people, that in turn can make improvements based on these results.