This protocol defines the accessibility testing methodology, acceptance criteria, and regression approach that will be applied throughout all subsequent project milestones. The protocol ensures that accessibility is evaluated consistently across Joomla frontend and backend interfaces using a combination of automated testing, expert manual evaluation, assistive technology testing, and end-to-end user journey validation.
Testing is performed against WCAG 2.2 Level A and AA requirements and ATAG 2.0 where applicable. Testing is performed against a representative sample of pages.
Accessibility testing follows a layered testing model.
No individual testing method is considered sufficient on its own.
| Testing Method | Purpose | Typical Issues Identified | Frequency |
|---|---|---|---|
| Automated testing | Detect machine-testable accessibility defects | Missing labels, colour contrast, ARIA misuse, duplicate IDs, landmark errors, heading hierarchy, missing form labels | On every pull request |
| Manual testing | Evaluate usability and requirements requiring human judgement | Keyboard operation, focus management, semantics, instructions, cognitive issues, visible focus, error recovery | Before publication of code changes |
| End-to-end journey testing | Validate complete workflows | Multi-page interaction, focus continuity, task completion, state changes, recovery paths | When introducing new components or pages or making significant updates to existing components and pages |
| Assistive Technology testing | Validate real user experience | Screen reader announcements, navigation order, interaction behaviour, dynamic updates | At least once a year |
Each layer complements the others and findings are combined into a single accessibility assessment.
The W3C Web Content Accessibility Guidelines Evaluation Methodology (WCAG-EM) provides a structured approach for evaluating the accessibility of websites and helps ensure that testing covers the full scope of a product rather than focusing only on individual components or known accessibility issues.
WCAG-EM can be used to support the accessibility of Joomla CMS, especially as part of manual testing. WCAG-EM provides guidance on using the methodology and considerations for specific situations. The conformance evaluation procedure is detailed under 5 main steps:
Define the scope of the evaluation - defining what is included in the evaluation; the goal of the evaluation; and the WCAG conformance level (A, AA, AAA).
Explore the product - identifying key views; key functionality; types of content, designs, functionality, etc.; required technologies.
Select a representative sample - guidance on structured and randomly selected views when it is not feasible to evaluate every view in a digital product.
Evaluate the selected sample - determining meeting WCAG; accessibility support for specific features; and recording evaluation steps.
Report the evaluation findings - aggregating and reporting evaluation findings; making evaluation statements; and calculating overall scores.
Automated testing tools can identify accessibility issues which are machine-detectable such as images with no alternative text and icon buttons missing an accessible name.
Axe-core is an automated testing tool which can be incorporated into unit test and end-to-end test pipelines and be used to test both components and views for machine-detectable issues. These tests can be run on every pull request or at regular intervals, such as nightly or weekly, to establish a baseline level of accessibility and then ensure that baseline is maintained or improved over time.
Axe-core should be configured to find issues associated with WCAG 2.0, WCAG 2.1 and WCAG 2.2 level A and AA. This can be done by specifying which tags should be included.
Axe-core can also be configured to show violations based on severity: critical, serious, moderate, minor. These severity levels are defined by Deque and not related to WCAG A and AA levels. If not specified, all levels are scanned. At a minimum, critical and serious violations should be tracked.
Example configuration with cypress:
it('Has no detectable a11y violations on admin page', () => {
cy.visit('/sample-url');
cy.injectAxe();
cy.checkA11y(null, {
runOnly: {
type: 'tag',
values: ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22a', 'wcag22aa']
}
includedImpacts: ['critical', 'serious', 'moderate', 'minor']
})
})
It is also possible to extract the checkA11y command and create a helper which removes the need to specify configuration for every scan on an individual basis. This also makes testing states easier.
The helper command:
Cypress.Commands.add('checkAccessibility', (context = null, options = {}) => {
cy.window().then((win) => {
return axe.run(context || win.document, {
runOnly: {
type: 'tag',
values: [
'wcag2a',
'wcag2aa',
'wcag21aa',
'wcag22aa'
]
}
}).then((results) => {
// Joomla-specific reporting/baseline handling
});
});
});
Page tests:
it('Has no detectable a11y violations on admin page', () => {
cy.visit('/sample-url');
cy.injectAxe();
cy.checkAcessibility()
})
State tests:
cy.get('.modal').should('be.visible');
cy.checkAccessibility('.modal');
The full list of configurable parameters can be found at Axe API documentation
No single criterion is fully covered by axe-core testing. For example, axe-core can detect missing alternative text for 1.1.1 Non-text content but it cannot assess the quality of the alternative text.
Covered by tags wcag2a and wcag2aa:
1.1.1 Non-text content
1.2.2 Captions (prerecorded)
1.3.1 Info and relationships
1.4.1 Use of color
1.4.2 Audio control
1.4.3 Contrast (minimum)
1.4.4 Resize text
2.1.1 Keyboard
2.2.1 Timing adjustable
2.2.2 Pause, stop, hide
2.4.1 Bypass blocks
2.4.2 Page titled
2.4.4 Link purpose (in context)
3.1.1 Language of page
3.1.2 Language of parts
3.3.2 Labels or instructions
4.1.1 Parsing (deprecated in WCAG 2.2)
4.1.2 Name, role, value
Covered by tags wcag21a and wcag21aa:
1.3.5 Identify input purpose
1.4.12 Text spacing
Covered by tags wcag22a and wcag22aa:
2.5.8 Target size (minimum)
The full list of rules covered by Axe-core can be found on Deque’s github repository: Axe-Core rule descriptions
In a well-established repository such as Joomla, there may be existing accessibility violations. The first objective is not “Joomla has zero accessibility issues”, but instead “A pull request cannot introduce a new accessibility issue detectable with automated tools”. This progressively helps to reduce the accessibility issues against the baseline and prevent regression.
In order to prevent existing violations from blocking new pull requests from being merged during the implementation phase of Axe Core, the use of severity levels can be applied incrementally rather than making all existing failures block CI pipelines. The subset of pages axe core is scanning can also be increased over time.
Suggested phases:
Introduce axe core and report violations as warnings but do NOT block pull requests. Document existing violations but exclude them from scans.
New critical violations block pull requests. All other violations are reported as warnings. Fix existing critical violations.
New serious violations block pull requests. All other violations are reported as warnings. Fix existing serious violations.
New moderate issues block pull requests. Fix existing moderate violations.
New minor issues provide block pull requests. Fix existing minor violations.
Axe core fully scans for all levels of severity against WCAG 2.0, 2.1 and 2.2 A and AA violations.
Unit tests can be used to cover a number of otherwise manual tests and can ensure that the accessibility of components does not regress over time when code is updated. Unit tests are best used to test component accessibility and can include checking accessible names, appropriate aria-* roles, aria state updates and keyboard functionality. The acceptance criteria can be used to write unit tests.
Here are some examples of tests that can be added for a button component.
The button component:
If the button is used as part of a disclosure widget, the disclosure widget should add the following tests:
The button that triggers the disclosure:
Other keyboard tests can also be added to the disclosure widget.
When the disclosure is visible:
Here are some examples of tests that could be implemented for a slightly more complex component, tabs.
The tabs component:
Code examples for the tab component:
it('has an accessible name', () => { expect(tablist.getAttribute('aria-label')).toBeTruthy(); });
it('makes only the active tab tabbable', () => {
expect(tabs[0].tabIndex).toBe(0);
for (let i = 1; i < tabs.length; i++) { expect(tabs[i].tabIndex).toBe(-1);
}
});
it('moves focus into the active panel on Tab', () => {
tabs[0].focus();
simulateTabKey();
expect(document.activeElement)
.toBe(document.querySelector('#panel1 button'));
});
it('marks only the active tab as selected', () => {
expect(tabs[0]).toHaveAttribute('aria-selected', 'true');
expect(tabs[1]).toHaveAttribute('aria-selected', 'false');
expect(tabs[2]).toHaveAttribute('aria-selected', 'false');
});
it('moves focus with ArrowRight', () => {
tabs[0].focus();
fireKey('ArrowRight');
expect(document.activeElement).toBe(tabs[1]);
});
it('wraps from last tab to first', () => {
tabs[2].focus();
fireKey('ArrowRight');
expect(document.activeElement).toBe(tabs[0]);
});
it('wraps from first tab to last', () => {
tabs[0].focus();
fireKey('ArrowLeft');
expect(document.activeElement).toBe(tabs[2]);
});
it('activates a focused tab with Enter', () => {
tabs[1].focus();
fireKey('Enter');
expect(tabs[1]).toHaveAttribute('aria-selected', 'true');
});
it('connects tabs to panels', () => {
tabs.forEach(tab => {
const id = tab.getAttribute('aria-controls');
expect(document.getElementById(id)).not.toBeNull();
});
});
it('gives every tabpanel an accessible name', () => {
panels.forEach(panel => {
expect(panel.hasAttribute('aria-labelledby')).toBeTrue();
});
});
it('updates tabindex when the active tab changes', () => {
tabs[1].focus();
fireKey('Enter');
expect(tabs[1].tabIndex).toBe(0);
expect(tabs[0].tabIndex).toBe(-1);
});
In the examples above, the accessible name was tested by testing the presence of an aria-label. It is possible to provide an accessible name in several ways and there are packages that can help with testing accessible name computation, such as dom-accessibility-api.
import { computeAccessibleName } from 'dom-accessibility-api';
describe('Tabs', () => {
it('gives the tablist an accessible name', () => {
const tablist = document.querySelector('[role="tablist"]');
expect(computeAccessibleName(tablist))
.toBe('Account settings');
});
});
Manual expert testing identifies issues that automated tools cannot reliably assess, including semantics, interaction design, keyboard behaviour, focus management, error handling, status feedback, cognitive load and complex workflows.
Manual testing should be completed before publication of code changes, for example when:
Testing should prioritise functionality that presents higher accessibility risk, including:
WCAG 2.2, WCAG-EM 2.0 and Acceptance criteria can be used to support manual testing efforts. WCAG 2.2 success criteria define the accessibility requirements being evaluated. WCAG-EM provides guidance for structuring the evaluation, determining the scope and selecting representative pages and functionality for testing. Acceptance criteria provide component- and feature-specific requirements based on WCAG requirements in a readable format.
Where a large number of pages or views use the same templates or components, representative sampling may be used. The sample should include common functionality, different page types, high-risk functionality and recently changed areas. Additional randomly selected pages may be included to broaden coverage. The current representative sample is included in 5.1 Mandatory User Journeys.
For the representative sample, we used the Mandatory Audit Units as provided during the proposal process. This made it easy to reproduce results going forward. That said, as more and more of the sample gets accessibility improvements, you might decide to replace individual views with those in the Optional Audit Units section and beyond.
The tested samples were (including the tested URLs where applicable):
In addition, we selected these three samples for the ATAG 2.0 report, as they had applicable functionality to be evaluated using the Authoring Tools Accessibility Guidelines:
The manual testing process consists of tests against the WCAG 2.2 AA and ATAG 2.0 AA criteria. Each success criterion checked is a checkpoint.
WCAG and ATAG only determine if a success criterion is passed or failed. The success criteria explain how content (or, in the case of ATAG, editing experiences) needs to be built. For example, WCAG’s 1.1.1 Non-Text Content describes the handling of images (and other non-text content), and what information needs to be given to pass the criterion.
In this case, identify all instances of Non-Text Content and determine if it meets the requirements for the specific case. This can be aided by using tooling like Polypane’s Images Outline view or bookmarklets.
Generally, when determining passing of a checkpoint, trust that the content is intentional, do not try to add or interpret the content. Determine something as a fail when it is clear that it does not meet the requirements.
WCAG does not ask for perfect websites. It provides a base layer of accessibility for everyone (at A and AA levels) to allow assistive technologies to do their thing. This often means letting bad practices slip as they are not violating the success criteria. A link with a button role might not be how you would write it, but in the testing phase, it is usually not a failure.
To determine if a checkpoint is met, consult the success criterion text first, then
Note that the ATAG implementation guidance was not updated since ATAG’s release in 2015 and thus has some optimistic assumptions on ARIA support. For example, the ATAG implementation guide alludes that landmarks would be sufficient to make keyboard use more efficient, while the more recently updated WCAG guidance reflects that keyboard-only users have no access to navigate between landmarks.
To test for accessibility support, especially for complex interface elements, consult assistive technologies like screen readers or voice control software to ensure that information properly is reflected in the Accessibility Tree and in practice in assistive technology. Keep assistive technology set to its defaults. Sometimes those defaults might not reflect your expectations, for example some screen readers might not by default read descriptions for some elements. This is not a failure as long as the information is available for the assistive technology to use.
While the initial audit uses the CAAT platform for testing, any testing platform will do, where possible, exports for other tools, like the free W3C-developed WCAG-EM Report Tool have been provided.
Many issues can be reproduced on multiple samples, for example issues with the logo or common elements on all pages. Instead of creating multiple issues, these issues should be combined into single issues with the samples listed.
State clearly if an issue is a violation (of a checkpoint) or a best practice advice.
Severity of the issue is estimated by including the context of the issue and the consequences of the checkpoint being violated. A small issue that is more annoying than a barrier might have a comparatively high severity rating when it is present multiple times per sample. Conversely, a barrier might have a comparatively low severity rating when there are ways to accomplish the task anyway or it affects rarely used samples. Severity is a subjective rating and does not necessarily correlate directly with priority fixing an issue. The levels of severity used are:
Each issue then briefly describes the issue, its impact on people with disabilities, screenshots and suggestions for addressing the issue as applicable.
In some cases adding simple code examples is sufficient, in other cases, the solution is better described in words for implementers who can make decisions in context.
To ensure that accessibility is permanently embedded into new features and when fixing bugs introduce the following procedures:
Definition of Done: Define that certain automated and/or manual tests need to be performed before a feature is deemed “done”. For example, require a clean axe-core sheet and a confirmation that the Acceptance Criteria for the used components are met.
Pull Request Review: Especially for pull requests that touch on the user interface of Joomla, require reviews by people from the excellent accessibility community. Consider making these reviews a required part for merging the change to core. This way, accessibility issues can be caught before they reach users.
QA Validation: As outlined in this document, regular quality assurance of the whole system is essential to ensure that features are not only implemented in an accessible way on their own but also work well with features implemented elsewhere.
End-to-end user journey testing validates accessibility across complete workflows rather than isolated pages or components. While component-level testing confirms that individual patterns are implemented correctly, accessibility barriers can emerge when multiple components are combined into a larger interaction flow.
End-to-end testing verifies that users can complete meaningful tasks from start to finish. This testing can be done with a combination of end-to-end testing as part of the code base which can be integrated into CI/CD pipelines and usability testing with users with disabilities.
End-to-end testing built into the codebase can run on a regular basis to help prevent regression whilst usability testing should happen at least once a year.
The following user journeys have been chosen. They can be used both for end-to-end tests within the codebase and for usability testing. They represent targeted sampling of important Joomla functionality. They should be supplemented with additional journeys when new functionality or significant changes are introduced.
| No. | Journey | Path (indicative) | Tests |
|---|---|---|---|
| J1 | Backend Login & Orientation | Login → Credentials → Submit → Dashboard → Guided Tour triggers → Navigate sidebar to Content > Articles | Auth, Dashboard, Navigation, Focus, Guided Tour |
| J2 | Create & Publish Article | Articles list → New → Title → Editor text → Category → Insert image via Media → Save & Close | Edit form, TinyMCE, Media integration, Validation |
| J3 | Filter List & Batch Action | Articles list → Category filter → Select multiple via checkbox → Toolbar “Unpublish” → Perceive feedback | List view, Filters, Checkboxes, Toolbar, Status |
| J4 | Create Menu Item (incl. Dialog) | Menus → Items → New → Select type (modal) → Choose type → Fill form → Save & Close | Dialog focus trap, Form, Nested interaction |
| J5 | Frontend Login & Edit Profile | Frontend login → Credentials → Submit → Profile → Edit fields → Save | Frontend auth, Cassiopeia, Frontend forms |
| J6 | Change Global Configuration | Dashboard → System → Global Configuration → Switch tab → Change setting → Save | Tab navigation, Complex form, Save feedback |
The mandatory journeys should be supplemented with additional journeys when new functionality or significant changes are introduced. Additional journeys should also be prioritised based on risk, frequency of use, complexity, known defects and planned remediation work. WCAG-EM can be used to help in selecting representative pages, views and functionality.
Random sampling may supplement representative sampling to broaden coverage, but should not replace targeted selection of high-risk and representative functionality. The sampling method and selected pages should be documented.
Automated end-to-end testing can be incorporated into development workflows to provide repeatable regression coverage. A representative sample of pages, such as those found in 5.1 mandatory user journeys, should be tested.
In the same way that Axe Core can be incorporated into unit tests, it can also be used to scan whole pages. This may highlight issues that occur on a page level that may not be found at the component level. Axe Core and Cypress should test the initial state of pages. It should also test pages in different interactive states such as with modals open or forms with and without error states since many accessibility issues are only detectable after user interaction.
Axe Core can be configured to scan for different levels of WCAG and for different levels of criticality. At a minimum, critical and serious WCAG 2.2 A and AA should be covered.
See 3.1.1 Axe Core Configuration for an example configuration and links to documentation.
Note: Axe Core works best on static pages when scanning full pages. Animations can cause certain tests to fail, or provide inconsistent results on each run. One such example is color contrast tests can fail if the scan happens mid animation. If possible, turn off animations when scanning pages.
End-to-end tests can be used to cover a number of otherwise manual tests and can ensure that the accessibility of components does not regress over time when code is updated. End-to-end tests can include checking keyboard functionality, use of aria-* attributes, focus order, focus management and keyboard traps.
Here is a simple example of a cypress test for navigating to an admin panel, creating an article, entering a title and uploading an image with a keyboard.
describe('Article Creation and Publishing - Accessibility', () => {
beforeEach(() => {
cy.visit('/admin');
cy.injectAxe();
});
it('should allow content editor to create, save, and publish article with keyboard navigation', () => {
// Step 1: Navigate to article creation page
cy.get('[data-testid="new-article-btn"]')
.should('be.visible')
.focus()
.type('{enter}');
// VERIFY we actually navigated to the form
cy.url().should('include', '/admin/articles/new');
cy.get('[data-testid="article-form"]').should('be.visible');
// Step 2: Enter article title
const testTitle = 'Accessibility Best Practices in Web Development';
cy.get('input[name="title"]')
.focus()
.type(testTitle);
// VERIFY title was actually entered
cy.get('input[name="title"]').should('have.value', testTitle);
// Step 3: Upload image
cy.get('[data-testid="image-upload-btn"]')
.focus()
.type('{enter}');
// VERIFY dialog opened
cy.get('[role="dialog"]')
.should('be.visible')
.should('have.attr', 'aria-modal', 'true');
// VERIFY focus moved into dialog
cy.get('[role="dialog"]').should('contain', cy.focused());
// Upload file
cy.get('input[type="file"]')
.selectFile('cypress/fixtures/sample-image.jpg');
// VERIFY file was actually uploaded by checking for success message
cy.get('[data-testid="upload-status"]')
.should('have.attr', 'role', 'status')
.should('contain', 'Image uploaded successfully')
.should('be.visible');
// Close dialog
cy.get('[data-testid="dialog-close-btn"]').focus().type('{enter}');
// VERIFY dialog actually closed
cy.get('[role="dialog"]').should('not.exist');
// VERIFY focus returned to upload button
cy.get('[data-testid="image-upload-btn"]').should('have.focus');
})
})
Testing with users who rely on assistive technologies provides validation of real-world accessibility and usability. Expert testing can identify many technical issues, but users with lived experience provide insight into whether workflows are understandable, efficient and practical.
End to end testing with users should be performed at least once a year. Scripts for user tests have been pre-written for the most common user journeys to allow for comparable results over time.
An end-to-end journey passes when:
A journey fails when:
Assistive technology (AT) testing validates the real-world accessibility experience of users who rely on assistive technologies, including screen readers, voice input, keyboard alternatives and other accessibility tools.
While automated and manual expert testing can identify many accessibility issues, assistive technology testing verifies whether implementations work as expected in practice. Assistive Technology Testing can be completed as part of manual expert testing and as part of usability testing when testing end to end user journeys. It should be done at least once a year.
Both frontend and backend environments should be tested on desktop and mobile with the following screen reader and browser combinations.
Note that in backend environments, desktop and tablet versions are prioritized and mobile testing is limited to verification of basic operability for core user journeys on mobile. Full mobile conformance is not required for the backend.
| Assistive Technology | Browser | Scope |
|---|---|---|
| NVDA | Chrome | Required |
| NVDA | Firefox | Required (may be combined with Chrome where behaviour is identical) |
| JAWS | Microsoft Edge | Required |
| VoiceOver (macOS) | Safari | Required |
At least one mobile combination shall be tested.
| Assistive Technology | Browser |
|---|---|
| VoiceOver (iOS) | Safari |
| Talkback | Chrome |