How do you go about choosing an accessible content management system (CMS)? What are the main criteria for success? And how to ensure ease of use for authors including screen reader users?
The Centre for Inclusive Technology (CFIT), which is based in the headquarters of the National Council for the Blind of Ireland (NCBI), looked at several popular CMSs in order to assess which would be most suitable.
Our approach was to look at how these CMSs work out of the box and no complex heuristics were applied in order to simulate how many other users would approach the adoption of a CMS in the real world. The assessment method was an intuitive approach with some basic core tasks such as adding content and administration.
Expert Screen Reader Evaluation by Paul Traynor CFIT.
Author: Joshue O Connor Senior Accessibility Consultant CFIT
- What are the main criteria for success?
- How to ensure ease of use for authors including screen reader users?
- How accessible and usable are some of the more popular open source CMSs?
- Plone and Drupal Advanced Features comparison
- Modifying the Interface
- Accessible WYSIWYG Editors
- Practical Alternatives to using the embedded WYSIWYG Editors
There are many commercial and open source CMSs available today and making the right choice can be very daunting.
Some factors to consider are the technical skill of the staff who will use it and the behind the scenes work needed to set up and customise the system. The amount of time it can take to properly check the viability of a CMS is itself something that can put many off choosing and implementing a reasonable open source system, so many will go to companies who offer a commercial package they have themselves developed and offer to take care of the content migration and customisation. For those with a large budget this may be a viable solution, but for community based groups, disability advocate groups and others with little or no budget, are there systems available that can make setting up and administration a plausible option with a nominal amount of technical knowledge needed?
For the purpose of this paper we are going to look at some popular and free CMS packages that could suit the needs of groups like these. Aside from basic set up and any related costs including hosting, etc., we need to consider if the successful system will be capable of outputting standards compliant and valid code. This whole exercise can be a daunting task for people who are not technically minded. Many may not even be aware that there are efforts to ensure that accessibility is already considered and, where possible, built into the CMS. This under the hood accessibility is courtesy of the Authoring Tool Accessibility Guidelines (ATAG). These guidelines are there to help CMS developers create tools that have a user interface that is accessible, as well as outputting accessible content. The guidelines aid the developer who builds the system to take care of any automatic processes and to intervene if some input is needed from the user in order to improve or enhance the accessibility of the content. However, there is also a responsibility on the part of content authors to ensure that they are endeavouring to create good accessible content. There may be an unrealistic expectation that whatever the author throws at the system will be magically transformed.
What are the main criteria for success?
Our goals were to find:
- A CMS which is functional, accessible and usable to authors who have basic IT skill level and knowledge of using word processors, but no fluency with markup languages like HTML.
- A CMS that can be recommend and promoted as an accessible Web content management solution for community/disability groups, charities and others with a limited budget.
- If possible, the WYSIWYG editor should be accessible to screen reader users. The usability of the mechanism is also important, so we are not merely considering if it is technically accessible but also if it is easy and pleasant to use.
How to ensure ease of use for authors including screen reader users?
All the CMSs that we examined were tested by an experienced screen reader user with a strong technical background. He is a power user with many years of experience using various screen readers such as JAWS and Window Eyes. The CMSs were evaluated in a critical manner with attention to the quality of the markup and the finer touches and details that can be employed by developers to enhance the user experience for blind people.
CFIT pay particular attention to usability in our testing of any ICT. We are not interested in merely ticking the boxes or minimal compliance but in a pleasant and rich user experience that facilitates better interaction, whether the user is creating content or administrating the CMS.
How accessible and usable are some of the more popular open source CMSs?
In order to give these tests a real world flavour and to ensure they were ecologically valid, we consciously did not use any particular testing method or script in order to access how intuitive these systems are out of the box.
For the tests we looked at:
- Quick and Easy
- Expression Engine
Our feedback derives from user testing and observation of a screen reader user performing the following basic tasks, as well as the same tasks being performed by a user with no Assistive Technology/Special User Agent requirements and with an average computer skill level.
These tests were facilitated and observed by an experienced CFIT accessibility consultant.
The tasks included:
- Uploading content and, where possible, editing and formatting content (using a WYSIWYG editor).
- Creating new pages (Category/Section headings and sub categories/headings).
- Basic administration of user groups and permissions.
For testing the WYSIWYG editors we used a similar methodology to Peter Krantz of Standards Schmandards.
These tasks were performed for all of the CMSs that we tested. The results are not conclusive and are included here to be indicative of our experience and subjective assessment of the pros and cons of using each CMS.
Before we started looking at Jadu, we contacted the developers to ask about the accessibility of their CMS. They replied, pointing out their
The developers then stated that they have plans to introduce an accessible user interface in forthcoming releases.
Visually, the graphic style of the Mambo interface was pleasant to work with and the style of the Windows operating systems graphics would no doubt be appealing to many users and would not be too much of a departure from what they are used to, so this could be an advantage.
From the perspective of instant usability, our first impression of Mambo was quite good, but on starting to perform the tasks we found it difficult to then find the uploaded content.
Mambo has a peculiar method of structuring its content with a curious mix of sections/categories that we found to be confusing, but once the user had familiarized themselves with the way the system works then it could be comfortably used. However, using Mambo was found to be difficult for a screen reader user so we could not confidently recommend Mambo as a usable and accessible CMS.
Expression Engine is a blog focused CMS which lacks more advanced functionality.
From a screen reader users perspective it was frustrating to use. There were no structural headings and the blog feature was very difficult to navigate and figure out. Title and edit boxes were not marked up well and our tester found it unintuitive.
From a usability perspective our first impression is quite good and we liked the interface. It is visually quite smart and we found it quite intuitive to use. Adding posts is very easy as is creating links. It automatically asks for additional information, such as the
title attribute, when creating a link. Creating a link is a matter of selecting your desired link text and then clicking the link button. These additional options pop up then.
Module installation is straightforward and there are an extensive list of utilities such as an SQL and Plug-in Manager. However, as it is inaccessible we will not recommend it.
Textpattern is primarily a blogging tool rather than a CMS, and a big downside to this application is the inability to have more than one level in the navigational hierarchy. Our testing with a screen reader revealed similar results to Expression. However, it has a very simple interface and textile text formatting system.
Textile is a way of formatting text without using a WYSIWYG editor.
Joomla is a new version of Mambo and it has made some great improvements in its overall accessibility.
For a screen reader user, we found it to read very well with JAWS and Window Eyes, with no focus problems and no mixing up of lists or combo boxes, etc. In our testing it was found to be Easy to understand, and enjoyable to use.
Some more specific positive observations were:
- All checkboxes were found to be well labelled and read correctly using Window Eyes, JAWS and both Firefox and IE6. Various menu settings such as user menu, main menu, etc. were checked.
- The log in boxes are fine, no problems with them.
Some problems encountered were:
- Some links reading On Mouse Over could not be activated by pressing the Enter key.
- Various items had checkboxes, etc. that weren't very intuitive. The labels didn't convey their purpose effectively to the screen reader user.
- Radio buttons read well but their labelling could be improved. They often were not understandable as to what purpose they served.
If these problems could be addressed, we would recommend Joomla.
Quick and Easy
We very much liked Quick and Easy. Some feedback from screen reader testing is as follows:
Very simple interface.
Easy to get around and not cluttered.
All items were well labelled and easy to understand as to their purpose.
Quick and Easy is made for a user who wants simplicity and a quick way to update web content.
I enjoyed my experience of using Quick and Easy.
All lists, list boxes, combo list boxes edit boxes read well with both JAWS and Window Eyes.
I noticed that none of the various elements read incorrectly with either screen reader.
Quick and Easy is a low cost commercial product and could be suitable to a wide variety of community based groups. The administration interface is accessible and standards compliant. There is a user determined choice of HTML tool, so different people managing the same site can choose whatever suits them. Quick and Easy has some advanced functionality, such as built in blog, a mailing list application and a photo gallery plug-in, but may not be as extendable as some other open source CMSs such as Plone or Drupal. However, we do feel it was one of the best we tested.
From a usability perspective our first impression of Plone was that it is not that intuitive. This is primarily down to the labels and page types but this could be improved, as Plone is highly customisable. It is very feature rich out of the box and this may be why it feels rather unwieldy and a little intimidating.
From a screen reader users perspective, we did find the following positive points, in no particular order of importance:
- Skip to content/Skip navigation links.
- Good heading structure.
- No Frames.
- Form elements are well marked up.
The negative points are:
- Overall lack of consistency between what elements are visible when in forms mode/virtual PC mode.
- The naming conventions for items used in the interface are a little unintuitive. Use of terms like Smart Folder wasn't great, and we had no idea what a Smart Folder does. However, reading the manual would no doubt shed some light on this, which as previously stated, we have not done for this test in order to assess the CMS's level of instant usability.
When we tried to add a page we found two frames not correctly or not clearly labelled.
We also noted some peculiar (and inconsistent) behaviour when using JAWS 5. Pressing the F key (or tab key) should allow the user pick out elements such as buttons, edit boxes, list boxes, etc. and this worked fine. However, when using the arrow keys to navigate through these elements, no associated labels or identifying information were read out. Now we have to again test this with JAWS 7 and also other screen readers, such as Window Eyes, to see if this observed behaviour is consistent or something peculiar to JAWS 5.
Overall, Plone was a good CMS and highly customisable and extendable. We would recommend it.
Xoops has turned out to be a favourite during our tests with our screen reader user who was very pleased with it. He found it easy to use, understandable and enjoyable. Form elements are well labelled and the layout is simple and uncluttered.
However, from a usability perspective we did not like Xoops. We found the interface unintuitive and bare and we just did not like the feel of the application.
In terms of the interface out of the box, Drupal takes the opposite approach to Plone. The interface is simple uncluttered and clean. They adopt a modular approach which allows you to customise Drupal so it fits your requirements, adding other modules such as a blog, etc. as you need to, and thereby presenting the user with a simple to understand and easy to navigate interface, which does not overload the user when they first open the application.
Drupal is well laid out, with good headings/structure. Access controls and checkboxes are marked up well.
It could however be improved. For example the labelling of checkboxes in the blog administration page is not very good. There are checkboxes that allow the administrator to set permissions for anonymous and authenticated users that are also not labelled very well. This makes it difficult for a screen reader user to administer the site well as they cannot associate each checkbox with its relevant command.
However, these are our first impressions and we feel that Drupal is one of the best that we have come across and would recommend it with some customisation.
Typo3 has a confusing interface. It uses a tree structure but the interface is inconsistent. The language used is also very technical, which would put off novice users. It uses inaccessible language for buttons/commands like
Reload the tree from the server. Even adding a page/content is difficult with more unusable tech speak like
the page module is activated from the backend menu being used.
Typo3 also uses frames that are not correctly labelled (if at all) and there is a lack of simple heading structure.
There are however edit boxes/buttons with are labelled well but there are also peculiar behaviours. For example, when jumping across links our screen reader user found that he was also jumping through unidentified frames. We cannot therefore recommend Typo3.
Plone and Drupal Advanced Features comparison
With both Drupal and Plone all of these feature are available either out of the box or as a module plug-in.
- Implementation & Setup
- Custom Content Types / Templates
- Pod casting
- RSS syndication
- XML syndication
- Content Reuse
- Friendly URLs
- Support for Multiple file formats
- Publication Scheduling
- Content Versioning
- Granular Privileges
- LDAP Authentication
- XHTML web output
However, XML syndication and Publication Scheduling may require substantially more development time in Drupal. Also Plone is written in Python, which may be an issue for those without skills in that language.
Modifying the Interface
The interface for each system is XHTML / CSS based so each can be enhanced and improved. Quick and Easy, Joomla, Plone or Drupal would satisfy the website requirements of many community based authors and groups. However, the Drupal and Quick and Easy interface, without modification, are more screen reader friendly than Plone. Drupal and Quick and Easy have a clean and simple user interface that will be a help to (non-technical) content creators.
Plone has more features available out of the box but many of these features may not be needed.
Accessible WYSIWYG Editors
CFIT have concerns about the general accessibility of WYSIWYG editors. The two editors we looked at are XStandard and TINY MCE. While using XStandard we encountered some problems. The screen reader could not properly interact with it and it seemed to leave the user hanging without any real idea where he was within it.
It also needs to be keyboard accessible. We contacted the XStandard development team and they have produced a Mac version that is keyboard accessible, which we have not yet tested. They also mentioned that they are currently working on a keyboard accessible Windows version of the editor and hope to have it out by the end of 2006.
We found Tiny MCE to be somewhat accessible but we have concerns about the quality of output; structure is often not maintained.
Practical Alternatives to using the embedded WYSIWYG Editors
Use an editor with an accessible interface such as the freeware editor PSPPad. The screen reader user can them upload the structured content via the CMS. The drawback is the learning curve required on the part of the user but it seems to be the most viable option at the moment.
For more on comparisons of editors Peter Krantz of Standards Schmandards has done some more extensive testing of a wide variety of CMSs.
When embedded WYSIWYG editors improve the accessibility of their interfaces this will have a major positive impact on the overall accessibility and enhanced usability of a CMS.
Hi - what about the one interface overtly and actively pursuing accessibility??:
Posted by Mike Rawling on
As mentioned in the article during our tests we found Xstandard to be better than many of the other editors available but we did encounter some issues. While I really think Xstandard is a good editor and can only get better, at the time of writing our user tester found it difficult to use, and he is very much a power user.
Posted by Josh on
Regarding XStandard, you note:
Does that mean it has been released, delayed, or was that a typo and meant for end of 2007?
Posted by Anup on
You mentioned peculiar and inconsistent behaviour using JAWS 5 with Plone and that you would have to do some testing with JAWS 7 and Window Eyes. What did the further tests reveal?
Posted by Max on
a very timely post, as one of the things touched on during the SXSW WaSP meeting was a need for more guidance on standards-compliant and accessible CMS solutions.
Posted by patrick h. lauke on
Outstanding overview. Thanks for doing all this work. I may try some of these myself, like Quick and Easy, but better will be when I encounter a client who needs the back end to be accessible. I love WordPress -- surprised it wasn't mentioned -- and I use it a lot, but I'm not overly impressed with the admin.
Posted by Mike Cherim on
Hi Josh, just to clarify, we (Belus Technology) have never claimed that XStandard is screen reader accessible (i.e. it cannot yet be used to author content using a screen reader as an intermediary device). As far as we know, no WYSIWYG editor is making this claim yet (it's a huge job and one that we are working on).
What we do say is that XStandard produces accessible markup and the following article confirms our claims:
Also, we are fulfilling our promise regarding keyboard accessibility. In the OS X and Windows version of XStandard Version 2 (out in April-May 2007) XStandard's interface will be fully keyboard accessible. So you are welcome to test XStandard for keyboard accessibility at that time.
Hope that clarifies what we say we do and what we do not say we do.
Posted by Vlad Alexander on
I use Wordpress and have found it to be very accessible, and lovely to use, it can also be very powerful the only situation it struggles with is multiple user groups and privileges.
Posted by Anthony Brewitt on
It was 2006. For an overview of where Xstandard is at please go to their site. [http://www.xstandard.com/]
Max, we haven't at this point proceeded with anymore testing but will probably do another round later this year. Suffice to say we have noticed (in other tests) that a user will often be able to extract semantics from a form field by using the tab key, or navigating by searching for form fields using quick keys - that they may not get by using the arrow keys - regardless of what version.
This may not be *always* the case but it is something that we have observed. So the purport is - in order to ensure interoperability make sure that you always mark up and label your form fields correctly.
Thanks xhtml, Patrick and Mike for your comments.
Posted by Josh on
Thanks for your comment. We tested Xstandard as it was when embedded in a CMS, by my own reckoning, the best WYSIWYG editor available at the time - and if it happened to be accessible then that was a definite bonus.
Will do. Thanks, and good luck with your venture.
Posted by Josh on
ATutor might have been a good choice to review here. It is WCAG 1 and ATAG 2 (when it arrives) conformant. The ATutor team added the accessibility features currently part of TinyMCE. ATutor also has a builtin accessibility checker (AChecker), and has developed a plugin for TinyMCE that can link it into the AChecker web servies. ATutor conforms with XHTML 1. All around, the user, manager, and administrator interfaces are accessible (not just the user interface).
Posted by GG on
Initial testing for the new version of Joomla (version 1.5) under Window Eyes 6 and JAWS 8 has shown that whilst Window Eyes can negotiate some of the redesigned backend interface, JAWS has serious problems with an AJAX/mootools depedent backend administration portal of Joomla 1.5.
In this way, the older, now grandfathered edition of Joomla is more accessible than the new version.
Posted by Lawrence Meckan on
I'm using Textpattern. I agree that the application lacks a good navigation hierarchy to use at the front end. I've developed a plugin for textpattern that solves this problem. I'm currently testing it and I'm thinking of releasing it somewhere in the future.
Posted by Jan on
Great work, thank you.
Anyway, there are as many ideas as there are people. Each of us is unique and we may consider other options. As a very personal point of view "spaw" editor is the friendliest, compact and functional I have ever seen. Better than Tiny or FCK.
And anyone may ask themselves, "What is basically a CMS?" It is a website made so many times that become a kind of standard for more or less of us or a tool easy to use to make a new website faster than before? Is a CMS a shell or an Add On for a website? Some of programmers find actual CMS/Net Frames very annoying, some of us find it very useful, it depends where you look at and what is your programming level.
Again thank you for collecting and publishing all details above.
Posted by lari on
I recently found a new product called Nenest which I think exactly meet your goals:
1. A CMS which is functional, accessible and usable to authors who have basic IT skill level and knowledge of using word processors, but no fluency with markup languages like HTML.
>> No programming, no database designing at all.
2. A CMS that can be recommend and promoted as an accessible Web content management solution for community/disability groups, charities and others with a limited budget.
>> It's really cheap, from free to $99
3. If possible, the WYSIWYG editor should be accessible to screen reader users. The usability of the mechanism is also important, so we are not merely considering if it is technically accessible but also if it is easy and pleasant to use.
>> I love their WYSIWYG editor - I can directly insert a picture from my local computer; it accepts other special elements like chemical structure (As I know it's the only one provide this feature to chemists)
Posted by Tony on
Two CMSs particularly claiming accessibility support are Papoo http://www.papoo.de and Immediacy http://www.immediacy.co.uk. I didn't have the time to test them, but it would be interesting to validate their claims.
Also WordPress as a simple CMS for less complex websites might be an option. At least it's a widespread tool already familiar to many developers and administrators.
Posted by Martin Kliehm on
love the article, many thanks to you all for taking the time to compile it.
Just one point I would like you to clarify for me. The article stated that Quick and Easy was one of the best that had been tested but it didn't actually go as far as recommending it.
Was this an over sight or is Quick and Easy *not* recommended?
Posted by Michael_H on
Thanks for you comment. We definitely would recommend Quick and Easy.
Posted by Josh on
Mambo is now six years old and like many mature applications, it was not built with accessibility in mind. Accessibility has, however, become a priority for the project and we are in the process of refactoring Mambo to become an accessible CMS "out of the box".
Mambo 4.7, which is currently under development and should be released within a few months, will be accessible on the frontend, to provide a better visitor experience. The changes we have made will also allow sites to be delivered as validating, XHTML1.1 compliant sites should the site designer so wish.
The administrator backend will have some small usability improvements in 4.7 however we are not attempting to "add" accessibility to the backend. The Mambo team feel usability and accessibility cannot be treated as addons so we intend to completely rebuild the backend.
Mambo is the only open source CMS that has a member of the Guild of Accessible Web Designers on the development team.
We thank you for including Mambo in your tests and we hope that your results will make a nice reference point for us in future years, when we can look back and see how far we have come with providing an accessible CMS.
Posted by Lynne Pope on
Good article. I was wondering if similar research is available about the big commercial players (Vignette, IBM, etc.)
Posted by Michelangelo on
That would make for some interesting research. Maybe in an updated article we could look at the more commercial offerings and compare.
Posted by Josh on
Thanks for this article. Very pragmatic approach.
We use textpattern as a CMS for it's simplicity and the adaptability to our needs. I've tried to audit our efforts according to http://www.techdis.ac.uk/ advice, at http://isdevelopment.weblog.glam.ac.uk/
The only reservation I'd have is that we are still trying to get people in the organisation to realise that sites change and that making them accessible and usable is a process. As you point out, the expectation that an accessible CMS will take care of it all for you is unrealistic, but still unfortunately all too common.
Posted by Kev Mears on
I tested almost cms from that list and the best is Joomla. It is so easy to work with.
Programming is so easy now.No need advanced knowledges for Joomla.
I recommend it to everybody.
Posted by Iulian Ghisoiu on
Nice comparison. However one, not well known - but perfect, CMS is missing. You should give a try to Clever Leap CMS. Check out demo at www.cms.cleverleap.com. It is very easy to use, enables handling of structured content (which most of the CMSes doesn't). Give it a try and post feedback.
Posted by Stefan on
I am using both v2evolution and wordpress for my blogs. But both of them has serious drawbacks when it goes to CMS. I hope tireless people like you help solve them.
Posted by John on
I am afraid you lost your credibility at "Expression Engine is a blog focused CMS which lacks more advanced functionality."
Makes one wonder whether you actually did take a look at it? Not a deep look, at any rate. (Disclaimer: I develop websites with ExpressionEngine professionally.)
Posted by Ingmar Greil on
I'm a visually impaired user and I use the jaws screen reader.
Joomla is awful in respect of screen reader friendliness, as is mambo.
QNE is pretty user friendly, but unless you pretty tech minded, installing it is a pain in the neck, also, they only provide a download as a .tar, no zip available, very major drawback.
the most straightforward I've found as yet is called "CMS made Simple", sorry I don't have the URL.
obviously there's a learning curve as with anything, but it's certainly screen reader friendly and very extendible.
Posted by Darren H on
I love to see comparisons like this, and the more the better because just one only scratches the surface, and needs taken with a pinch of salt.
For example, the list of systems reviewed is hardly exhaustive. Also, there is a lot of 'we would not recommend' kind of dialog, which is largely subjective when it comes to the usability stuff.
Systems are always changing too, so it's only fair that follow-up reviews take place, which might show (or not) whether system developers are actually making strides to improve their systems, which subsequently would be a good (unsaid) indicator of their interests for accessibility to begin with.
I could fairly say that systems get misrepresented to a certain degree (as commenter #26 might attest to). I'm a Textpattern user myself and though it does attract many people for it's clean "blog" abilities, it is certainly capable of running a site of any size or function you can think of (with a wealth of plugins to aide in the effort).
However, I do realize this is largely a focus on accessibility, and in that respect Textpattern no doubt has some things to improve (and in fact they are doing it for future releases, including deeper site structure abilities) but the Textpattern review here was nevertheless a bit vague and seemingly dismissive (no doubt unintentionally) which makes me wonder if each system had equal shares of research time. Textpattern is a lightweight yet powerful CMS, for sure...skills not included.
Again, love the comparative study, but we need more of them, and over time.
Posted by Destry Wion on
CPS www.cps-project.org, in its latest 3.4.4 version, is accessible both in its frontend and backend. CPS is already WAI-A compliant out of the box, and web sites and portals can be made WAI-AA compliant.
We hope CPS can be included in next accessible content management system tests.
Accessibility is a very important constraint for CPS developments since CPS is used in many state agencies and educational portals where accessibility is mandatory.
CPS is very well known in France and Spain and used worldwide.
Posted by CPS on