Terug naar overzicht Onderzoek naar toegankelijkheid

Return on Investment of Webrichtlijnen

In commission of ECP-EPN, Platform for the Information Society, the University of Twente conducted a study on the potential costs and benefits of implementation of the Webrichtlijnen for non-governmental organizations, like businesses, corporations, not-for-profit and charity organizations, and other private parties. The Webrichtlijnen are compulsory for Dutch governmental organisations. Although the impression in the market exists that implementing guidelines have a positive effect on different indicators of cost and benefit, it is not possible to make any financial or economic conclusions, due to the lack of hard evidence of effects of implementation of guidelines.

Background

Return on Investment for WebrichtlijnenIn 2004, the Dutch government published the Webrichtlijnen, an extensive set of guidelines for the development of high quality, maintainable and usable websites. In 2011, the updated version of the Webrichtlijnen, which includes the 2.0 version of the Web Content Accessibility Guidelines, became part of the ‘comply or explain’ list of Dutch Open Standards.

In commission of ECP-EPN, Platform for the Information Society, the University of Twente conducted a study on the potential costs and benefits of implementation of the Webrichtlijnen for non-governmental organizations, like businesses, corporations, not-for-profit and charity organizations, and other private parties.

Research question

The research question in this project was: What is the return on investment of (the implementation of) the Webrichtlijnen. To assess the return of the investments (ROI) what organizations do for the implementation of the Webrichtlijnen, we must have a view of costs and benefits. At one hand, we need to know how much human, financial and other resources (the ‘costs’ side) have been invested in creating, maintaining and adapting those parts of the organizational sites that are affected by the Webrichtlijnen implementation. At the other hand, we need to determine the yield or returns of the website (the ‘benefits’ side) to assess the ROI. Returns can be measured in economic/financial indicators (e.g. reduced development costs or increased number of purchases), but also in other types of indicators (e.g. positive publicity, reduced number of complaints, switch of customers from telephone to the less expensive web channel). In addition to the financial indicators, we also studied social, technical and legal indicators.

Research design

To address the question of costs and benefits of implementing the Webrichtlijnen, a study with two parallel parts was conducted. 

For the first part of the study, we selected a set of business cases. Four very different organizations were followed in detail in the process of creating or adapting sites to (better) meet the Webrichtlijnen standards. It concerned the software corporation Microsoft (with two sites), the Arnhem-Nijmegen University of Applied Sciences, the KWF Cancer Society and the web store Man and Shaving, in cooperation with its web shop developer Directshop. Their sites were analyzed and reviewed at four moments for correct implementation of the Webrichtlijnen (before, 2x during, after re-design). In addition, the organizations received support from two external teams of experts, tailored to the special implementation issues or questions they were confronted with. Representatives of the four organizations were interviewed at regular intervals about costs, benefits and experienced problems with the implementation of the Webrichtlijnen.

The first part of the study is reported in: Business case study Costs and benefits of implementation of Webrichtlijnen, E. Velleman and Th. van der Geest (2011).

In the second part of the study, two groups of reference organizations participated. One reference group of 30 organizations was aware of possible accessibility problems, as appeared from their contacts with the Accessibility Foundation. They were interviewed at the start of the project about their views of the Webrichtlijnen and the associated costs and benefits. Their sites were checked for implementation of the Webrichtlijnen at two moments, including additional measures like level of accessibility, page loading speed, and mobile device suitability. Additional interviews were conducted after six months and one year to explore their insight and perceptions of costs and benefits.

A second reference group, consisting of 50 random organizations, filled out the questionnaire at the start of the project, reporting their views of the Webrichtlijnen and the associated costs and benefits.

The second part of the study is reported in: Cost-benefit analysis of implementing web standards in private organizations, Th. van der Geest, E. Velleman and M. Houtepen (2011).

The results

Evidence and beliefs

One can only perform a cost-benefits analysis when organizations are aware of the actual indicators like financial investments, savings and returns that they gain from the efforts for their websites. It proved to be problematic to collect data about investments and returns, for two reasons. One important reason is that investments in the website are mostly personnel costs. The personnel costs of building and developing a website can be determined by most organizations, especially if they have used subcontractors for this work. But the personnel costs of content creation and content management is usually divided over many different departments and people. The Webrichtlijnen are about both building/development and content creation/management. That makes it hard for many organizations to quantify their website costs in general, and the Webrichtlijnen-related costs specifically. Larger organizations have a better overview of costs than smaller organizations.

A second important reason is that many organizations have a view on what they want to accomplish with their site, but do not collect the data that could demonstrate or prove that they indeed reap the intended benefits. If the goal of the website is primarily informational, it is hard to demonstrate a ‘return’ anyway. Larger organizations and web stores collect, more often than other organizations, information that helps them to assess whether their websites achieves the intended goals and benefits. In practice, the data that the other organizations collect are at best about visitor numbers and search engine effectiveness. Only a few participating organizations collect data specifically related to adherence to Webrichtlijnen, or to viable criteria for usability or accessibility. That makes it difficult to prove that benefits, like increasing visitor numbers, are specifically caused by the implementation of the Webrichtlijnen.

It is surprising that the participating organizations collect or use so little solid information about their website’s returns en effectiveness that can be related to the actual design and content of their website. Each of the business case organizations has the impression that the costs of their website are reduced thanks to the (further) implementation of the Webrichtlijnen. It has become easier to change or add content and other components to the website. So the perception of (working on) implementation of  the Webrichtlijnen is positive. Those organizations that are implementing Webrichtlijnen feel that it makes their website better usable, more accessible, and better in general. However due to the lack of solid data about website effectiveness and returns, they cannot support this claim. The lack of data about realized benefits makes it very hard to convince organizations of the relevance of Webrichtlijnen compliance on the basis of financial and economic arguments.

Convincing arguments

Our study revealed a number of topics and themes that can be used to convince organizations that they should start applying the Webrichtlijnen.

Both in the business cases and in the reference groups, there was a strong interest in the effect of applying Webrichtlijnen on search engine findability and optimization (SEO). The business case organizations use the SEO argument internally to get support for the implementation of the Webrichtlijnen. Solid evidence about the effect of Webrichtlijnen on SEO would draw other organizations over the line.

The rapid increase of mobile internet access also can be used as an argument for implementing the Webrichtlijnen. Sites that follow the guidelines perform better on various platforms, including mobile platforms. Organizations can be convinced of the value of applying the Webrichtlijnen when they see that it makes their sites more future-proof. Our study provides this evidence.

Two business cases were working on implementing the Webrichtlijnen from the perspective of ‘good governance’. The management of one business case, Microsoft, has included accessibility in its mission. For another business case, the HAN University of Applied Sciences, accessibility has become a legally required certification criterion in their educational assessment. The argument of social responsibility could be used as a convincing argument for many other organizations.

The Webrichtlijnen could also become a useful instrument in tender procedures. Organizations can simply demand that a new site is meeting the Webrichtlijnen. This requirement can be checked by external organizations, thus forcing the market to deliver work of high quality, even when organizations do not have the required expertise internally for checking the quality of results.

The business case organizations realize that it is more expensive to make accessibility improvements to existing websites (retrofitting) than to take the Webrichtlijnen into account from the start. Attempts to convince other organizations and businesses to implement Webrichtlijnen should hence be directed primarily at the next release or at new websites.

Efforts could also be directed at content management system (CMS) vendors. They should be convinced to make Webrichtlijnen implementation one of their selling points and to bring products to the market that make it very easy for their clients to create and maintain Webrichtlijnen-proof websites.

Awareness and expertise

Awareness of what the Webrichtlijnen encompass and what the effect of applying them could be is low in most organizations. This study resulted in a list of indicators that can help to raise the awareness of costs and benefits of compliance to Webrichtlijnen.

Even organizations that are aware of the importance of Webrichtlijnen or accessibility guidelines indicate that they do not have much expertise in this field. They delegate the issue to their external web builders, but have no means or expertise to check whether the external vendors have built sites according to the guidelines. It is important that current and future web builders and editors receive more and better training and education, so they can share their expertise in the organizations.

The four organizations in our business case study believe that they benefit of applying the Webrichtlijnen. Just a few organizations collect data that proves that implementation of the Webrichtlijnen results in better sites, webpages loading faster, better findability, better performance on mobile phones, more visitors and a more positive public reputation. These organizations could play a role as an ‘ambassador’ for the Webrichtlijnen in contacts with other organizations.

To conclude

Our business case organizations in general have the perception that applying the Webrichtlijnen has a positive effect on different cost and benefit indicators. However, we conclude that at the moment it is not possible to come up with solid evidence for cost benefits of applying the Webrichtlijnen to a website, due to the lack of data about the effects of implementation of the standards, Efforts need to be focused on creating awareness, distributing information and building expertise of costs and benefits of websites, and of the effects of implementing the Webrichtlijnen on costs and benefits.

In the discussion about return on investment and the costs and benefits a strong emphasis is placed on financial arguments, but there are more types of benefits conceivable. In this project a list of different types of cost-benefit indicators was created together with stakeholders, which can be used for further policy development and research on the implementation of Webrichtlijnen.

Download

Download Cost Benefit Study of Implementation of Dutch Webguidelines (pdf-file, 1,920kb).

Gerelateerd

Tags:
Categorie:
Projecten en publicaties
  • Delen op Facebook
  • Delen op Twitter
  • Delen op LinkedIn