Demanding special user testing – Case RuutiBudjetti

1 Introduction

This document reports the results of user tests carried out on the City of Helsinki’s RuutiBudjetti service for young people. The purpose of the testing process was to achieve an understanding of how accessible the voting phase of the service is to young people with special challenges related to the use of digital services. Based on the aforementioned, the tests were carried out exclusively with so-called special users who had several limitations related to cognition and sensory functions. All the testers were young people in the vocational education age.

The test results were used to create a number of recommendations for improving the usability of the voting section of the service. The results and individual recommendations derived from them are described in greater detail in chapters 3 and 4 of this report. The testing arrangements, the profile of the test users and the test cases are described in the following chapter.

2 Research arrangement

2.1 Background

RuutiBudjetti is part of the City of Helsinki’s online services related to participatory budgeting. The services provide Helsinki residents with an opportunity to influence the development of their own residential area through projects proposed by residents themselves. RuutiBudjetti is a participatory budgeting model for 6th–9th graders implemented in cooperation with schools and youth centres. It works as a website on which young people can propose and vote on projects to be implemented in the area of their school.

All of Helsinki’s participatory budgeting services follow “voting rounds”. When a new round begins, users can propose projects to be included in the voting. However, projects are eliminated and developed before the voting in a collaboration between users and experts. The results of the actual voting phase (and the materials from previous phases) are left on display on the website.

One special characteristic of RuutiBudjetti is that it enables young people to propose and vote on projects in the area of their own educational institution; the pupil’s home area or municipality makes no difference. The City takes the voting result into consideration at the end of the round when it assesses the implementation of proposals in each voting area.

2.2 Objectives

The objective of the user testing process was to establish how accessible the voting phase of the RuutiBudjetti service is to users to whom using online services is clearly more difficult than normal.

In this report, we refer to such users as special users in short.

The term ‘special user’ refers to quite an extensive group of different people. Some have challenges caused by sensory disabilities or motor skill limitations, making online services difficult to use for them. For example, a visually impaired person may have to rely on a screen reader program alone or utilise strong magnification and a high-contrast display mode.

A person’s cognitive abilities also often hinder their ability to use online services. Aspects such as language skills, literacy and limitations related to memory and understanding may be significant hindrances in utilising information technology and digital services or learning to use them. Moreover, if a person has both sensory and cognitive impairments, the array of challenges widens further. In such cases, the user can be referred to as a multi-impaired user. In the presently reported user testing process, all participants were classified as multi-impaired special users.

The objective of the user testing process was thus to establish how well the RuutiBudjetti service is suited to multi-impaired young users. In particular, the objective was to find out how well the test users would be able to vote in the service, which is the most important function of RuutiBudjetti to the majority of users.

The service and the testers were selected based on the fact that RuutiBudjetti is aimed at an exceptionally broad and diverse group of users who are also young and untrained in civic participation. As such, users should be able to use the service without any special background knowledge on participatory budgeting. Furthermore, users should be able to cast their votes in the service at least passably even when using the service with an aid device or when unwilling or unable to concentrate on the instructional texts on the pages and the contents of the user interface. In other words, the service must be exceptionally easy to use.

2.3 Test users

The test users were selected by a cooperation partner of the City of Helsinki to whom working with multi-impaired special users is part of everyday work. As regards sensory impairments, the assignment stated that at least visually impaired users who operate online with an aid device were preferred. Because it was known that the testers would be multi-impaired in any case, no other requirements were set.

All of the testers were young, and their age distribution corresponded with that of upper secondary education students who have completed comprehensive school.

The table below presents the users’ sensory impairments and the test environment. The users’ cognitive impairments are not specified in detail, as no official information regarding them was available. However, the effects of the impairments are taken into consideration in the test observations.

#	Sensory impairment	Test environment
1	The user is blind.	Personal laptop computer with the latest version of the Jaws screen reader.
2	No sensory impairments.	Personal smartphone.
3	The user is deaf.	Personal smartphone. Assisted by two interpreters through whom the user communicated.
4	A visually impaired user who is able to operate by vision.	Personal laptop computer with the ZoomText magnification program.

Table 1: Test users

2.4 Testing method

The testing method selected was user testing in which a person invited to the test situation would perform a series of tasks that were planned in advance and carried out similarly by all users (see chapter 2.5). The tests were carried out in October 2021 in the premises of the City of Helsinki’s cooperation partner in the Helsinki metropolitan area.

Only one tester was present in the testing facility at a time, and the participants did not see or hear the others perform their tests. The time reserved for each test was 60 minutes, but the testing process could be ended earlier if all the tests and the tester interview were completed.

The users’ own devices served as the testing environment without exception (see Table 1 above).

Before starting the test, the testers were provided with general information on what the service was about and how the test situation would proceed. During the performance, if the user was unable to complete their task or had irreparably veered off the correct usage path, they were guided by Siteimprove experts; other than that, the user was to attempt to act independently.

After the performance, the tester was interviewed briefly regarding the testing process and the tester’s own observations during the test. The actual test analysis was carried out separately among Siteimprove experts after the tests.

2.5 Test cases

There were five test cases. An extra test case was also created to be used where necessary with users who performed in the actual voting process with few problems.

1. Entering the voting process

Objective: The user has entered the first view of the voting process in the service, in which they log in as a voter with the City of Helsinki’s test account.
Initial situation: The web browser is open and the address of the RuutiBudjetti service is already written in the address field. The user accepts the address and begins operating on the page independently once the page has loaded.
During the performance: The tester is asked to describe how they believe the voting process works in the service – in other words, whether they understand what they are supposed to do in the service.

2. Selecting the tester’s own voting area

Objective: The user has selected “Maunula” as their own voting area.
Initial situation: The user has logged in. The test user is a student at Helsinki Vocational College and Adult Institute.

3. Familiarisation with the information of the proposal (cooking club)

Objective: The user has found and opened the information of the correct proposal (cooking club) and found out what the location of the proposal is. The user has returned from the proposal information to the browsing view.
Initial situation: The browsing view for the voting process is open.

4. Adding proposals to the voting cart (communality + Kallahti Cottage)

Objective: The user has added proposals that they would like to vote for to their voting cart. The user has also listed the proposals by budget from smallest to largest. The user has moved to the preview screen.
Initial situation: The browsing view for the voting process is open.
During the performance: The user is first asked to change the listing of the proposals by budget, so that the cheapest proposals are listed first.

5. Submitting votes

Objective: The user has submitted their votes and logged out of the service.
Initial situation: The user has selected which proposals they want to vote for.

6. Extra test case: The user finds out how proposals are selected for implementation and when proposals are implemented.

Objective: The user continues using the service after logging out and becomes informed of how projects are selected for implementation and when projects are implemented.

3 Results

3.1 Summary

RuutiBudjetti is technically accessible and compatible with aid device programs. However, some individual shortcomings related to the use of aid devices were observed in the user interface (and have been previously reported in connection with an OmaStadi user test report), but they were so minor that they did not limit the testers’ ability to use the service with a screen reader program. The technical implementation is already so complete that the success of the test cases depended on the testers’ ability to use their respective aid devices.

In light of the aforementioned, the most important observation made in the testing process was that instead of technical shortcomings, the RuutiBudjetti voting service has usability-related issues that make it too difficult to use for users with significant cognitive impairments.

The reason for this is that the user must simultaneously understand several concepts that are foreign and generally difficult for them and be able to perceive correlations between these concepts in the voting process user interface. Without this perception capability, the user is only able to navigate the service at random, whereby the voting result does not necessarily correspond with the user’s true wishes.

Examples of difficult concepts include proposals, browsing vs. selecting proposals, picking a proposal for voting before the actual voting process, and what the purpose of each work phase in the voting process is.

Some of the accessibility issues in the service were caused by the testers’ own limitations, which are significant in the context of RuutiBudjetti as the task involved (voting) is more complicated than average. There is nothing that the service provider can do about these limitations.

However, shortcomings were observed in the user interface of and the instructions provided in the RuutiBudjetti service, the remedying of which would improve the usability of the website for everyone and especially the young target demographic of RuutiBudjetti, who may exhibit limitations observed among special users. Examples of such limitations include unwillingness or inability to read instructions and impatience in the mental structuring of the usage paths. Below is an account of the key problem areas.

An aspect that the shortcomings listed share is that they are focused on the voting phase view in which the user selects their proposals for voting. Even though the testers had difficulties before this view and partly even after it, these challenges were more related to personal limitations than adequately generalisable from the user interface, so they are not reported separately in this context.

3.2 Special users’ perspective on RuutiBudjetti

When interpreting the observations, it is important to understand how special users, particularly the multi-impaired users involved in the test, aim to operate in the service.

Special users perceive a smaller portion of an individual page view than so-called basic users, because to them, even basic use of the website is slow and arduous. They aim to browse the website “economically” and in an optimised manner, thus examining the view no more than what they deem necessary. The significance of the page structure (headings, technical areas, browsing order) is emphasised in the browsing methods, as linguistic limitations may render users unwilling or unable to read instructions.

What follows is that the vocabulary used in the controls (buttons, links, search controls) and instructions of the user interface should be as simple as possible and always consistent in the different sections of the service. In other words, synonyms may prevent users from associating functions with one another even when dealing with the same concept; similarly difficult are concepts formed from compound words that do not mean anything concrete in the common language. Special users are not necessarily able to deconstruct the concept into a function that they understand.

3.3 Names of the controls

3.3.1 Opening the information of proposals

The information page (dialogue) of a proposal is opened via a link that is part of the proposal card. Several testers had difficulties with understanding that they could receive additional information about the proposal via the link. They did observe the heading link and bottom part link (‘Show proposal’), but one user felt that because they were already examining the proposal, the link was of no use to them, as their objective was to find out the location of the proposal, not view “more” of it. Consequently, the user was left looking for the location on the card.

3.3.2 Closing the proposal dialogue

The “X” button that closes the dialogue is logically named “Close” for assistive technology. However, because users did not understand the role of the dialogue as a user interface structure, closing (as a word or visual symbol) was not the function that they were looking for. Rather, they attempted to return to the previous view.

3.3.3 Navigating the voting process

The buttons located at the bottom of the page view, with which the user can move to the next or previous work phase, were named with two words or phrases, e.g. “Go to” followed by the task performed in the work phase.

At first, all users except one had difficulties with understanding what the buttons do. For example, user #1 hesitated to press the button, while users #2 and #4 were almost completely unable to find the option to move forward.

The reason was that the users were unable to associate the name of the next phase with the objective, which was to “move forward”. They no longer remembered what the work phases of the process were and thus did not know that moving forward meant moving to a function with a certain name. Additionally, the name of the function as a separate abstract concept was also unclear to them, so when their memory ran out (or their understanding of the phases was lacking to begin with), they were unable to understand the purpose of the link as well.

3.3.4 Naming of the search functions of the browsing view in the mobile view

By default, the search field and the other search controls are hidden in the mobile view. The user must reveal them by using the ‘Filter’ button. As with the previously mentioned case, the test users were unable to understand how the filtering was connected to searching for proposals. That connection may not be obvious to a basic user, either, as filtering can also refer to the processing of previously displayed results. On the other hand, the filtering function technically filters results, all of which are displayed by default. As such, the name is logical for the platform, but not necessarily what the user has in mind.

3.3.5 Adding for voting

The element from which a proposal is selected for voting was named ‘Select proposal’. The name was too indirect for the testers: they were unable to associate the contextless “selection” with selecting a proposal for voting, even though they were in a work phase in which they were urged to select a proposal to vote for.

Similarly to the aforementioned observations, this case shows a need for more specific and concrete information than what a basic user would need. Special users are unable to place a function in the “larger context” in a similar manner.

3.4 Visual style and layout

3.4.1 Difficulties with perceiving the start and end points of proposal cards

It was surprisingly difficult for users operating by vision to perceive how an individual proposal card is distinguishable from the others in the browsing view. The users were partly confused about what information was part of the previous card and where the next one began.

The reason was that the cards were visually relatively tall, containing plenty of “empty” white space used for separating the different areas of the card. However, for a user with good vision and normal cognitive skills, the cards form a somewhat clear grid or stack. On the other hand, for the test users the individual information items of the card are difficult to understand to begin with, making it more difficult for the users to perceive one card whole as well. The large amount of empty space in a white card appeared to exacerbate this problem.

The cards are surrounded by a frame, but the frame is very dim. Additionally, the frame is also used inside the card to delimit items such as date, favourite, and comment information. This style made it even more unclear to the test users where the card ended.

3.4.2 Changing the listing order of proposals

Test users operating a mobile device by vision found the menu for arranging the results difficult to find. Tester #3 even needed help with locating the menu despite having good vision.

The reason appeared to be that it was difficult for the users to cognitively understand and thus anticipate which function they were dealing with, at least before finding it. This made mentally structuring the view by vision more arduous than for basic users who are able to automatically form a mental image of what the menu could look like or where it could be located. As such, the special users’ visual pattern recognition functions did not grasp the menu the same way, even if their eyes detected it. Seeing also involved a need to understand the function of the menu and the text therein, not just “see” it.

3.5 Instructions

A number of observations were made on the instructional texts of the service already in connection with preparing for the user testing process.

3.5.1 The voting process

See chapter 4.3.

3.5.2 Information about the service

In the opinion of our working group, the style of the instructions provided is not clear enough for the target demographic of the service, i.e. comprehensive school pupils. The texts appeared to be written more from the perspective of the City itself than pupils. Example:

Information gathering: “RuutiBudjetti starts with information gathering every spring. All youth services units implement the information gathering in a way that is suitable for the area’s young people. Information is often gathered via electronic surveys or at events, for example. The objective is to chart young people’s wishes, concerns and ideas. The pupils usually contribute to the information gathering process during their school day.”

The text first mentions that the RuutiBudjetti process starts with information gathering. Next, the text explains who does the information gathering, but not what it means. Then, the text mentions the work methods used. It is not until the second to last sentence that the text mentions what the purpose of the information gathering is and what it involves. This causes the reader to have to return to the beginning of the text and read it again in order to truly understand what was said there, as only now does the reader have a context in which they can cognitively place the information of the sentences.

The description of the ‘Negotiations’ phase in the ‘How will RuutiBudjetti progress?’ instructions is even more poorly structured. It is so bureaucratic and confusing that our working group was unable to understand it on our first reading. However, the instructions are intended for ages 12–16.

4 Recommendations

4.1 The names of key controls should be clarified

4.1.1 Opening the information of proposals

The ‘Näytä ehdotus’ (‘Show proposal’) text of the link on the proposal card could be complemented with a word or phrase to communicate that the link can be used to bring up more information about the proposal or its full description. Example: ‘Näytä koko ehdotus’ (‘Show full proposal’). This would still not be the easiest solution for special users, but it could be implemented without changing the user interface. On the other hand, what if a verb corresponding with the objective of showing the additional information was placed at the start? ‘Tutki ehdotusta’ (‘Examine proposal’) could thus also be a better alternative to the current solution.

4.1.2 Closing the proposal dialogue

The “X” button for closing the dialogue could be replaced with a text button saying “Return” (possibly with a left-pointing barb/arrow in front of the text). On the other hand, the bottom part of the dialogue already features a text button, but finding the button was not straightforward for users because the dialogue is somewhat long.

4.1.3 Navigating the voting process

The buttons in the bottom part of the side view should be named “Next” and “Previous” (or synonyms thereof). This remedying solution should not affect so-called basic users who are able to find the work phases of the process (and necessary links) in the top section of the page and may even remember them anyway. Furthermore, users are used to similar naming conventions when moving from one page to another in forms and online services. In turn, it is important to special users that they are able to clearly discern where the task at hand can be continued. Knowing what the next phase is is not relevant to them in advance. They perceive and mentally structure each view one at a time. The process ends when it ends.

4.1.4 Naming of the search and filtering functions of the browser view in the mobile view

The button that shows filters could be named “Find and filter proposals” instead of the current “Filter”. Another alternative is for the search field to be always displayed in the mobile view while the other filters are located behind a button (whereby the button should be named slightly differently).

The grounds for this (correct or not) are that searching with a search word may be the most popular filter among young people if it is typical for a young person to want to vote for a proposal they heard about from a friend or see if there are any projects related to, say, skateboarding or cooking.

The search log data of the service may reveal whether the aforementioned change is justifiable. If that is the case, the change would cause at least one search function to be actively provided in the mobile view as well without the user having to be able to open the filter dialogue.

4.1.5 Adding for voting

The element from which a proposal is selected for voting should indicate more clearly what the result or purpose of the selection is. Making a selection is by itself not a clear enough activity here.

“Vote for proposal” could otherwise be good, but the problem is that the user is not voting yet at this point, so there is a risk of misunderstanding. “Select for voting” would perhaps sound less binding.

4.2 Visual structure

4.2.1 The area of the proposal cards should be more visually distinguishable

The outer frame of the proposal card could be made darker to increase the contrast, and/or a background colour could be created for the card. The sections within the card (favourites, date, comments, etc.) should be framed in a style that is clearly distinguishable from the outer frame of the card – in other words, it must be indicated clearly that the card does not end here.

4.2.2 Changing the listing order of proposals in the mobile view

Thought could be given to whether changing the menu text size or other stylistic means (colouring of the area?) could be utilised to more clearly separate it from the cards that begin immediately below. As it stands, both the number of results and the arrangement criterion menu appear to be mixed with the card listing to a degree.

4.3 Instructions

4.3.1 Selecting the voting area

Current text:

From the list below, select the area in which you want to vote. The areas are delimited in accordance with youth work units. If you do not know what belongs to the areas, you can check it here.

When you log in with your school Wilma account, the service suggests the area of your school. You can change your area if you want to.

Proposal for a new text:

In the RuutiBudjetti service, Helsinki is divided into areas. You can vote for the proposals of one area. See what belongs to each area (=link that shows additional information about the areas in the dialogue).

If you have logged in with your Wilma account, the area of your school has already been selected for you. You can change the area if you want to.

4.3.2 Contents of the proposal browsing page

Current text:

Take your time to read through the proposals you can vote for and think about how you would distribute your votes. You have 3 votes.

You can vote for up to three proposals. You can give one vote to each proposal. If you wish, you can leave your votes unused if you do not want to vote for any of the proposals.

Proposal for a new text:

On this page, you can examine proposals and select proposals to vote for. You can select up to three proposals. You can also select fewer proposals if you do not want to vote for three. Your votes will not be confirmed until the next page.

4.3.3 Contents of the preview page

Current text:

Below, you can check the proposals that you are about to vote for. At this point, you can still change your selection if you notice an error. If your selection is good as it is, press the ‘Vote’ button.

Proposal for a new text:

On this page, you can check what proposals you are about to vote for. If you want to change your selection at this point, return to the previous page. If you are happy with your selection, click on ‘Vote’ near the bottom of the page.

(Note to the reader: ‘Vote’ is not a button but a link, so it does not appear to aid devices as a button. For this reason, we propose that the word ‘button’ not be used.)

4.3.4 Contents displayed after successful voting

Current text:

Your vote has been saved in the system and you can no longer change it.

The RuutiBudjetti voting will end on 30 October 2021. Once the voting has ended, the result will be verified by the advisory boards. You can receive more information about this from your local youth centre.

Proposal for a new text:

Your vote has been saved. You can no longer change the votes you gave. The voting results will be published on the RuutiBudjetti website once the voting phase has ended.

You do not have to do anything else regarding the voting. If you want to, you can now log out of RuutiBudjetti.

Question:

MPASSid text version: If the user immediately follows the link proposed, will they remain logged in to RuutiBudjetti without knowing it (through cookies)? If this is the case, the user should be notified of it; for example, the user can be urged to log out first. Or if the user stays logged in between services due to cookies, does logging out of OmaStadi cause them to log out of RuutiBudjetti as well, even if they do not return to RuutiBudjetti?

4.4 Further development of the service

4.4.1 The target user base should be taken into consideration better

The current version of RuutiBudjetti is implemented on the City’s shared participatory budgeting (OSBU) platform. As such, the service has inherited not only the technical implementation of OSBU, but a very similar service design and contents as well.

The opinion of our working group is that this is not an optimal solution in a service intended for comprehensive school pupils. The majority of the language of the service (previously referred to in chapter 3.5) follows an organisation-oriented style in which terms and concepts are used to explain how things are and what is happening. The young person must understand the budgeting rounds, the proposals as concepts and how the City’s internal work relates to them, etc.

This may be deliberate if the service also has an educational role. On the other hand, if the objective is to encourage and activate young people to participate and vote, the current service does not appear to be the most effective approach. Several of the challenges encountered by special users that are also described in this report stem from the complex structure of the service, which is reflected on aspects such as the names of the controls and the visualisations, which conform to the underlying technical architecture (e.g. how the filtering of search results is named).

We are wondering whether a future development objective for the service could be to create a lighter and more gamified user experience. Does a person who only wants to vote have to know about the work phases and the areas?

Another solution could be that the user is simply first asked what their areas of interest are (the user selects a certain number of subjects from a list), whereby the user interface offers proposals one at a time, e.g. as illustrated cards to which the user always gives a thumbs up or thumbs down or selects “I want to vote for this” / “I do not want to vote for this”, etc.

A user experience like this that is straightforward and bypasses concepts would also be very usable and accessible. But whether it would continue to correspond with the City’s objectives for RuutiBudjetti, we cannot assess within the context of this report.

5 Summary

Siteimprove Oy carried out user testing on the City of Helsinki’s RuutiBudjetti online service in order to establish how accessible and easy-to-use the voting phase of the service is for users to whom using digital services is particularly challenging. The test users selected were young people with several concurrent sensory or cognitive impairments. The subject of the testing process was the open version of the service published online.

The user testing indicated that the service is already technically very accessible. The users of aid devices were primarily hindered by their own skills with using assistive technology, not the technical solutions of the service.

As such, the most important observations made had to do with the usability of the service. For multi-impaired special users, the voting phase of RuutiBudjetti was too difficult to carry out without external help. Even when assisted, the users had difficulties with understanding what individual tasks or broader usage paths they were carrying out in the service.

This was caused in part by the fact that the users’ own functional impairments were so significant that a complicated service such as RuutiBudjetti was unavoidably challenging for them. In RuutiBudjetti, the user must be able to understand several complex and interdependent concepts in order to be able to examine and select proposals created in the service and proceed in the voting process fluently.

On the other hand, the testing also revealed some shortcomings in the user interface to which special users were quick to react but which can affect other users as well, especially the young target demographic of the service. After all, young people do not always have the patience and willingness to concentrate that a website like RuutiBudjetti requires.

This report proposes remedies for the usability issues observed and provides thoughts on how the service could be developed in the future. For example, the objective could be to make the voting phase more gamified and thus simpler and more accessible as well.

Authors

Tero Pesonen

Members of the user testing working group:

Kimmo Sääskilahti

Niko Kunelius

Krister Linnas

Siteimprove Oy

Telephone: +358 40 50 92 034

Unioninkatu 30
00100 Helsinki