Contents lists available at ScienceDirect
Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo
User-centred web design, usability and user satisfaction: The case of online
banking websites in Iran
Iman Dianata,*, Pari Adelia, Mohammad Asgari Jafarabadib, Mohammad Ali Karimic
a
b
c
Department of Occupational Health and Ergonomics, Tabriz University of Medical Sciences, Tabriz, Iran
Tabriz Health Service Management Research Center, Tabriz University of Medical Sciences, Tabriz, Iran
Department of Computer Engineering, Bushehr Branch, Islamic Azad University, Bushehr, Iran
A R T I C LE I N FO
A B S T R A C T
Keywords:
E-banking
E-satisfaction
Internet banking
Web site
The relationship of Web design attributes (personalisation, structure, navigation, layout, search and performance) and users' personal characteristics to website usability and user satisfaction was investigated among 798
online banking users in Iran. The design and usability of the evaluated websites were not satisfactory from the
users’ perspectives. Multivariate regression models indicated that Web layout and performance were the main
predictors of website usability, while personal characteristics including gender, age and Web usage experience of
users had no effect. User satisfaction was also influenced only by the Web design attributes (particularly Web
structure) and not by the personal characteristics of the users. There was also a significant relationship between
website usability and user satisfaction. The findings suggest that the website designers should focus more on the
Web design attributes (particularly Web layout and structure), regardless of the personal characteristics of their
users, to improve the usability and user satisfaction of websites.
1. Introduction
Websites are the most important communication tool and primary
interface for users who are searching for online information and products (Kim and Stoel, 2004). A website can be defined as a set of
connected interfaces and functional attributes designed to deliver high
levels of performance and usability to users (Lee and Koubek, 2010).
Organisations and companies generally use websites to achieve marketing and business objectives. In designing websites, highest priority
should be given to the users, as the main source of profit for the organisations/companies (Lee and Koubek, 2010). User preference is,
therefore, an important factor that should be taken into account in
relation to website design. The user preference, usually measured
through an interview or a questionnaire, is generally expressed in terms
of design, usability, performance, aesthetics, satisfaction, online shopping, information quality, brand, etc. (Abelse et al., 1998; Helander and
Khalid, 2000; Cyr and Bonanni, 2005; Zviran et al., 2006; Lee and
Koubek, 2010; Tandon et al., 2016). However, usability and user satisfaction in terms of system use and acceptance have been regarded as
two aspects of user preference that are of utmost importance in Web
success (Palmer, 2002; Muylle et al., 2004; Zviran et al., 2006; Lee and
Koubek, 2010). This is particularly the case for online banking, which
its users are growing rapidly worldwide due to its convenience and ease
of transactions and customer services (Yoon, 2010). Currently, almost
all Iranian banks are offering online banking. Iran, with one of the
highest rates of internet-using population in the world (over 33 million
users), has a constantly growing part of its population using online
banking (Hojjati et al., 2015). It is, therefore, important to investigate
and understand the usability and user satisfaction to ensure usercentred design of these websites.
1.1. User-centred web design guidelines
In recent years, increasing attention has been paid to the design of
websites by researchers from various disciplines. A number of publications have suggested guidelines and recommendations regarding
website design (For example, see Abelse et al., 1998; Shneiderman,
1998; Head, 1999; Palmer, 2002; Garrett, 2003; McCracken and Wolfe,
2004). However, as acknowledged by several investigators (Zviran
et al., 2006; Lee and Koubek, 2010), these design guidelines are either
insufficient or not detailed enough to be applicable to specific cases.
Abelse et al. (1998) proposed six criteria (including use, content,
linkage, structure, special features, and appearance) for constructing
Web sites with the user in the focus. Other investigators have
*
Corresponding author.
E-mail addresses: dianati@tbzmed.ac.ir (I. Dianat), pari.adeli@yahoo.com (P. Adeli), m_asgharl862@yahoo.com (M. Asgari Jafarabadi),
karimi_a@live.com (M.A. Karimi).
https://doi.org/10.1016/j.apergo.2019.102892
Received 31 May 2017; Received in revised form 6 November 2018; Accepted 30 June 2019
I. Dianat, et al.
considered three elements including information content, navigation
design, and visual design for website design (Cyr and Bonanni, 2005;
Cyr, 2008). Structure, layout, and aesthetic aspects were recommended
by Lee and Koubek (2010) as Web design attributes.
1.2. Website usability
Usability is a relatively complex concept because of the complexity
of human beings as well as the product features itself. For this, the
usability has been defined in several different ways and various
methods have been proposed to measure it. Flavián et al. (2006) define
the website usability as “the perceived ease of navigating the site or
making purchases through the Internet”. The contexts of use and user's
experience of interacting with the system are key elements of usability
in human–computer interaction (HCI) (Baber, C., 2005). To the authors'
knowledge, the relationship between usability and different Web design
attributes has not been explored in a multivariate context. As a result, it
is difficult to determine which Web design attributes are more important from a usability point of view.
In addition, while several previous studies have examined the effect
of website usability on user satisfaction when shopping online (Fu and
Salvendy, 2002; Schaupp and Bélanger, 2005; Flavián et al., 2006;
Casaló et al., 2008; Belanche et al., 2012; Tandon et al., 2016), much
less attention has been paid to other online services such as online
banking. And some of these studies have shown that usability is the
most important predictor of user satisfaction (Belanche et al., 2012;
Tandon et al., 2016), while others have not shown this to be the case
(Schaupp and Bélanger, 2005). Therefore, additional research may be
needed to shed further light on this issue.
1.3. Website user satisfaction
The rapid growth of online services emphasises the need for increased attention to user satisfaction as a key factor in designing
websites. Website user satisfaction can be defined as: “the attitude toward the website by a hands-on user of the site” (Muylle et al., 2004).
Research on user satisfaction is very important because it not only influences the customer behavioural outcomes such as loyalty, trust and
purchase intention (Shankar et al., 2003; Gustafsson et al., 2005;
Flavián et al., 2006), but also is a key factor to profitability (Chiou and
Shen, 2006).
Researchers have established various models from different perspectives to describe the user satisfaction (Doll et al., 1994; Szymanski
and Hise, 2000; McKinney et al., 2002; Muylle et al., 2004; Schaupp
and Bélanger, 2005). Doll et al. (1994) proposed a model of website
user satisfaction that consisted of five system attributes including content, format, accuracy, timeliness, and ease of use. A number of researchers have highlighted the impact of website's general design
(Szymanski and Hise, 2000; Liu et al., 2008; Yoon, 2010; Guo et al.,
2012; Tandon et al., 2017) or some specific aspects of website design
such as interface consistency (Ozok and Salvendy, 2000) or website
quality (McKinney et al., 2002; Kim and Stoel, 2004; Ha and Im, 2012)
on user satisfaction. However, these studies have not specifically focused on the impact of different Web design attributes on user satisfaction. In one of the few attempts to address this issue, Zviran et al.
(2006) found that content and search capabilities of commercial websites were predictors of user satisfaction. From a user-centred design
point of view, this is particularly important because better understanding in this area may lead to the development of more specific
guidelines and recommendations with regard to website design, and
consequently to more user satisfaction. In addition, there are variations
between different cultures in terms of design preferences and user satisfaction (Marcus and Gould, 2000; Cyr, 2008; Liu et al., 2008), which
underscore the need for further research in this area.
1.4. Users’ personal characteristics
A review of literature reveals that very few studies have examined
the role of demographic differences in relation to website design preferences. In an e-business setting, Simon (2001) examined differences in
perception of website design (in terms of information richness, communication effectiveness, and the communication interface) between
genders and found that female users prefer websites with less clutter
and fewer graphics compared to male users. Relatively similar findings
were reported by Simon and Peppas (2005), who evaluated simple and
complex product websites. Another study, conducted by Cyr and
Bonanni (2005) on website design in e-business, demonstrated gender
differences in perception of website design (in terms of information
design, navigation design and visual design) and website satisfaction.
Some other researchers have examined the influence of a specific design
factor (e.g. aesthetic perception of websites) and reported significant
gender differences in this regard (Cousaris et al., 2008; Tuch et al.
(2010). The users’ attitudes and behaviours may vary according to their
other personal characteristics and culture (Yoon, 2010). It is, therefore,
apparent that further research is required to examine the individual
differences such as gender, age, occupation, education and Web usage
experience in relation to various aspects of Web design as well as to
usability and user satisfaction. Better understanding and knowledge in
this area will have important implications for the Web design process
and help designers to consider the needs and expectations of different
users and design inclusively for them.
1.5. Rationale
Based on the above mentioned background, the aim of this study
was to answer the following research questions:
1) How are the user-centred Web design, usability, and user satisfaction of online banking websites in Iran?
2) How is the relationship of user-centred Web design and users' personal characteristics (including gender, age, occupation, education
and Web usage experience) to usability and user satisfaction of
online banking websites?
3) How is the relationship between usability and user satisfaction of
online banking websites?
2. Methodology
2.1. Study setting, participants and procedure
A total of 798 online banking users of four main banks (Table 1) in
Iran participated in this study. The study was conducted over a fivemonth period between August and December 2016 in Tabriz–Iran.
Users who were older than 18 years, and had used the corresponding
online banking system at least for one year, once a week, were considered as the target population for the study. Permission to access the
research sites and conduct the study was obtained from the bank authorities involved. One investigator then visited the selected bank
branches for data collection. A questionnaire was administered to collect data about demographic factors (including age, gender, occupation,
education, and Web usage experience), Web design attributes (by using
Table 1
Four online banking websites evaluated in the study.
Websites
URL
Website
Website
Website
Website
https://www.bmi.ir
http://www.banksepah.ir
http://www.bsi.ir
http://www.tejaratbank.ir
1
2
3
4
I. Dianat, et al.
the user-centred Web design (UCWD) questionnaire), website usability
(by using the system usability scale [SUS]), and user satisfaction of
websites (by using the end-user satisfaction (EUS) assessment scale).
The participating users were instructed to complete the questionnaire in
the bank premises and return it to the investigator before leaving.
Participation in this study was voluntarily and written consent was
obtained from each user before participation. Ethical approval for the
study was granted from the ethical review committee of the Tabriz
University of Medical Sciences.
2.2. Instrument and outcome measures
The instruments used in the study included the following: 1) UCWD
questionnaire (Abelse et al., 1998), 2) SUS (Brooke, 1996), and 3) EUS
assessment scale (Doll et al., 1994). Demographic details including age,
gender, occupation (office worker, manual worker, university student,
and other), education level (diploma, undergraduate and postgraduate), and Web usage experience (years) were also recorded.
Items regarding the UCWD were based on the questionnaire developed originally by Abelse et al. (1998), and further developed by
Zviran et al. (2006). This is a valid and reliable tool which contains 17
questions regarding the website personalisation (3 items, personalisation of the site and complete information on basic facts and product
features), structure (2 items, logical categorisation of the content and
multimedia/graphics that strictly support the site purpose), navigation
(2 items, error handling and navigation aids), layout (4 items, nonoverlapped and consistent navigation bars, update clues indicating
user's location on the site and page arrangement), search (3 items,
dealing with misspellings and synonyms and presenting the results in
relevant order), and performance (3 items, cross-browser compatibility
and browser-specific requirements as well as errors such as JavaScript
error messages of the site). Using a 5-point Likert scale (from 1 – very
low to 5 – very high) respondents rated their reactions to the usercentred design of banking websites.
Website usability was measured using the SUS proposed by Brook
(1996). The SUS is a widely used and validated tool for measuring the
perceived usability of a wide range of products and user interfaces,
including websites (Brooke, 1996; Zviran et al., 2006). The SUS is a 10item scale that evaluates various aspects of usability of a system or
product. All items have 5-point Likert scale responses (from 1 – strongly
disagree to 5 – strongly agree), and the scale gives an overall usability
score ranging from 0 to 100. Higher scores indicate higher levels of
usability.
Finally, the EUS assessment scale (Doll et al., 1994), which is a valid
and reliable instrument and consists of 12 items, was used to measure
the users’ satisfaction from banking websites. It consists of five criteria
for assessing the user satisfaction including content (4 items, provision
of required, precise and sufficient information), accuracy (2 items, accuracy of the system and satisfaction with it), format (2 items, useful
format and clear information), ease of use (2 items, ease of use and user
friendliness of the system) and timeliness (2 items, required and up to
date information). Each item is rated on a 5-point Likert scale (from 1 –
very low to 5 – very high), where a higher score indicates a higher
satisfaction level.
2.3. Validity and reliability issues
The original version of the SUS has been translated and revised into
Farsi (Iranian language), and therefore has an established reliability
and validity (Dianat et al., 2014, 2015). The English versions of UCWD
and EUS scales were also translated into Farsi and verified for content
validity by ten experts (ergonomists and psychologists). In addition,
internal consistency reliability (using Cronbach's alpha) and stability
reliability (using Intraclass correlation coefficient – ICC) of the measures on a sample of 81 participants were also found to be good. The
Cronbach's alpha for the SUS was 0.82. The Cronbach's alpha for the
Table 2
Users’ personal characteristics and their relationships with the study variables
(n = 798).
n (%)
Gender
Male
Female
Age (yr)
< 30
30–45
> 45
Occupation
Office worker
Manual worker
University student
Other
Education
Diploma
Undergraduate
Postgraduate
Web usage experience
≤2
3–4
≥5
Web type
Website 1
Website 2
Website 3
Website 4
Total
UCWD score
SUS score
EUS score
Mean (SD)
Mean (SD)
Mean (SD)
515 (64.5)
283 (35.5)
42.5 (6.0)
41.3 (5.8)
51.8 (7.5)
52.3 (8.2)
41.3 (6.3)
40.8 (6.3)
86 (10.8)
601 (75.3)
111 (13.9)
43.2 (4.8)
41.9 (6.2)
42.3 (5.5)
52.1 (6.9)
52.0 (7.9)
51.6 (7.5)
41.5 (6.3)
41.2 (6.4)
40.8 (5.5)
458 (57.4)
96 (12.0)
86 (10.8)
158 (19.8)
41.8
43.3
41.4
52.3
51.7
53.2
53.1
51.9
41.3
42.1
40.2
40.5
88 (11.0)
486 (60.9)
224 (28.1)
(yr)
296 (37.1)
303 (38.0)
199 (24.9)
200
199
200
199
(25.1)
(24.9)
(25.1)
(24.9)
798 (100)
(6.1)
(5.3)
(5.1)
(6.1)
(7.6)
(7.8)
(8.5)
(7.8)
(6.3)
(5.2)
(6.9)
(6.5)
42.3 (5.9)
42.2 (5.8)
41.8 (6.3)
53.6 (7.3)*
52.3 (7.9)
50.9 (7.2)
41.9 (7.4)
41.2 (6.2)
41.0 (6.4)
42.8 (5.7)*
41.6 (6.2)
41.7 (5.9)
52.4 (7.7)
51.5 (8.0)
52.0 (7.3)
41.3 (6.4)
40.6 (6.0)
41.7 (6.6)
42.6 (5.2)***
42.9 (5.1)
39.0 (6.1)
43.9 (6.2)
53.3 (7.3)***
52.7 (7.5)
49.5 (9.0)
52.5 (6.2)
41.4
40.7
40.6
41.9
42.1 (5.9)
52.0 (7.7)
41.1 (6.3)
(5.6)
(5.7)
(7.2)
(6.5)
Statistically significant values are shown in bold.
*p < 0.05; **p < 0.01; ***p < 0.001.
UCWD and its subscales were as: total UCWD (0.78), personalisation
(0.69), structure (0.73), navigation (0.71), layout (0.75), search (0.71),
and performance (0.72). The Cronbach's alpha for the EUS and its
subscales were as: total EUS (0.88), content (0.86), accuracy (0.76),
format (0.72), ease of use (0.68) and timeliness (0.70). The ICC index
for each of the three measures was 0.99 (95% CI = 0.98 to 1.00).
2.4. Data analysis
Statistical analysis of the data was performed using STATA software,
V.13 (StataCorp, College Station, Texas 77845 USA). Normality of the
numeric variables was checked by Kolmogorov–Smirnov test. Data
were presented as mean (standard deviation−SD) and median (minmax) for the numeric normal and non-normal variables, respectively, as
appropriate, and frequency (%) for categorical variables.
Between–group comparisons of baseline measures and demographic
variables were performed using independent t-test, analysis of variance
(ANOVA) or Chi-square test, as appropriate. Univariate and multivariate regression modelling was also used to examine the relationship
between study variables. In the multivariate models, the effects of
confounders were adjusted and the categorical variables were entered
as indicators. To comprehensively examine the relationship between
usability and user satisfaction with other study variables, and considering the effect of confounders, a three step hierarchical modelling
was utilised in three models. In the first model, the gender, age, educational level, occupation, Web type and Web usage experience were
selected as the independent variables and the usability (SUS score) and
user satisfaction (total EUS score) were selected as dependent variables.
The second model examined the relationship of independent variables
in the Model 1 plus Web design attributes (dimensions of UCWD including rationalisation, structure, navigation, layout, search and performance) to usability and user satisfaction as dependent variables. The
third model examined the relationship of independent variables in the
I. Dianat, et al.
Table 3
Univariate regression analyses for the relationship between study variables.
Variables
Usability
B
Gender
Male
Female
Age (yr)
< 30
30-45
> 45
Educational level
Diploma
Undergraduate
Postgraduate
Occupation
Office worker
Manual worker
Student
Others
Web type
Website 1
Website 2
Website 3
Website 4
Web usage experience (yr)
1–2
3–4
≥5
UCWD
Personalization
Structure
Navigation
Layout
Search
Performance
Usability (SUS score)
End-user satisfaction
SE
95% CI
B
Lower bound
Upper bound
SE
95% CI
Lower bound
Upper bound
1.00
0.444
–
0.575
–
−0.68
–
1.57
1.00
−0.532
–
0.469
–
−1.45
–
0.38
1.00
−0.062
−0.484
–
0.897
1.117
–
−1.82
−2.67
–
1.69
1.70
1.00
−0.361
−0.731
–
0.732
0.913
–
−1.80
−2.52
–
1.07
1.06
1.00
−1.249
−2.647
–
1.300
1.359
–
−3.80
−5.31
–
1.30
0.02
1.00
−0.762
−0.946
–
1.067
1.115
–
−2.85
−3.13
–
1.33
1.24
1.00
1.509
1.405
0.275
–
0.872
1.344
0.649
–
−0.20
−1.23
−0.99
–
3.22
4.04
1.55
1.00
0.760
−1.142
−0.802
–
0.711
1.097
0.530
–
−0.63
−3.29
−1.84
–
2.15
1.01
0.23
1.00
−0.636
−3.850***
−0.762
–
0.765
0.764
0.765
–
−2.13
−5.35
−2.26
–
0.86
−2.34
0.73
1.00
−0.751
−0.795
0.429
–
0.635
0.634
0.635
–
−1.99
−2.03
−0.81
–
0.49
0.44
1.67
1.00
−0.940
−0.393
–
0.635
0.712
–
−2.18
−1.79
–
0.30
0.100
1.00
−0.721
0.365
–
0.518
0.581
–
−1.73
−0.77
–
0.29
1.50
0.995***
1.840***
1.845***
1.566***
1.040***
1.698***
–
0.162
0.206
0.020
0.097
0.175
0.293
–
0.67
1.43
1.44
1.37
0.69
1.12
–
1.31
2.24
2.24
1.75
1.38
2.27
–
1.505***
3.158***
2.060***
1.381***
0.930***
1.204***
0.295***
0.125
0.136
0.157
0.077
0.142
0.241
0.027
1.25
2.89
1.75
1.22
0.64
0.73
0.24
1.75
3.42
3.69
1.53
1.21
1.67
0.34
Statistically significant values are shown in bold.
*p < 0.05; **p < 0.01; ***p < 0.001.
Model 2 plus usability (SUS score) to user satisfaction as dependent
variable. Standardised regression coefficients (β) and their corresponding 95% confidence intervals (CIs) as well as explanatory power
(adjusted R-square – R2) were presented to indicate the predictive
strength of independent variables and the explanatory power of each
model, respectively. P values less than 0.05 were considered as statistically significant for all tests.
(p < 0.001) (Table 2). Web usage experience had also a significant
effect on the mean UCWD score (p < 0.05), so that a longer experience
(more than 2 years) of using the internet resulted in lower scores on the
UCWD scale. However, no significant difference was found in the
UCWD score in terms of the age and gender of the participants.
3. Results
The mean (SD) SUS score was 52.0 (7.7). The ANOVA results indicated a statistically significant difference between the four evaluated
websites in term of their usability (p < 0.001). A significant difference
was also found in the SUS scores in terms of the education level of the
respondents (p < 0.05). This indicated that those participants with a
lower level of education (e.g. diploma) rated higher scores on the SUS
than those with a higher level of education. Other demographic variables had no significant effect on the SUS scores.
3.1. Demographic details
A total of 900 online banking customers were approached, of which
833 (response rate = 92.5%) accepted to participate in the study. Of
those that completed the questionnaire, 35 were excluded for incomplete questionnaires. Therefore, the responses from 798 (515 males
and 283 females) were included in the analyses. The age of participants
ranged from 19 to 55 years (mean = 33.9 years; SD = 6.1 years). They
had been using the internet banking between 1 and 6 years
(mean = 3.3 years; SD = 1.4 years). More than half of the respondents
(57.4%) were office workers, and had undergraduate (60.9%) education.
3.3. Usability
3.4. End-user satisfaction
The mean (SD) score for the EUS scale was 41.1 (6.3). The results of
the study showed no statistically significant difference within the subgroups of the demographic factors in terms of the EUS score.
3.2. User-centred web design
3.5. Regression models
The mean (SD) score of the total UCWD scale was 42.1 (5.9).
According to the ANOVA results, the mean UCWD score was significantly different between the four websites evaluated in the study
The results of the univariate and multivariate regression analyses
are summarised in Table 3 through 5. According to the results of
multivariate regression modelling presented in Table 4, both Web
I. Dianat, et al.
Table 4
Multivariate regression analysis for the relationship between study variables and website usability.
Model 1
β
Gender
Male
Female
Age (yr)
< 30
30-45
> 45
Educational level
Diploma
Undergraduate
Postgraduate
Occupation
Office worker
Manual worker
Student
Others
Web type
Website 1
Website 2
Website 3
Website 4
Web usage experience (yr)
1–2
3–4
≥5
UCWD
Personalization
Structure
Navigation
Layout
Search
Performance
Model 2
β
95% CI
Lower bound
Upper bound
1.00
0.044
–
−0.40
–
1.86
1.00
0.040
−0.006
–
−2.09
−2.99
–
0.96
0.84
1.00
−0.127
−0.209**
–
−4.67
−6.33
–
0.46
−0.89
1.00
0.053
0.075
0.021
–
−0.57
−0.01
−1.04
–
3.11
5.68
1.78
1.00
−0.042
−0.227***
−0.040
–
−2.29
−5.59
−2.29
–
0.78
−2.55
0.83
1.00
−0.022
0.029
–
−1.67
−1.09
–
0.95
2.15
95% CI
Lower bound
Upper bound
1.00
−0.087
−0.139*
–
−3.62
−4.67
–
0.73
−0.13
1.00
−0.060
−0.141***
−0.048
–
−2.36
−3.92
−2.21
–
0.21
−1.14
0.48
−0.038
0.052
0.004
0.456***
−0.006
0.204***
−0.51
-.13
−0.43
1.19
−0.39
1.20
0.15
0.77
0.49
1.69
0.32
2.25
Model 1: Adjusted for gender, age, educational level, occupation, Web type and Web usage experience. Model 2: Model 1 plus Web design attributes (personalization,
structure, navigation, layout, search and performance).
Adjusted R2: Model 1 = 0.042; Model 2 = 0.293.
Statistically significant values are shown in bold.
*p < 0.05; **p < 0.01; ***p < 0.001.
design attributes and individual factors were associated with website
usability. However, the relationships of Web design attributes such as
layout (β = 0.456, 95% CI: 1.19 to 1.69, p < 0.001) and performance
(β = 0.204, 95% CI: 1.20 to 2.25, p < 0.001) were stronger than the
association of individual factors such as education level (β = −0.139,
95% CI: -4.67 to −0.13, p < 0.05). About 29% of the variance in usability was accounted for by variables in the model.
With regard to the relationship of Web design attributes and individual factors to user satisfaction, the results of the multivariate regression analysis (shown in Table 5) indicated that Web design attributes had strong association with user satisfaction, whereas individual
factors (such as age, gender and Web use experience) had no effect.
Among the Web design attributes, Web structure had the strongest association (β = 0.494, 95% CI: 2.16 to 2.76, p < 0.001), while other
attributes such as layout (β = 0.171, 95% CI: 0.26 to 0.62, p < 0.001),
personalisation (β = 0.131, 95% CI: 0.28 to 0.71, p < 0.001), search
(β = 0.120, 95% CI: 0.25 to 0.74, p < 0.001) and performance
(β = 0.085, 95% CI: 0.23 to 0.94, p < 0.001) had weaker associations.
Altogether, these five variables explained 51% of the variance in user
satisfaction model. The results also showed a significant association
between website usability and user satisfaction (β = 0.060, 95% CI:
0.01 to 0.09, p < 0.05).
4. Discussion
With dramatic increasing number of websites and users who have to
use them as a routine in their living and work activities, the issue of
usability and user satisfaction of these interfaces are getting more and
more important for both companies/organisations and users. In this
study, online banking websites were selected to model the relationship
between Web design attributes and users' personal characteristics to
usability and user satisfaction in this regard. The relationship of website
usability and user satisfaction was also examined in this research. The
main findings of the study were that the design and usability of the
evaluated websites, despite their high importance, were not satisfactory
from the users' perspectives. Users' personal characteristics influenced
the Web design and usability scores, but not the user satisfaction.
Usability and user satisfaction differed by the manner in which they
were influenced by the Web design attributes and users' personal
characteristics. The former was influenced by both Web design attributes and users' personal characteristics, while the latter was only affected by Web design attributes. Website usability itself had a significant impact on user satisfaction. These findings can help to better
understand the users’ needs and expectations, and consequently to
optimise the website design.
With regard to website usability, the SUS scores obtained for different banking websites ranged between 49 and 53, which indicates a
moderate level of usability. This finding highlights a call for better ergonomic design of banking websites evaluated in the study. From an
ergonomic point of view, poor or unacceptable website usability may be
attributed to the fact that most Web developers have been mistakenly
guided by the challenges of technology rather than by users’ needs
I. Dianat, et al.
Table 5
Multivariate regression analysis for the relationship between study variables and end-user satisfaction of websites.
Variables
Model 1
β
Gender
Male
1.00
Female
−0.036
Age (yr)
< 30
1.00
30-45
−0.038
> 45
−0.058
Educational level
Diploma
1.00
Undergraduate
−0.082
Postgraduate
−0.088
Occupation
Office worker
1.00
Manual worker
0.032
Student
−0.039
Others
−0.057
Web type
Website 1
1.00
Website 2
−0.060
Website 3
−0.049
Website 4
0.029
Web usage experience (yr)
1–2
1.00
3–4
−0.059
≥5
0.014
UCWD
Personalisation
Structure
Navigation
Layout
Search
Performance
Usability (SUS score)
Model 2
β
95% CI
Lower bound
Upper bound
–
−1.42
–
0.46
–
−2.09
−2.99
–
0.96
0.84
–
−3.25
−3.50
–
1.02
1.01
–
−0.89
−3.59
−2.00
–
2.17
1.16
0.34
–
−2.16
−1.99
−0.86
–
0.39
0.54
1.73
–
−1.86
−1.14
–
0.32
1.56
0.128***
0.498***
−0.007
0.199***
0.121***
0.096***
Model 3
β
95% CI
Lower bound
Upper bound
0.27
2.18
−0.35
0.34
0.26
0.32
0.70
2.79
0.27
0.68
0.74
1.01
95% CI
Lower bound
Upper bound
0.131***
0.494***
0.28
2.16
0.71
2.76
0.171***
0.120***
0.085***
0.060*
0.26
0.25
0.23
0.01
0.62
0.74
0.94
0.09
Model 1: Adjusted for gender, age, educational level, occupation, Web type and Web usage experience. Model 2: Model 1 plus Web design attributes (personalization,
structure, navigation, layout, search and performance). Model 3: Model 2 plus suability (SUS score).
Adjusted R2: Model 1 = 0.005; Model 2 = 0.510; Model 3 = 0.512.
Statistically significant values are shown in bold.
*p < 0.05; **p < 0.01; ***p < 0.001.
(Helander and Khalid, 2000). It is, therefore, essential to overcome the
challenges associated with website functionality, while not compromising its usability. Based on our findings, two aspects of Web design
including Web layout and performance were the main predictors of
usability, while personal characteristics of users such as gender, age and
Web usage experience had no effect. This means that, in terms of usability, special attention should be given to the website layout issues
such as navigation bar design and page arrangement as well as to the
Web page performance metrics including browser-related and error
management issues to improve the usability of banking websites under
study. However, it should be pointed out that usability problems may
be unique to each type of website depending on its content and application, and therefore further research is needed to draw firm conclusions on this issue.
As shown in this research, all the Web design attributes were highly
related to user satisfaction, except for the navigation, which is generally
not surprising in view of previous research (Cyr, 2008; Liu et al., 2008;
Yoon, 2010). But more importantly, the website structure had the
strongest relationship in this regard. This result is, in part, consistent
with the finding of Zviran et al. (2006), who found that user satisfaction
was more related to content and search than other attributes of different types of commercial websites (e.g. online shopping, customer
self-service, trading, and publish/subscribe). Although the amount,
heterogeneity and applications of websites are so great that it is difficult
to give general guidelines and recommendation for the Web design, it
seems that the structure of a website is perhaps a more important design consideration than other attributes for this type of websites. More
specifically, improved organisation of the information in the site (e.g.
logical categorisation of the content and use of appropriate multimedia/graphics) is likely the most effective way to increase user satisfaction. It should, however, not be forgotten that the relationship
between user-centred design and user satisfaction may be differentially
affected depending on the website content and application. For example, some investigators have highlighted the importance of information content and system quality in terms of delivery of the information as most important indicators of user satisfaction in online
shopping environments (McKinney et al., 2002). Thus, further studies
evaluating this relationship in other types of websites are recommended. However, such studies should preferably focus on the
impact of different Web design attributes on user satisfaction in a
multivariate context, rather than on the website's general design or on
some specific aspects of website design.
It is also of interest to note that according to the multivariate regression models, demographic and individual variables such as gender,
age and Web usage experience had no influence on the user satisfaction
of websites. There are very few studies that have examined users' personal characteristics and preferences in relation to website satisfaction.
In a study by Cyr and Bonanni (2005) about website design in e-business, the authors found significant gender differences in perceptions of
website design and website satisfaction, which contradicts the findings
I. Dianat, et al.
of this study. This may be attributed to either differences in the types of
websites being evaluated or to cultural differences in these studies.
Similarly, the regression models showed no relationship between age or
Web usage experience of users and their satisfaction with websites.
These findings are relatively new to the literature and have important
implications for the website design. These findings may rule out the
possibility of individual differences in terms of user satisfaction of
websites. Based on these findings, it can be concluded that website
designers should focus more on Web design attributes as a more important design consideration than individual differences for increasing
the satisfaction of their users. Again, these findings provide a basis for
informing the relationship between users’ personal characteristics and
their satisfaction from banking websites but would need further validation for other types of websites and cultural backgrounds.
One of the main contributions of this study is that the relationship of
Web design attributes and users’ personal characteristics to website
usability and user satisfaction was evaluated in a multivariate context.
However, it should be noted that the development of recommendations
regarding website design attributes in this study was based on online
banking websites with emphasis on usability and user satisfaction as
two important measures of Web success. Obviously, UCWD can be
evaluated from different point of view such as performance, error, etc.
Moreover, Web design attributes considered important by users may
differ depending on the website content and application. In addition,
this study was conducted in Iran, where the culture and social structure
differ greatly from those of the developed or other developing countries. Therefore, further studies on different types of websites, using
different measures of UCWD, in other settings or communities are required to have a better understanding and knowledge in this regard.
Moreover, the findings presented in this research identify the most
important website design parameters in terms of usability and user
satisfaction which then provide a basis for further research. To take
account of this, the website layout and structure which have shown to
be the most important parameters in this regard should be evaluated by
both subjective (e.g. user preference, ease of use, etc.) and objective
(e.g. use of eye-tracking in navigation bar design) assessments.
5. Conclusion
In conclusion, the results presented in this study provide an insight
into the design of banking websites, with emphasis on usability and
user satisfaction as two important measures of Web success. Designing a
good website seems to be one of the challenges that Iranian banking
industry may encounter to achieve their goals, particularly in a limited
and competitive market. Two Web design attributes including Web
layout (e.g. navigation bar design and page arrangement) and performance (e.g. browser-related and error management issues) were shown
to be the main predictors of perceived usability, while users' personal
characteristics including gender, age and Web usage experience had no
effect in this regard. User satisfaction was also influenced only by the
Web design attributes and not by the personal characteristics of the
users. Among the Web design attributes, Web structure (e.g. improved
organisation of the information in the site) showed the strongest association with user satisfaction, followed by layout, personalisation,
search and performance. Website usability also influenced the user
satisfaction. The lack of association of users’ personal characteristics
with usability and user satisfaction is important and suggests that the
Web designers should focus more on the Web design attributes, regardless of the personal characteristics of their users, to improve the
usability and user satisfaction of websites.
Acknowledgements
The authors wish to acknowledge the support and assistance provided by the bank authorities, and all the participants who collaborated
in this research.
References
Abelse, E.G., White, M.D., Hahn, K., 1998. A user-based design process for Web sites.
Internet Res. 8, 39–48.
Baber, C., 2005. Evaluation in human–computer interaction. In: Wilson, J.R., Corlett, N.
(Eds.), Evaluation of Human Work. Taylor & Francis, Boca Raton.
Belanche, D., Casaló, L.V., Guinalíu, M., 2012. Website usability, consumer satisfaction
and the intention to use a website: the moderating effect of perceived risk. J. Retail.
Consum. Serv. 19, 124–132.
Brooke, J., 1996. SUS: a “quick and dirty” usability scale. In: Jordan, P.W., Thomas, B.,
Weerdmeester, B.A., McClelland, A.L. (Eds.), Usability Evaluation in Industry. Taylor
& Francis, London.
Casaló, L., Flavián, C., Guinalíu, M., 2008. The role of perceived usability, reputation,
satisfaction and consumer familiarity on the website loyalty formation process.
Comput. Hum. Behav. 24, 325–345.
Chiou, J., Shen, C., 2006. The effects of satisfaction, opportunism, and asset specificity on
consumers' loyalty intention toward internet portal sites. Int. J. Serv. Ind. Manag. 17,
7–23.
Cousaris, C., Swierenga, S., Watrall, E., 2008. An empirical investigation of color temperature and gender effects on web aesthetics. J. Usability Stud. 3, 103–117.
Cyr, D., 2008. Modeling website design across cultures: relationships to trust, satisfaction
and e-loyalty. J. Manag. Inf. Syst. 24, 47–72.
Cyr, D., Bonanni, C., 2005. Gender and website design in e-business. Int. J. Electron. Bus.
3, 565–582.
Dianat, I., Ghanbari, Z., Asghari-Jafarabadi, M., 2014. Psychometric properties of the
Persian language version of the system usability scale. Health Promot. Perspect. 4,
82–89.
Dianat, I., Nedaei, M., Nezami, M.A.M., 2015. The effects of tool handle shape on hand
performance, usability and discomfort using masons' trowels. Int. J. Ind. Ergon. 45,
13–20.
Doll, W.J., Xia, W., Torkzadeh, G., 1994. A confirmatory factor analysis of the end-user
computing satisfaction instrument. MIS Q. 18, 453–461.
Flavián, C., Guinalíu, M., Gurrea, R., 2006. The role played by perceived usability, satisfaction and consumer trust on website loyalty. Inf. Manag. 43, 1–14.
Fu, L., Salvendy, G., 2002. The contribution of apparent and inherent usability to a user's
satisfaction in a searching and browsing task on the Web. Ergonomics 45, 415–424.
Garrett, J.J. (Ed.), 2003. The Elements of User Experience: User-Centered Design for the
Web. New Riders Publications, Indiana.
Guo, X., Ling, K.C., Liu, M., 2012. Evaluating factors influencing customer satisfaction
towards online shopping in China. Asian Soc. Sci. 8, 40–50.
Gustafsson, A., Johnson, M.D., Ross, I., 2005. The effects of customer satisfaction, relationship commitment dimensions, and triggers on customer retention. J. Mark. 69,
210–218.
Ha, Y., Im, H., 2012. Role of web site design quality in satisfaction and word of mouth
generation. J. Serv. Manag. 23, 79–96.
Head, A.J. (Ed.), 1999. Design Wise: A Guide for Evaluating the Interface Design of
Information Resources. Information Today Inc., Medford, NJ.
Helander, M.G., Khalid, H.M., 2000. Modeling the customer in electronic commerce.
Appl. Ergon. 31, 609–619.
Hojjati, S.N., Monavvarian, A., Roozbahani, F.S., 2015. Evaluation of factors influencing
internet banking acceptance by internet users in Iran. Int. J. Adv. Netw. Appl. 6,
2503–2519.
Kim, S., Stoel, L., 2004. Apparel retailers: website quality dimensions and satisfaction. J.
Retail. Consum. Serv. 11, 109–117.
Lee, S., Koubek, R.J., 2010. The effects of usability and web design attributes on user
preference for e-commerce web sites. Comput. Ind. 61, 329–341.
Liu, X., He, M., Gao, F., Xie, P., 2008. An empirical study of online shopping customer
satisfaction in China: a holistic perspective. Int. J. Retail Distrib. Manag. 35,
919–940.
Marcus, A., Gould, E.W., 2000. Crosscurrents: cultural dimensions and global Web userinterface design. Interactions 7, 32–46.
McCracken, D.D., Wolfe, R.J. (Eds.), 2004. User-centered Website Development: A
Human–Computer Interaction Approach. Pearson Prentice Hall Inc., Upper Saddle
River, NJ.
McKinney, V., Yoon, K., Zahedi, F., 2002. The measurement of Web customer satisfaction:
an expectation and disconfirmation approach. Inf. Syst. Res. 13, 296–315.
Muylle, S., Moenaert, R., Despontin, M., 2004. The conceptualization and empirical validation of web site user satisfaction. Inf. Manag. 41, 543–560.
Ozok, A.A., Salvendy, G., 2000. Measuring consistency of web page design and its effects
on performance and satisfaction. Ergonomics 43, 443–460.
Palmer, J.W., 2002. Web site usability, design, and performance metrics. Inf. Syst. Res.
13, 151–167.
Shankar, V., Smith, A.K., Rangaswamy, A., 2003. Customer satisfaction and loyalty in
online and offline environments. Res. Market. 20, 153–175.
Schaupp, L.C., Bélanger, F., 2005. A conjoint analysis of online consumer satisfaction. J.
Electron. Commer. Res. 6, 95–111.
Shneiderman, B., 1998. Designing the User Interface: Strategies for Effective
Human–Computer Interaction. Addison-Wesley Longman Publishing Co. Inc.,
Boston, MA.
Simon, S., 2001. The impact of culture and gender on web sites: an empirical study. ACM
SIGMIS - Data Base 32, 18–37.
Simon, J., Peppas, S., 2005. Attitudes towards product website design: a study of the
effects of gender. J. Mark. Commun. 11, 18–37.
Szymanski, D.M., Hise, R.T., 2000. E-satisfaction: an initial examination. J. Retail. 76,
309–322.
I. Dianat, et al.
Tandon, U., Kiran, R., Sah, A.N., 2016. Customer satisfaction using website functionality,
perceived usability and perceived usefulness towards online shopping in India. Inf.
Dev. 32, 1657–1673.
Tandon, U., Kiran, R., Sah, A.N., 2017. The influence of website functionality, drivers and
perceived risk on customer satisfaction in online shopping: an emerging economy
case. Inf. Syst. E Bus. Manag. 16, 57–91.
Tuch, A.N., Bargas-Avila, J.A., Opwis, K., 2010. Symmetry and aesthetics in website
design: it's a man's business. Comput. Hum. Behav. 26, 1831–1837.
Yoon, C., 2010. Antecedents of customer satisfaction with online banking in China: the
effects of experience. Comput. Hum. Behav. 26, 1296–1304.
Zviran, M., Glezer, C., Avni, I., 2006. User satisfaction from commercial web sites: the
effect of design and use. Inf. Manag. 43, 157–178 .
Received February 10, 2020, accepted March 16, 2020, date of publication March 19, 2020, date of current version March 30, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.2981892
Usability of Mobile Applications: A Systematic
Literature Study
PAWEä WEICHBROTH
Department of Software Engineering, Faculty of Electronics, Telecommunications and Informatics, Gda´sk University of Technology, 80-233 Gda´sk, Poland
e-mail: pawel.s.weichbroth@gmail.com
ABSTRACT Since the release of the first mobile devices, the usability of on-board applications has been the
concern not only of software vendors but hardware manufacturers as well. The academia community later
willingly joined the discussion on usability in terms of theory and empirical measurement, having experience
and knowledge in desktop settings. At first sight, such a background should guarantee a solid foundation
to conduct research on software usability in a new setting. However, a preliminary study on the subject
matter revealed methodological disorder in contemporary literature. As a matter of fact, a need emerged to
review existing usability definitions, attributes and measures to recognize all associated aspects. In order
to fill this void, we conducted a systematic literature review on usability studies indexed by the Scopus
database and devoted to mobile applications. The input volume covers 790 documents from 2001 to 2018.
The data analysis shows that the ISO 9241-11 usability definition has been adopted in an unchanged form
and popularized as the standard by the HCI community. Secondly, in total, 75 attributes were identified and
analysed. The most frequent are efficiency (70%), satisfaction (66%) and effectiveness (58%), which directly
originate from the above definition. Subsequently, the less frequent are learnability (45%), memorability
(23%), cognitive load (19%) and errors (17%). The last two concern simplicity (13%) and ease of use (9%).
Thirdly, in the evaluation of usability, controlled observation and surveys are two major research methods
applied, while eye-tracking, thinking aloud and interview are hardly used and serve as complementary to
collect additional data. Moreover, usability evaluations are often confused with user experience dimensions,
covering not only application quality characteristics, but also user beliefs, emotions and preferences. All these
results indicate the need for further research on the usability of mobile applications, aiming to establish a
consensus in the theory and practice among all interested parties.
INDEX TERMS Mobile applications, usability, attributes, measures, usability evaluation methods, systematic literature review.
I. INTRODUCTION
Amobile application is defined as ‘‘a software application
developed specifically for use on small, wireless computing
devices, such as smartphones and tablets, rather than desktop or laptop computers’’ [1]. A recent Statista report shows
that in 2017 smartphones had a share of 77% of the global
mobile device market [2], and more than 32% of the global
population used a smartphone [3].
Although technological progress has been made regarding mobile devices equipped with computing power, leading
to a shift from desktop computers, many limitations and
The associate editor coordinating the review of this manuscript and
approving it for publication was Mario Luca Bernardi
VOLUME 8, 2020
.
challenges still remain [4]. From the many identified, usability has been the main concern, since the users of an
application, and their judgment, ultimately decide on its
success or failure [5]–[7]. Since the inception of the first
smartphones, the subject of mobile application usability has
gained attention both in academia communities and in the
software vendors industry. While researchers are focused
on formulating theories [8], modelling frameworks [9], and
constructing methods and techniques [10], [11] for new settings, manufacturers simply desire to deliver high quality
products [12].
Despite the abundance of research devoted to studies of
mobile application usability on the one hand, and design
patterns, prototyping tools and software frameworks on the
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
55563
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
other, the term tends to be vague and loose, weakening the
ability to capture its real facets and impeding the construction of measures. As a consequence, such methodological
disorder violates the core assumptions and principles laying
beneath the foundations of the usability notion. Therefore,
considering the need for the emergence of a usability definition, its attributes and measures, along with evaluation
methods, valid for mobile applications, in this paper we made
an attempt to find reliable answers by conducting a systematic
literature review. We expect that the obtained results can be
used not only by researchers to perform further studies in this
area, but also for practitioners engaged in mobile application
development and quality-in-use evaluation to better understand the characteristics and measures of the notion.
The main contributions of this study include: (i) an
evidence-based discussion of the usability definition, its
attributes and measures, (ii) and an up-to-date map of the state
of the art in usability evaluation methods (UEMs), adopted
for and adapted to mobile applications, covering publications
from 2001 to 2018.
The rest of the paper unfolds as follows. Section 2 provides
the background on the subject addressed, and related work.
Section 3 describes the research methodology. The definition
and execution of the literature review are respectively presented in Sections 4 and 5. Section 6 provides an analysis
of the extracted data, while the results are further discussed
in Section 7, along with the future research directions. The
conclusions are raised in Section 8.
II. BACKGROUND
Most people tend to use products that are easy to understand,
work as expected, and eventually deliver value. In the context
of the software engineering, system usability plays the crucial
role in shaping perceived quality in use by its users [13], [14].
Usability is the study of the intersection of between systems and users, tasks and expectations in the context of
use. Since many software products have been determined to
be insufficient to meet user needs, several comprehensive
studies have been conducted so far under the term usability,
which move towards a better understanding and relevant
measurement, aiming to cover all valid phenomena in one
framework or model [15]–[17].
The results of the study, introduced by Weichbroth
[18], show that over time the definition of usability has
evolved. In 1991 the Organization for Standardization (ISO),
in response to the emergence of the need of the software
community to standardize some facets of software products,
publicized the 9126 standard, which defines usability as ‘‘a
set of attributes of software which bear on the effort needed
for use, and on an individual assessment of such use, by a
stated or implied set of users’’ [19].
Then, in 1998, ISO refashioned the usability definition in
the ISO 9241-11 norm, which actually states that usability
is ‘‘the extent to which a product can be used by specified
users to achieve specified goals with effectiveness, efficiency
and satisfaction in a specified context of use’’ [20], [21].
55564
While some argue that it is the most recognizable definition [18], others maintain that ‘‘a generally accepted usability
definition still does not exist, as its complex nature is hard to
describe in one definition’’ [22], [23].
The other usability definition can be found in ISO/IEC
25010 [24], which replaced the ISO/IEC 9126 standard from
2001 [25], and specifies usability as the ‘‘degree to which a
product or system can be used by specified users to achieve
specified goals with effectiveness, efficiency and satisfaction
in a specified context of use’’. Here, it is worth noting that
these two latest paraphrased definitions, however differently
particularized, still share exactly the same three virtues which
affect the user’s ability to achieve specified goals.
Since the inception of the first official usability definition,
one might argue that a great plethora of usability attributes
have been taken into consideration regarding the ability to
use particular software products, ranging from monolithic
systems to lightweight web pages. Having said that, based
on the literature search and analysis, in view of usability
attributes that contribute to the quality in use of the desktop
software, the latest study [18] shows that the most frequent
are efficiency, satisfaction, learnability and effectiveness. The
least frequent are understandability and operability, memorability, errors, attractiveness and accessibility.
To collect all necessary data in order to improve the quality
of particular software facets, a variety of usability evaluation methods (UEMs) have been developed and empirically
tested. One of the most recognized UEMs concern the family
of user testing methods [26]–[28], in particular think-aloud
protocol [29]–[31], question-asking protocol [32]–[34], performance measurement [35]–[37], log analysis [38]–[40], eye
tracking [41]–[43], and remote testing [44]–[46]. Secondly,
inspection methods, intended to be used by experts [47],
refers to heuristic evaluation [48]–[50], cognitive walkthrough [51]–[53], perspective-based inspection [54]–[56],
and guideline reviews [57]–[59]. Thirdly, inquiry methods,
designed to gathering subjective data from users, utilize both
quantitative (questionnaires [60]–[62]) and qualitative (interviews [63]–[65] and focus groups [66]–[68]) techniques.
Furthermore, some authors also distinguish analytical modelling methods such as cognitive task analysis [69]–[71], task
environment analysis [72]–[74] and GOMS analysis (Goals,
Operators, Methods and Selection rules) [75]–[77].
Regarding the context of this study, Zhang and Adipat
(2005) propose a generic framework for conducting usability
tests for mobile applications through discussing existing
methodologies and usability attributes [78]. As challenges,
they point to the unique features of mobile devices and
wireless networks which influence the usability of mobile
applications, including mobile context, multimodality, connectivity, small screen size, different display resolutions,
limited processing capability and power, and restrictive data
entry methods. In the case of research methodologies for
usability testing, they point to controlled laboratory experiments and field studies. While former limitations are ignorance of the mobile context and the preservation of reliable
VOLUME 8, 2020
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
network conditions and other environmental factors, then
later, the lack of sufficient control over participants in a study,
and dealing with issues such as the selection of environmental conditions, evaluation performance, data collection and
condition control. They also identify nine generic usability
attributes: learnability (ease of use), efficiency, memorability,
errors, user satisfaction, effectiveness, simplicity, comprehensibility (readability) and learning performance.
Hussain and Kutar (2009) introduce a hierarchical GQM
(Goal Question Metric) model to evaluate mobile usability [79]. On the top level, they place three quality characteristics: effectiveness, efficiency and satisfaction. On the middle
level, six guidelines are conceptualized: simplicity, accuracy,
time taken, features, safety and attractiveness. Eventually,
on the bottom, there is a mapping between questions and
metrics, which enables the collection of quantitative data in
order to evaluate usability.
Kronbauer et al. (2012) propose a hybrid model for the
evaluation of smartphone application usability [80]. In this
study, the hybrid approach blends two methods for data
capture, namely, Logging and ESM (Experience Sampling
Method). The first one is based on data collection related to
user interaction with an application. Using sensors available
in smartphones for contextual data collection, such as luminosity intensity and the device’s position, allows the performance of statistical analysis regarding usability. The second
one is based on the collection of users’ feelings towards a
specific product through questions. These two methods are
respectively used to measure efficiency, effectiveness and
satisfaction.
Harrison et al. (2013) developed the PACMAD (People
At the Centre of Mobile Application Development) usability
model, which identifies three major dimensions affecting the
overall usability of a mobile application: the user, the task
and the context of use [81]. However, the last one plays a
crucial role, as an application may be used in multiple and
very different contexts (e.g. environment, physical location,
user’s state or activity performed). The model encompasses
seven attributes, which together reflect the usability of an
application: effectiveness, efficiency, satisfaction, learnability, memorability, errors and cognitive load. In some studies
the model has been adopted to set up testing and evaluation
frameworks [82], [83]. The novelty of the model concerns
cognitive load as a new usability attribute. The authors claim
that it can be observed that users of mobile applications
often perform additional tasks, such as walking, while using
the mobile device. For this reason, these additional tasks
impact the user’s performance, arguing by example of a
walking user who in the same time is texting a message
which reduces walking speed as s/he is concentrating on
typing (sending) the message. More recently, cognitive load
has been acknowledged [84], or disregarded [85], as one of
the usability notions.
Actual usability, located in the frames of the quality-in-use
model by Lew and Olsina (2013), comprises effectiveness,
efficiency, learnability in use, and communicability [86].
VOLUME 8, 2020
They also emphasize the difference between the context of
mobile applications and traditional, desktop or web applications. The context does not only concern hardware limitations
(e.g. size of the screen), but also other factors, such as: user
activity, day/time of day, location, user profile, device and
network performance.
Obviously, there are many more usability models, individually applicable to particular domains, such as mobile
banking [87], or healthcare [88]; however, they were excluded
from the discussion due to their specific attributes, classified
as superior with respect to the others.
III. RESEARCH METHODOLOGY
A systematic literature review (SLR) in its nature differs from
traditional narrative reviews by adopting a replicable, scientific and transparent process methodology. By design, it aims
to reduce cognitive bias by providing an audit trail of the
associated assumptions and procedures, reviewers decisions
and conclusions on the one hand, and by identifying and
documenting key scientific contributions to a field or question
on the other.
In order to provide a body of knowledge on the usability
of mobile applications, we performed a systematic literature
review by adopting and adapting the approach provided by
Kitchenham and Charters [89], [90], since a large majority of
the reported SLRs in software engineering has been carried
out in respect to their guidelines [91].
According to the research design employed, this study
consists of three steps, performed in a fixed sequence. Interdependency is revealed in the one-way output/input relations.
Step 1 in the research methodology involves defining the
research questions and the review protocol, which encompasses the data source and search strategy, the inclusion and
exclusion criteria and the definition of the search string. The
outcome of this step is described in Section 4. Step 2 in the
research methodology involves executing the search string
carried out on the database engine. Next, the obtained results
are extracted and further processed. The outcome of this step
is given in Section 5. Step 3 in the research methodology
involves reviewing, analysing and reporting each data record,
in order to consequently find and document answers for a
defined set of the research questions. The outcome of this step
is described in Section 6.
IV. SYSTEMATIC LITERATURE REVIEW DEFINITION
A. RESEARCH QUESTIONS DEFINITION
Investigating the gap in usability between desktop and mobile
settings, the following three questions arose:
RQ1. How has usability for mobile applications been
defined?
RQ2. What are the usability attributes for mobile applications?
RQ3. How have usability attributes for mobile applications
been defined, and which measures and evaluation methods
have been used?
55565
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
TABLE 1. The general search query criteria.
TABLE 3. The exclusion criteria (EXCLUDE) to the subject area
(SUBJAREA).
TABLE 2. The inclusion criteria (LIMIT-TO) to the subject area (SUBJAREA).
These three interrogative statements provide the overall
framework for conducting this study, by giving direction and
setting up boundaries.
B. DATA SOURCE AND SEARCH STRATEGY
In line with the research methodology, step 1 involves
a systematic search of the scientific literature on the
topic of mobile application usability. Performed on Scopus,
the largest abstract and citation database of peer-reviewed
literature, counting over 71 million records [92], the search
strategy aims at identifying indexed publications. A key issue
when formulating a search strategy is to define the period of
time to set up time boundaries. Being in our interest to obtain
reliable and concise answers to the questions, we determined
the closing date in December 2018.
TABLE 4. The inclusion criteria (LIMIT-TO) for the document
type (DOCTYPE).
TABLE 5. The inclusion (LIMIT-TO) and exclusion (EXCLUDE) criteria for
the language.
C. SEARCH QUERY DEFINITION
The search query was defined by the presence of ‘‘usability’’
and the string ‘‘mobile application’’ in titles, abstracts and
keywords. These unique and specific terms, joined together
in that order and in the extent of such meta-data, embody the
authors’ common declaration of their research objectives and
the adopted context of their performed studies. The summary,
in terms of the search query construct, is given in Table 1.
D. INCLUSION AND EXCLUSION CRITERIA
In accordance with our research objective and questions,
the first applied inclusion criterion relates to the subject
area, which alternatively includes: computer science, engineering, mathematics, social sciences, or decision sciences.
Table 2 presents the summary of the search query construct
in this scope.
In this study, usability is considered in the context of
software, which is a concern of computer science and is also
closely associated with the other abovementioned disciplines.
In this line of thinking, we exclude irrelevant subject areas
(e.g. Medicine, Health Professions, Chemistry and others).
Table 3 depicts the summary of the search query construct in
this scope.
The second inclusion (exclusion) criterion was the document type which alternatively encompasses: conference proceedings, journal articles or book chapters. On the other
55566
hand, we did not take into account conference reviews and
other reviews, which present non-scientific contributions.
Table 4 outlines the summary of the search query in this
scope.
Not all scientists regard conference proceedings as a reliable and valuable source of knowledge. However, from our
point of view, our judgement was not solely based on the
document type, but on scrupulous reading and conscientious
content analysis.
The third inclusion (exclusion) criterion was the language, exclusively limited to English. Therefore, two other
(Portuguese and French) were excluded. Table 5 depicts the
summary of the search query construct in this regard.
English has become the modern lingua franca in the modern world. The major international standardization bodies
publish norms and standards in English, and communication
channels between experts and communities follow the same
rule as well.
V. SEARCH EXECUTION
A. SEARCH AND SELECTION
In the first run, the search query (Table 1) produced
1,615 document results. To this volume, the inclusion
VOLUME 8, 2020
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
FIGURE 1. The distribution of the number of publications per year.
and exclusion criteria were applied, defined respectively
in Tables 2–5. The search strings, given in all these tables,
were eventually combined by the relevant Boolean operators.
The final search query construct, which entirely fulfils all the
requirements, is given below.
TITLE-ABS-KEY (usability AND ‘‘mobile application’’) AND (LIMIT-TO (SUBJAREA, ‘‘comp’’) OR
LIMIT-TO (SUBJAREA, ‘‘engi’’) OR LIMIT-TO (SUBJAREA, ‘‘math’’) OR LIMIT-TO (SUBJAREA, ‘‘soci’’) OR
LIMIT-TO (SUBJAREA, ‘‘deci’’) OR EXCLUDE (SUBJAREA, ‘‘medi’’) OR EXCLUDE (SUBJAREA, ‘‘heal’’) OR
EXCLUDE (SUBJAREA, ‘‘ceng’’) OR EXCLUDE (SUBJAREA, ‘‘envi’’) OR EXCLUDE (SUBJAREA, ‘‘phys’’) OR
EXCLUDE (SUBJAREA, ‘‘mate’’) OR EXCLUDE (SUBJAREA, ‘‘bioc’’) OR EXCLUDE (SUBJAREA, ‘‘ener’’)
OR EXCLUDE (SUBJAREA, ‘‘psyc’’) OR EXCLUDE
(SUBJAREA, ‘‘arts’’) OR EXCLUDE (SUBJAREA, ‘‘eart’’)
OR EXCLUDE (SUBJAREA, ‘‘nurs’’) OR EXCLUDE
(SUBJAREA, ‘‘chem’’) OR EXCLUDE (SUBJAREA,
‘‘neur’’) OR EXCLUDE (SUBJAREA, ‘‘econ’’) OR
EXCLUDE (SUBJAREA, ‘‘agri’’) OR EXCLUDE (SUBJAREA, ‘‘immu’’) OR EXCLUDE (SUBJAREA, ‘‘phar’’))
AND (LIMIT-TO (DOCTYPE, ‘‘cp’’) OR LIMIT-TO (DOCTYPE, ‘‘ar’’) OR LIMIT-TO (DOCTYPE, ‘‘ch’’)) AND
(EXCLUDE (PUBYEAR, 2019)) AND (LIMIT-TO (LANGUAGE, ‘‘English’’)) AND (EXCLUDE (LANGUAGE,
‘‘Portuguese’’) OR EXCLUDE (LANGUAGE, ‘‘French’’))
The results summary was checked in order to verify
whether all the criteria were successfully applied. In total,
the final search query eventually produced 887 documents,
published between 2001 and 2018. The details of the volume
data are as follows, while the numbers in brackets indicate the
total number of publications: (a) published in English (887),
VOLUME 8, 2020
(b) the subject area is from: computer science (803), decision sciences (40), engineering (198), mathematics (197)
and social sciences (103), and (c) the document type is:
conference proceedings (666), journal articles (196) or book
chapters (25). The peak year is 2017 (140), followed by
the years 2015 (110), 2018 (104) and 2016 (101), with an
average of 74 documents published annually between 20082018 (Figure 1).
The distribution of the number of publications increases in
linear. However, in 2018 a fall was observed in comparison to
the previous year, but still above the year 2016. The majority
of documents were published by Springer in Lecture Notes
in Computer Science, including sub-series Lecture Notes
in Artificial Intelligence and Lecture Notes in Bioinformatics (148), while the largest contributor among journals is
the Journal of Telecommunication, Electronic and Computer
Engineering (12). The top three countries, the USA (136),
Germany (81) and Malaysia (66), accounted for over 31% of
the countries the authors were affiliated to.
B. DATA EXTRACTION
Having imported the reference data (authors, document
title, year, and digital object identifier) to an external
spreadsheet, we systematically searched for each record
in full-text databases hosted by particular publishers and
indicated as the source of the document. From the list
of 887 records, in total 790 (89%) documents were fully
available, while using a HAN system licensed account.
To extract the data, three independent reviewing procedures
were prepared and executed, respectively for each research
question.
In the first run, each available document was screened
with the aim to identify and recognize a usability definition
55567
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
TABLE 6. The list of adopted usability definitions for mobile settings.
TABLE 7. The shares of adopted usability definitions for mobile settings.
referenced by the author(s). The document was classified as
relevant if: (a) usability, as a term, was explicitly defined
and (b) correctly referenced. If the authors provided more
than one definition and did not indicate a particular one as
valid, then the first one given was assumed to be adopted.
Eventually, 66 (8%) documents were classified as relevant as
the input for analysis, with the aim of formulating an answer
to the first research question.
In the second run, each document was screened again to
determine the overall quality and its relevance. A document
was classified as relevant if: (a) the subject of the research
was addressed to the usability of mobile applications, and
(b) was not biased by a context of the research, such as:
(i) application type or (ii) user-specific properties, such as:
age, occupation, sex or (iii) system-specific support features,
like visually impaired or disability. The review of the list
produced 53 (7%) documents as relevant as the input for
analysis with the aim of formulating an answer to the second
research question.
In the third run, the above list was reviewed and examined
again with the aim of extracting attribute definitions, measures and UEMs. The document was classified as relevant
if: (a) usability attributes being the subject of the study were
explicitly defined, whereas a measure was valid if it captures
the quantitative data which accurately describes one particular usability attribute. Ultimately, 39 (5%) documents were
classified as relevant as the input for analysis with the aim of
formulating an answer to the third research question.
VI. DATA ANALYSIS
This section addresses the analysis of the data extracted
from the studies in accordance with the three defined
55568
research questions. We used a qualitative content analysis, which focuses on the characteristics of language
as a communication channel, with attention to the specific subjects, narrowed and directed by particular research
questions.
RQ1. How has usability for mobile applications been
defined?
To this day, none of the authors have introduced any
formal definition of usability associated with an application (system) running on a mobile device. Therefore, all identified and recognized definitions have been
adopted from the existing general norms, standards and
definitions.
The great majority of authors (88%) have defined usability
solely in terms of the ISO 9241-11 norm, while others have
also made reference to ISO 25010 (4,5%) and ISO 9126 (3%)
norms, as well as to the IEEE Glossary (1,5%), the Nielsen
(1,5%) and Bevan (1,5%) definitions. Table 6 includes the
full text of these six definitions, whereas Table 7 depicts
findings of the shares of adopted usability definitions for
mobile settings.
RQ2. What are the usability attributes for mobile applications?
In total, 75 usability attributes were identified and analysed. Among them, the most frequent are efficiency (70%),
satisfaction (66%) and effectiveness (58%). Less frequent
are learnability (45%), memorability (23%), cognitive load
(19%) and errors (17%). The last two concern simplicity (13%) and ease of use (9%). The remaining attributes
occurred four times or less. Table 8 outlines the details in
this regard (the attributes which occurred only once are not
included).
VOLUME 8, 2020
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
TABLE 8. The list of adopted usability attributes for mobile settings.
RQ3. How have usability attributes for mobile applications been defined, and which measures and evaluation
methods have been used?
The foremost attribute, efficiency is the ability of a user
to complete a task with speed and accuracy. Efficiency is
measured in a number of ways, such as the duration spent
on each screen, the duration to complete a given task (a set of
tasks), and the user’s error rate. Two evaluation methods are
used: controlled observation and survey.
Satisfaction is a user’s perceived level of comfort and
pleasure, or a user’s perceived level of fulfilment of his
expectations and needs. Satisfaction is measured only by
using survey, with predefined statements with the Likert-scale
rating system, which is typically used to capture a user’s
intangible attitude towards an application.
Effectiveness is the ability of a user to complete a task in a
given context. It is measured by the number of successfully
completed tasks, the number of steps required to complete a
task, the number of double taps unrelated to the operation of
an application, and the number of times that a back button is
used by the mobile device (not the application).
Learnability is defined twofold. First-time learnability
refers to the degree of ease with which a user can interact with a newly-encountered system without getting guidance or referring to documentation. It is measured by the
number of attempts to solve a task, the number of assists
VOLUME 8, 2020
during performing a task, and the number of errors performed
by a user. Learnability over time, on the contrary, is the
capacity of a user to achieve proficiency with an application.
Typically, a user’s performance during a series of tasks is
observed to measure how long it takes these participants
to reach a pre-specified level of proficiency. Similarly to
effectiveness, two evaluation methods are used: controlled
observation and survey.
Memorability is the degree of ease with which a user
can remember how to use an application effectively. It is
measured by asking users to perform a series of tasks after
having become proficient with the use of the application,
and afterwards asking them to perform similar tasks after a
period time of inactivity. To determine how memorable the
application was, a comparison is made between the two sets
of results. In this case, the eye-tracking technique is also used
as the method to collect gaze data which is further used to
evaluate usability.
Cognitive load refers to the amount of mental activity
imposed on a user’s working memory during application
usage. Cognitive load theory differentiates cognitive load into
three types: extraneous, intrinsic and germane. Firstly, extraneous cognitive load refers to instructional and presentation
schemas, caused by the mental activities and elements that
do not directly support application usage. Secondly, intrinsic cognitive load refers to the task complexity, caused by
55569
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
TABLE 9. The top most frequent usability attributes, their measures and associated usability evaluation methods (UEMs) for mobile settings.
the number of elements in a task and the degree to which
these elements are related to each other. Thirdly, germane
cognitive load refers to the amount of mental effort used to
form schemas and actively integrate new information with
prior knowledge during application usage. In the practice of
cognitive load measurement, instruments such as a subjective
rating scale, a thinking aloud dual task protocol or eye tracking are in common use.
Errors refers to the amount and type of errors which occur
during task performance by a user. On the other hand, it is
the ability of an application to recover from occurred errors.
Both these definitions also respectively reflect the measures
of attribute.
Simplicity is the degree of being easy to understand or being uncomplicated in form or design, described by
such characteristics as the number of menu levels, the number
of performed gestures to reach a destination object, and
the duration of searching a button to perform a specific
function. On the other hand, simplicity is the level of comfort
with which a user is able to complete a task, measured by
predefined statements with the Likert-scale rating.
Ease of use is the perceived level of the user’s effort related
to usage of the application. The survey instrument is used
to collect data from users on perceptions concerning their
experienced interaction with the application.
Table 9 presents a summary in which each attribute is
associated with the valid measures, along with the usability
55570
evaluation methods used to collect the necessary data to
improve particular software artefacts.
From the variety of available methods, the most frequent
is survey, based on the questionnaire instrument, which has
been used to collect data from a sample of the participants,
as a representation of the population of interest. Controlled
observation of the user while interacting with an application
is the second most frequent method applied to usability evaluation. The remaining three, namely eye-tracking, thinking
aloud and interview, are hardly used and serve as complementary to collect additional data. Table 10 presents the details
showing the number of occurrences of all identified UEMs
applied for particular attributes.
VII. DISCUSSION
Based on the obtained results, we argue that the ISO 9241-11
definition has been widely accepted in a non-changed form,
and since the inception of research on mobile application
usability, has been, de facto, popularized as the standard by
the HCI community. Having said that, it is worth noting that
other definitions are not contrary to each other. Moreover,
they have in common the software capability to interact with
a user, yet emphasize different aspects of his/her proficiency.
On the other hand, usability is always associated with the
product, except for the IEEE Glossary and Bevan definitions,
which focus first and foremost on the user.
VOLUME 8, 2020
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
TABLE 10. The number of occurrences of usability evaluation methods (UEMs) applied to particular usability attributes.
TABLE 11. The percentage of studies concerning usability attributes of mobile applications.
The most frequent attributes originate from the usability
definition adapted to mobile applications. In such a case,
the main usability characteristics are device-agnostic. In other
words, efficiency, satisfaction and effectiveness are valid for
studying the usability of both desktop and mobile applications. In a similar manner, however with minor extensions,
the remaining attributes have been assimilated as well.
By design, cognitive load is related to the mental effort
required by the user to perform tasks using a mobile device.
While it is neither novel, nor high-ranking in usability
research, it has now gained a larger audience due to the
fact that a user’s attention is usually divided among other
simultaneously performed tasks.
If one breaks down usability into two parts, one gets two
nouns: ‘‘use’’ and ‘‘ability’’. According to this line of thinking, the ability to use an application, in particular, means
the ability to learn, memorize, navigate and operate. Besides
this, ease of use seems related to the sense of presence of
these abilities, facilitated by the errorless behaviour of the
application.
If one takes into account the research methods applied
to the problem of usability evaluation, the attributes of the
studied phenomena can be divided into two groups: quantitative and qualitative. However, based on such criteria, every
attempt to formulate distinct groups will have its pros and
cons, because each attribute has been measured depending
on observation and survey. Nevertheless, the existing measures can be unambiguously classified if one still makes a
clear distinction between facts and opinions. In other words,
quantitative-oriented attributes have the advantage of being
clearly definable and objectively measurable, using measures
that are not influenced by the user’s personal judgement.
VOLUME 8, 2020
On the other hand, one can point to user-oriented measures,
and on the contrary, to application-oriented measures. Last
but not least, it appears that existing measures intertwine
user and application performance in one integrated artefact.
It seems obvious that observational data are required to
discover an application’s bottlenecks and general areas for
improvement, thereby optimizing its operational capabilities
by reducing the time and effort involved in its usage.
To collect quantitative and qualitative data, questionnaires
and controlled observation, respectively, have been typically
applied, occasionally supported by eye-tracking, thinking
aloud and interview techniques (see Table 10). In order to
obtain numerical measures, a retrospective audio/video analysis is performed, while in some studies, third-party tools
were installed which log all user interaction with an application with the date and time of the event, including the buttons
that they chose, the gestures that were made and the functions that were recalled. After completing the task scenario,
a user was asked to rank their agreement (disagreement) with
predefined statements on a Likert scale or other rating scale.
In comparison with the results obtained from studies with
similar objectives, conducted by Coursaris and Kim [93]
and Harrison et al. [81], our findings are consistent in the
extent of the top three attributes, which concern efficiency,
satisfaction and effectiveness (see Table 11). An increased
interest in learnability and memorability can also be noticed,
while errors and cognitive load are less appreciated. While
simplicity and ease of use have not been indicated before,
and being complementary, are neither novel nor visionary.
The rest of the attributes seem to be extensions of existing
ones, however, unbigoted by usability, they correspond to
both explicit and implicit application properties. On the other
55571
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
hand, if one explores the user’s preferences instead of his/her
ability to use an application, the results refer more to the user
experience domain, yet less to usability.
A. LIMITATIONS
Although this study contributes to the field of humancomputer interaction, certain limitations exist within the
research design.
Firstly, one of the major limitations is that only one data
source was involved. However, the indexation process covers
varied sources of scientific content, ranging from conference
proceedings to journal articles, which are reviewed each year
to ensure quality standards are maintained.
Secondly, inclusion and exclusion criteria, as they permit
documents only published in English, may be a subject of
critique. In this manner, our intention was not to disregard
other foreign languages, but was determined by the global
status of the English language in modern science.
Thirdly, regarding the search query construct, including
only the terms ‘‘usability’’ and ‘‘mobile applications’’ might
have excluded potentially relevant documents concerning
other related studies (i.e. user experience or design thinking)
from the scope of the search results, and later, from the analysis, though one should bear in mind that evidence is defined
as the ‘‘synthesis of the best quality scientific studies on a
specific topic (. . . )’’ [90]. Nevertheless, by design, the goal
of this study was to provide an evidence-based contribution
on the usability of mobile applications, thus this limitation
is simply the result of the application of SLR methodology
principals.
Ultimately, the applied reviewing procedures might be seen
as too strict or hard to follow. However, we assumed to identify only such attributes and measures which can be replicated
in any extent, and arbitrarily extended if necessary.
B. FUTURE RESEARCH
The obtained results uncover the trend in time of producing ‘‘new’’ attributes which unnecessarily contribute to the
usability of mobile applications. And yet, one might try to
assume that there are still some vulnerable properties laying
beneath their quality of use. Nevertheless, one of the issues
which unfolds definitely concerns how to consolidate the
existing attributes into one compact model which reflects all
identified and relevant usability facets.
Moreover, in addressing the topic of usability evaluation,
there is still little known about the simulation methods which
might admittedly replace both experts and users in application
evaluation in view of selecting its properties and behaviour.
On the one hand, software vendors will benefit by reducing
engaged time and effort, while on the other, the users will take
advantage of the better application in daily usage. Therefore,
a second suggestion is practical in nature, and relates to the
matter of developing a tool in which implemented methods to
automate and simulate the users’ interaction with the application may reduce the participation of both experts and users.
55572
Currently, we are developing a usability inspection method
which aims to fully automate application testing due to
evaluating its compliance with efficiency and effectiveness
requirements and to detecting bugs and errors. The latest
application version enables usability engineers to perform
video analysis annotation, which aims at measuring the duration of actions on a time scale, embedded on a ribbon within
a visual diagram editor. Moreover, it also allows the tasks on
the layers to be graphically decomposed into smaller units
(subtasks). The first results are promising, showing that if
we isolate user activities from application responses, then
it allows us to analyse and evaluate both the user and the
application separately, which adequately produces a reliable
outcome for interface designers and developers.
VIII. CONCLUSION
The results of the systematic literature review show that
the ISO 9241-11 definition has been adapted by the majority as the baseline in the studies of mobile application
usability. In total, 75 attributes were distinguished in the
body of 790 documents, indexed by the Scopus database.
The most frequent are efficiency (70%), satisfaction (66%)
and effectiveness (58%), which originate from the above
definition. Afterwards, the less frequent are learnability
(45%), memorability (23%), cognitive load (19%) and
errors (17%). The last two concern simplicity (13%) and
easy of use (9%). The remaining attributes occurred four
times or less.
We observed that 91% of documents lack a usability definition. While not providing a formal and legitimate definition
even seems to be acceptable in some circumstances, measuring and explaining the facets of the phenomena exclusively
on the grounds of common sense might be questionable.
As a matter of fact, over 90% of documents did not meet the
inclusion criteria for analysis, although some report valuable
results. On the other hand, a small number of the remaining
documents zealously ‘‘produced’’ new attributes with associated ‘‘unique’’ measures, which usually concerned unobservable properties, measured by a set of explicit statements.
Many of these constructs lack theoretical foundations and
empirical evidence to expose their worthiness.
To complicate the matter even further, most of the introduced attributes have focused on user beliefs, emotions, preferences, perceptions, physical and psychological responses,
behaviours and accomplishments that occur before, during
and after application use, which concern, in particular or as
a whole, dimensions of user experience [21]. Such a combination of objective and subjective assessments eventually
produces an outcome which refers to neither application
usability nor user experience. This reflects an ignorance to
methodological rigour, which negatively affects the validity
of the results.
REFERENCES
[1] M. Rouse and I. Wigmore. (2013). Definition. Mobile App. [Online].
Available: https://whatis.techtarget.com/definition/mobile-app
VOLUME 8, 2020
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
[2] Statista. (2019). Global Mobile Device Market Share in 2017 and
2018, by Device Type. [Online]. Available: https://www.statista.com/
statistics/183530/worldwide-market-share-mobile-device/
[3] Statista. (2019). Smartphone User Penetration as Percentage of Total
Global Population From 2014 to 2021. [Online]. Available: https://
www.statista.com/statistics/203734/global-smartphone-penetration-percapita-since-2005/
[4] H. M. Salman, W. F. Wan Ahmad, and S. Sulaiman, ‘‘Usability evaluation
of the smartphone user interface in supporting elderly users from experts’
perspective,’’ IEEE Access, vol. 6, pp. 22578–22591, 2018.
[5] M. Sikorski, ‘‘Beyond product usability: User satisfaction and quality
management,’’ in Proc. CHI Extended Abstr. Hum. Factors Comput.
Syst. (CHI). New York, NY, USA: ACM, 2000, pp. 61–62.
[6] H. Feng, T. Hoegler, and W. Stucky, ‘‘Exploring the critical success
factors for mobile commerce,’’ in Proc. Int. Conf. Mobile Bus., Jun. 2006,
p. 40.
[7] E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, and
C. Pages-Arevalo, ‘‘Validation of navigation guidelines for improving
usability in the mobile Web,’’ Comput. Standards Interfaces, vol. 52,
pp. 51–62, May 2017.
[8] N. Tractinsky, ‘‘The usability construct: A dead end?’’ Hum.-Comput.
Interact., vol. 33, no. 2, pp. 131–177, Mar. 2018.
[9] B. Shackel, ‘‘Usability—Context, framework, definition, design and evaluation,’’ Interacting Comput., vol. 21, nos. 5–6, pp. 339–346, Dec. 2009.
[10] J. Kjeldskov and J. Stage, ‘‘New techniques for usability evaluation
of mobile systems,’’ Int. J. Hum.-Comput. Stud., vol. 60, nos. 5–6,
pp. 599–620, May 2004.
[11] K. Gulzar, J. Sang, M. Ramzan, and M. Kashif, ‘‘Fuzzy approach to
prioritize usability requirements conflicts: An experimental evaluation,’’
IEEE Access, vol. 5, pp. 13570–13577, 2017.
[12] P. Weichbroth, ‘‘Delivering usability in IT products: Empirical lessons
from the field,’’ Int. J. Softw. Eng. Knowl. Eng., vol. 28, no. 7,
pp. 1027–1045, Jul. 2018.
[13] M. S. Bashir and A. Farooq, ‘‘EUHSA: Extending usability heuristics for
smartphone application,’’ IEEE Access, vol. 7, pp. 100838–100859, 2019.
[14] R. Parente Da Costa, E. D. Canedo, R. T. De Sousa, R. De Oliveira
Albuquerque, and L. J. Garcia Villalba, ‘‘Set of usability heuristics for
quality assessment of mobile applications on smartphones,’’ IEEE Access,
vol. 7, pp. 116145–116161, 2019.
[15] A. Seffah, M. Donyaee, R. B. Kline, and H. K. Padda, ‘‘Usability measurement and metrics: A consolidated model,’’ Softw. Qual. J., vol. 14,
no. 2, pp. 159–178, Jun. 2006.
[16] S. Winter, S. Wagner, and F. Deissenboeck, ‘‘A comprehensive model of
usability,’’ in Proc. IFIP Int. Conf. Eng. Hum.-Comput. Interact. Berlin,
Germany: Springer, 2007, pp. 106–122.
[17] K. Majrashi and M. Hamilton, ‘‘A cross-platform usability measurement
model,’’ Lect. Notes Softw. Eng., vol. 3, no. 2, p. 132, 2015.
[18] P. Weichbroth, ‘‘Usability attributes revisited: A time-framed knowledge
map,’’ in Proc. Federated Conf. Comput. Sci. Inf. Syst., Sep. 2018,
pp. 1005–1008.
[19] Software Enginnering—Product Quality, Standard ISO/IEC 9126:1991,
ISO/IEC, Geneva, Switzerland, 1991.
[20] Ergonomic Requirements for Office Work With Visual Display Terminals (VDTs)—Part 11: Guidance on Usability, Standard ISO 924111:1998(en), ISO, Geneva, Switzerland, 1998. [Online]. Available:
https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-1:v1:en
[21] Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts, ISO 9241-11:2018(en), ISO, Geneva, Switzerland,
2018. [Online]. Available: https://www.iso.org/obp/ui/#iso:std:iso:9241:11:ed-2:v1:en
[22] J. R. Lewis, ‘‘Usability: Lessons learned. . . and yet to be learned,’’ Int.
J. Hum.-Comput. Interact., vol. 30, no. 9, pp. 663–684, Sep. 2014.
[23] D. Quiñones and C. Rusu, ‘‘How to develop usability heuristics:
A systematic literature review,’’ Comput. Standards Interfaces, vol. 53,
pp. 89–122, Aug. 2017.
[24] Systems and Software Engineering—Systems and Software Quality
Requirements and Evaluation (SQuaRE)—System and Software Quality
Models, Standard ISO/IEC 25010:2011, ISO, Geneva, Switzerland, 2011.
[25] Software Engineering—Product Quality—Part 1: Quality Model,
Standard ISO/IEC 9126-1:2001, ISO, Geneva, Switzerland, 2001.
[Online]. Available: https://www.iso.org/standard/22749.html
[26] J. Nielsen, ‘‘Iterative user-interface design,’’ Computer, vol. 26, no. 11,
pp. 32–41, Nov. 1993.
VOLUME 8, 2020
[27] N. Bevan, ‘‘Usability is quality of use,’’ in Advances in Human Factors/Ergonomics, vol. 20. Amsterdam, The Netherlands: Elsevier, 1995,
pp. 349–354.
[28] J. M. C. Bastien, ‘‘Usability testing: A review of some methodological
and technical aspects of the method,’’ Int. J. Med. Inform., vol. 79, no. 4,
pp. e18–e23, Apr. 2010.
[29] T. Boren and J. Ramey, ‘‘Thinking aloud: Reconciling theory and
practice,’’ IEEE Trans. Prof. Commun., vol. 43, no. 3, pp. 261–278,
3rd Quart., 2000.
[30] S. Richardson, R. Mishuris, A. O’Connell, D. Feldstein, R. Hess, P. Smith,
L. McCullagh, T. McGinn, and D. Mann, ‘‘‘Think aloud’ and ‘Near live’
usability testing of two complex clinical decision support tools,’’ Int.
J. Med. Inform., vol. 106, pp. 1–8, Oct. 2017.
[31] G. Deniz and P. O. Durdu, ‘‘A comparison of mobile form controls
for different tasks,’’ Comput. Standards Interfaces, vol. 61, pp. 97–106,
Jan. 2019.
[32] T. Kato, ‘‘What ‘question-asking protocols’ can say about the user interface,’’ Int. J. Man-Mach. Stud., vol. 25, no. 6, pp. 659–673, Dec. 1986.
[33] T. Obata, T. Daimon, and H. Kawashima, ‘‘A cognitive study of invehicle navigation systems: Applying verbal protocol analysis to usability
evaluation,’’ in Proc. Vehicle Navigat. Inf. Syst. Conf. (VNIS), Oct. 1993,
pp. 232–237.
[34] R. Nagpal, D. Mehrotra, and P. K. Bhatia, ‘‘The state of art in
Website usability evaluation methods,’’ in Design Solutions for UserCentric Information Systems. Harrisburg, PA, USA: IGI Global, 2017,
pp. 275–296.
[35] et. al. R. Nagpal, ‘‘FAHP approach to rank educational Websites on
usability,’’ Int. J. Comput. Digit. Syst., vol. 4, no. 4, pp. 251–260,
Oct. 2015.
[36] A. Hinderks, M. Schrepp, F. J. Domínguez Mayo, M. J. Escalona, and
J. Thomaschewski, ‘‘Developing a UX KPI based on the user experience questionnaire,’’ Comput. Standards Interfaces, vol. 65, pp. 38–44,
Jul. 2019.
[37] J.-P. Jain and W.-C. Shen, ‘‘A study of the optimization of app design with
regards to usability and user experience—a case study of SunlineApp,’’ in
Proc. Eng. Innov. Design, 7th Int. Conf. Innov., Commun. Eng. (ICICE),
Hangzhou, China. CRC Press, May 2019, p. 175.
[38] A. Holzinger, ‘‘Usability engineering methods for software developers,’’
Commun. ACM, vol. 48, no. 1, pp. 71–74, Jan. 2005.
[39] L. F. Gonçalves, L. G. Vasconcelos, E. V. Munson, and L. A. Baldochi,
‘‘Supporting adaptation of Web applications to the mobile environment
with automated usability evaluation,’’ in Proc. 31st Annu. ACM Symp.
Appl. Comput. (SAC). New York, NY, USA: ACM, 2016, pp. 787–794.
[40] J. Grigera, A. Garrido, J. M. Rivero, and G. Rossi, ‘‘Automatic detection
of usability smells in Web applications,’’ Int. J. Hum.-Comput. Stud.,
vol. 97, pp. 129–148, Jan. 2017.
[41] J. Falkowska, J. Sobecki, and M. Pietrzak, ‘‘Eye tracking usability testing
enhanced with EEG analysis,’’ in Proc. Int. Conf. Design, User Exper.,
Usability. Cham, Switzerland: Springer, 2016, pp. 399–411.
[42] P. Weichbroth, K. Redlarski, and I. Garnik, ‘‘Eye-tracking Web usability
research,’’ in Proc. Federated Conf. Comput. Sci. Inf. Syst., Oct. 2016,
pp. 1681–1684.
[43] P. Realpe-Muñoz, C. A. Collazos, J. Hurtado, T. Granollers,
J. Muñoz-Arteaga, and J. Velasco-Medina, ‘‘Eye tracking-based
behavioral study of users using e-voting systems,’’ Comput. Standards
Interfaces, vol. 55, pp. 182–195, Jan. 2018.
[44] T. Tullis, S. Fleischman, M. McNulty, C. Cianchette, and M. Bergel,
‘‘An empirical comparison of lab and remote usability testing of Web
sites,’’ in Proc. Usability Professionals Assoc. Conf., 2002, pp. 1–8.
[45] J. Dong, S. L. Martin, J. M. Mullaly, and A. R. Tannenbaum,
‘‘Method, system and program for performing remote usability testing,’’
U.S. Patent 6 526 526, Feb. 25, 2003.
[46] J. Sauer, A. Sonderegger, K. Heyden, J. Biller, J. Klotz, and
A. Uebelbacher, ‘‘Extra-laboratorial usability tests: An empirical comparison of remote and classical field testing with lab testing,’’ Appl.
Ergonom., vol. 74, pp. 85–96, Jan. 2019.
[47] M. J. Scott, F. Spyridonis, and G. Ghinea, ‘‘Designing for designers: Towards the development of accessible ICT products and services
using the VERITAS framework,’’ Comput. Standards Interfaces, vol. 42,
pp. 113–124, Nov. 2015.
[48] J. Díaz, C. Rusu, and C. A. Collazos, ‘‘Experimental validation of a set of
cultural-oriented usability heuristics: E-commerce Websites evaluation,’’
Comput. Standards Interfaces, vol. 50, pp. 160–178, Feb. 2017.
55573
P. Weichbroth: Usability of Mobile Applications: Systematic Literature Study
[49] T. C. Lacerda and C. G. von Wangenheim, ‘‘Systematic literature review
of usability capability/maturity models,’’ Comput. Standards Interfaces,
vol. 55, pp. 95–105, Jan. 2018.
[50] D. Quiñones, C. Rusu, and V. Rusu, ‘‘A methodology to develop usability/user experience heuristics,’’ Comput. Standards Interfaces, vol. 59,
pp. 109–129, Aug. 2018.
[51] I. Connell, A. Blandford, and T. Green, ‘‘CASSM and cognitive walkthrough: Usability issues with ticket vending machines,’’ Behav. Inf.
Technol., vol. 23, no. 5, pp. 307–320, Sep. 2004.
[52] T. Hollingsed and D. G. Novick, ‘‘Usability inspection methods after
15 years of research and practice,’’ in Proc. 25th Annu. ACM Int.
Conf. Design Commun. (SIGDOC). New York, NY, USA: ACM, 2007,
pp. 249–255.
[53] L. Alonso-Virgós, J. P. Espada, and R. G. Crespo, ‘‘Analyzing compliance and application of usability guidelines and recommendations by
Web developers,’’ Comput. Standards Interfaces, vol. 64, pp. 117–132,
May 2019.
[54] Z. Zhang, V. Basili, and B. Shneiderman, ‘‘Perspective-based usability
inspection: An empirical validation of efficacy,’’ ...
Purchase answer to see full
attachment