Creative Education
2011. Vol.2, No.3, 208-219
Copyright © 2011 SciRes. DOI:10.4236/ce.2011.23029
Findings from European Benchmarking Exercises on
E-Learning: Value and Impact
Ebba Ossiannilsson
Department of Industrial Engineering and Management, Oulu University, Oulu, Finland.
Email: Ebba.Ossiannilsson@oulu.fi
Received February 22nd, 2011; revised March 10th, 2011; accepted March 20th, 2011.
Benchmarking is becoming a more and more commonly used method for quality assurance and enhance- ment
in higher education today. The aim of this study was to explore the impact and value of benchmarking e-learning
in universities. Two European benchmarking initiatives, E-xcellence+ through the European Associa- tion of
Distance Teaching Universities [EADTU] and an e-learning benchmarking exercise through the Euro- pean
Centre for Strategic Management of Universities [ESMU], were studied using an exploratory multiple case study
method and cross-case analysis of the findings. The findings indicated the value and impact of a variety of as-
pects of benchmarking e-learning on different levels, and stressed the need for a holistic and contextual ap-
proach as both benchmarking and e-learning are complex and comprehensive phenomena. This study also shows
that benchmarking, in line with national and international quality boards/agencies, is a powerful tool for making
improvements to teaching and learning in higher education.
Keywords: Benchmarking, Critical Success Factors, E-Learning, Quality Enhancement
Introduction
This paper focuses on research questions concerning the
value and impact of benchmarking e-learning in higher educa-
tion, which until now has remained a fairly unexplored area.
This study uses as points for departure the experiences of two
European benchmarking initiatives; E-xcellence+, which was
carried out by the European Association of Distance Teaching
Universities (EADTU) (Ossiannilsson & Landgren, in press;
Ubachs, 2009; Williams & Brown, 2009) and the e-learning
benchmarking exercise, carried out by the European Centre for
Strategic Management of Universities (ESMU) (Comba et al.,
2010; Ossiannilsson & Landgren, in press; Williams & Rotheram,
2010). Moriarty (2008) argued that although the literature on
benchmarking is overwhelming, research and evidence on the
value and impact of benchmarking is still missing. This re-
search aims to contribute to the formation of a deeper under-
standing and knowledge of the value and impact of bench-
marking e-learning in higher education. This research does not
claim to focus on benchmarks as such, and neither has this
study been conducted using a special method for analysing
impact.
Major changes are taking place in European higher education
today, and quality in education and research is the key to sup-
porting development and innovation. In this context, enhancing
the performance of universities and modernising university
management must be on the agenda for all university leaders
and decision-makers in Europe (Bates, 2010). A clear under-
standing and transparency of the modes of operations and
processes with the aim of continuous improvement is needed.
Universities are facing new challenges in the twenty-first cen-
tury, as they are required to be competitive not just in terms of
their educational, social, managerial and technological aspects,
but they are also called to work globally as drivers for innova-
tion and to contribute to sustainable development. In this envi-
ronment, in parallel with institutional, regional, national and
international demands and challenges, improving the perform-
ance and quality of universities has become more important
than ever. At the same time, respect for individual students and
accountability for the use of funding has to be taken into ac-
count, in addition to contributing to economic growth (Euro-
pean Commission, 2009; van Vught, 2008a, 2008b). Enhanced
quality and increased transparency are strong driving forces
behind competition in education and research. Ossiannilsson
and Landgren (in press) showed in their study on three interna-
tional benchmarking projects that the identification of success
factors for e-learning and its implementation in the educational
arena is of the utmost significance.
Benchmarking has become a more commonly used method
for quality assurance and enhancement in higher education, as it
deals with identifying gaps and making changes, but also with
improvement and successful implementation of new procedures
and schemes (Bacsich, 2009a, 2009b; Ossiannilsson, 2010,
2011). In a recent study, Ossiannilsson (in press) advocated the
use of benchmarking as a method and a powerful tool for mak-
ing improvements to teaching and learning (concerning e-
learning/blended learning) in the twenty-first century, and to
support improved governance and management in higher edu-
cation. The study showed that further research has to be done
according to holistic perspectives in order to answer questions
on the value and impact of benchmarking e-learning in higher
education. These themes include: why is benchmarking used,
what is benchmarking e-learning? When is benchmarking used?
etc. Thus, this study is based on multiple case studies, together
with cross-case analyses, in order to contribute to the existing
knowledge and understanding of the value and impact of ben-
chmarking e-learning in higher education.
In the following section, the concepts of benchmarking and
E. OSSIANNILSSON 209
e-learning will be further elaborated in order to set the scene.
After this, the projects will be described briefly, and this will be
followed by a summary of the findings from the multiple case
studies and the cross-case analysis, with the focus on value and
impact. Finally, some reflections and conclusions will be pre-
sented.
The Concept of Benchmarking
The concept of benchmarking has its origins in the manage-
ment and business context. Nevertheless, it started in the pri-
vate sector, and was introduced by the Xerox Corporation in the
ways in which they conducted their successful development.
Looking what others were doing, and especially what their
competitors were doing, led Xerox to make changes internally
in order to improve their quality and processes and to enable
the company to gradually regain its market position. Their
original definition of benchmarking was:
...a process for improving performance by constantly iden-
tifying, understanding and adapting best practices and pro-
cesses followed inside and outside the company and imple-
menting the results. The main emphasis of benchmarking is on
improving a given business operation or a process by exploit-
ing best practices’, not on best performanc’. Simply put,
benchmarking means comparing ones organization [sic] or a
part of it with that of the other companies.1
Camp one of the most frequently cited scholars with regard
to benchmarking (cited in Bacsich, 2009a; Hämäläinen, Jessen,
Kaartinen-Koutaniemi, & Kristoffersen, 2003; Johnson &
Seborg, 2007; Ossiannilsson, in press; ReVica, 2009; van
Vught, 2008a), explored the benchmarking process using five
continuous steps: determining what to benchmark, forming a
benchmarking team, identifying benchmarking partners, col-
lecting and analysing benchmarking information and finally
taking action. The steps are illustrated in Figure 1.
Benchmarking has developed into an essential tool for or-
ganizations, and is regarded as a vital component of good man-
agment practice. It is internationally respected, not just in busi-
sses, organisations and management, where the concept came
from, but also now in education and higher education (Moriarty,
2008). Moriarty and Smallman, (2009: p. 484) stated that “the
locus of benchmarking lies between the current and desirable
states of affairs and contributes to the transformation processes
that realise these improvements”. Furthermore, Moriarty (2008)
discussed the fact that although the literature on benchmarking
is overwhelming, research and evidence on the value and im-
pact of benchmarking is still missing. The criticisms of bench-
marking are, according to Moriarty (2008), based mainly on
lack of information, difficulties with implementation and a lack
of theory. He stressed that benchmarking may require another
definition, and that benchmarking is intended to be a means
towards the end of achieving a more desirable organisational
state of affairs. Benchmarking may identify the changes which
are necessary to achieve that end. The concept of change seems
to be inherent in benchmarking. Benchmarking is, however, not
just about change, but about improvement, or as Harrington
(1995) put it, “all improvements is [sic] change, but not all
change is improvement” (cited in Moriarty, 2008: p. 29). Mo-
Figure 1.
Illustration inspired by the benchmarking process described by Camp,
1989.
riarty continued by stating that benchmarking is not just about
making changes, as it is more about identification and success-
ful implementation. Therefore, he suggested a provisional defi-
nition: “Benchmarking is an exemplar-driven teleological proc-
ess operating within an organization [sic] with the objective of
intentionally changing an existing state of affairs into a supe-
rior state of affairs.” (Moriarty, 2008: p. 30).
Benchmarking is a rather new phenomenon in higher educa-
tion, especially with regard to e-learning (Bacsich, 2009a; Os-
siannilsson, in press; Ubachs, 2009; van Vught, 2008a, 2008b).
Quality assurance, quality indicators, benchmarks and critical
success factors for e-learning have neither been taken seriously
nor incorporated into regular quality assurance procedures in
higher education. These concepts have not been conceptualised,
and are even sometimes taken as being synonymous. The con-
cept of quality in e-learning has been discussed, considered and
managed in a very disconnected manner, and it has not been
embedded in learning and quality contexts (National Agency of
Higher Education in Sweden (NAHE), 2008; Soinila & Stalter,
2010). Ossiannilsson also showed in earlier studies (2010, in
press, 2011) that there is a lack of experience of the value and
impact of benchmarking in higher education.
The ESMU has worked with benchmarking in several pro-
jects in different areas within higher education. The definition
of benchmarking used by ESMU is:
Benchmarking is an internal organizational [sic] process
which aims to improve the organizations performance by
learning about possible improvements of its primary and/or
support processes by looking at these processes in other, bet-
ter-performing organizations (van Vught et al., 2008: p. 16).
As shown in the definitions above, benchmarking is very
much a process designed to enhance quality, to identify gaps
and to bring about the implementation of changes. Benchmark-
ing initiatives are often conducted as self-evaluations, including
systematic data and gathering information from predefined
benchmarks, as well as formulating roadmaps. The goal of
1Xerox Corporation (1979). Xerox. The Benchmarking Story.URL (last
checked 10 June 2011) http://www.icmrindia.org/free%20resources/case-
studies/xerox-benchmarking-2.htm
E. OSSIANNILSSON
210
benchmarking is to formulate, together with other colleagues,
strengths and challenges for the purpose of improvement (Os-
siannilsson, in press; van Vught et al., 2008b). The benefits of
benchmarking were expressed by the ESMU in 10 statements:
self-assess institution, better understand the process, measure
and compare, discover new ideas, obtain data to support deci-
sion-making, set targets for improvement, strengthen institu-
tional identity, enhance reputation, respond to national per-
formance indicators and benchmarks [and] set new standards
for the sector” (van Vught et al., 2008b).
Participating in a benchmarking process can potentially lead
to change in the area being investigated. In addition, an in-
creased awareness, both individual and collective, of the or-
ganisation itself occurs as a result of participation, which can be
considered as a direct and substantial value (Ossiannilsson, in
press, 2011). This awareness can often lead to reflections. Re-
flections are crucial and can even be a method for organisa-
tional change (Højrup, 2004; Ossiannilsson & Landgren, in
press).
The Concept of E-Learning
Information, Communication and Technology (ICT) has be-
come embedded in the social and economic sectors, and ought
to be similarly embedded in the education and training systems.
However, new and more efficient ways of operating, supporting
pedagogical and organisational innovation need to be found. In
the twenty-first century and with the new millennium learners
existing in a digital world, the e-phenomenon has to be embed-
ded in all learning and educational activities in order to push
the boundaries in daily life, in a global sustainable environment
(Bates, 2010; Bonk, 2010; Conole, 2010; Ehlers & Pawlowski,
2006). Hence, several scholars have argued that there is no
longer a need for definitions, as e-learning has implications on
a vast number of fields (Ehlers & Schneckenberg, 2010; John-
son, Smith, Willis, & Haywood, 2011; O’Reilly & Batelle,
2009). However, research has shown that respect has to be
given to critical success factors in educational environments in
order to succeed in the field of e-learning and to benefit learn-
ers, teachers and management (Ossiannilsson & Landgren, in
press). McLoughlin and Lee (2008) explored success factors
under the headings of personalisation, participation and produc-
tivity. Bonk (2009) went a step further, and expressed this con-
cept as ubiquitous learning (u-learning), with the focus in the
‘you perspective’ on personalisation and the learner’s rights and
responsibilities. The EADTU emphasises within their E-excel-
lence Associates label four success factors regarding e-learning,
namely accessibility, interactivity, flexibility and personalisa-
tion (Ubachs, 2009). Connectivism is also considered as an-
other concept which is essential to success i.e. that knowledge
is distributed across networks of connections, and learning
therefore consists of the ability to construct and traverse these
networks (Siemens, 2005). The concepts and success factors
related to e-learning in the twenty-first century will surely
change the current learning climate, and may have an impact on
how benchmarking e-learning in higher education will be con-
ducted in the future and the kinds of quality-related issues
which matter (Ossiannilsson & Landgren, in press).
Management is fundamental regarding the integration and
implementation of ICT (Ubachs, 2008; Sangra, 2008). Sangra
(2008) stated that the integration of technology, organisation
and pedagogy is crucial for success, as all of the elements are
needed to increase productivity and processes. New challenges
for universities in the twenty-first century include bringing
together all of the aspects of e-learning in a holistic framework,
and perceiving these concepts in context (Ehlers & Schnecken-
berg, 2010; NAHE, 2008; Soinila & Stalter, 2010). Bates (2009)
argued for the need for experimentation, innovation and vision
where there are challenges, in order to bring together three
competing factors: increasing access; increasing quality or im-
proving outcomes, and reducing costs.
The Projects
Two European benchmarking initiatives regarding e-learning
were recently implemented. The first one was the E-xcellence+
project, carried out during 2008 through the EADTU (Ubachs,
2009). The second was an e-learning benchmarking exercise,
carried out during 2009 by the ESMU in collaboration with the
EADTU (Comba et al., 2010; Williams & Rotheram, 2010).
The focus of this article is on these two projects. Neither the
research, nor this paper, will primarily focus on single bench-
marks, indicators, critical success factors or methodologies as
such, but rather on the value and impact of benchmarking on
e-learning. Both initiatives aimed to identify good practices
with regard to e-learning by learning from other participants.
The E-Xcellence+ Project
The EADTU (Ubachs, 2009) initiated the E-xcellence project
in 2004 as part of the e-learning programme on behalf of their
members, who had indicated that specific e-learning criteria
were missing from the current quality assurance systems in
place in Europe. The E-xcellence instrument was developed in
co-operation with The European University Association (EUA),
United Nations Educational, Scientific and Cultural Organiza-
tion (UNESCO) and The European Association for Quality
Assurance in Higher Education (ENQA), in order to comple-
ment the existing quality assurance systems in higher education,
rather than to interfere with them. The project is more explored
and investigated in more detail in research papers by Ossian-
nilsson (2010, 2011) and by Ossiannilsson and Landgren (in
press). The quality benchmarking assessment instrument which
was developed covered pedagogical, organisational and techni-
cal frameworks, with special attention being paid to accessibi-
lity, flexibility and interactivity. After the piloting phase, and
under the E-xcellence Associates label, personalisation was also
highlighted. The instrument was based on a manual covering 33
benchmarks on e-learning, across three main areas: manage-
ment (strategic management), products (curriculum design,
course design and course delivery) and support (student and
staff support), with indicators relating to benchmarks, guidance
for improvement and references how to achieve excellent levels
of performance. In addition, assessors’ notes were included to
provide a more detailed description of the issues and ap-
proaches, as well as the tools, comprising the online instrument,
the quick scan and the full assessment. E-xcellence+ was pilot-
ing during 2008 at local seminars (Ossiannilsson, 2011), and
three universities carried out the full assessment, together with
site visits. Several universities carried out the quick scan. Uni-
versities who conducted the full assessment on the level of
excellence and underwent site visits, worked out roadmaps and
committed themselves to carrying out the benchmarking pro-
cess for e-learning in higher education every other year awarded
E. OSSIANNILSSON 211
the E-xcelle nce Associates label.
The E-Learning Benchmarking Exercise
The e-learning benchmarking exercise, devised by the ESMU
in co-operation with the EADTU (Comba et al., 2010; Williams
& Rotheram, 2010), was carried out by nine European universi-
ties during 2009-2010. This initiative combined the ESMU’s
collaborative benchmarking approach through a comparison of
the good practices of different universities with the EADTU’s
more individualistic approach, as described above. The exercise
started with the self-evaluated quick scan from the E-xcellence+
scheme (Ubachs, 2009). Furthermore, two workshops with in-
ternational experts were carried out, and roadmaps for im-
provement were designed by the partners. In order to find some
kind of common point of departure within this partnership,
e-learning was considered to be.2
Material and Methods
In order trying to explore a multifaceted phenomenon in
depth, this study used an exploratory multiple case study stra-
tegy, as described by Yin (2003, 2009). A mixed-methods ap-
proach was applied, utilising a combination of qualitative data
sources and integrated methods for data analysis (Yin, 2003,
2009; Creswell & Clarke, 2007). The data were mainly pro-
cessed using cross-case analysis, with embedded and multiple
units of analyses.
Respondents
The cases for the current study were selected from both pro-
jects: E-xcellence+ (three respondents) and the e-learning
benchmarking exercise (four respondents). One of the respon-
dents from the latter project took part in both projects (Table 1).
The respondents from the E-xcellence+ project were chosen
because they had been the first in Europe to be awarded the
E-xcellence Associates label. All nine partners from the e-
learning benchmarking exercise were invited to take part in the
study, and five of them agreed (however, one agreed too late to
be included in this paper). In the following work, the respon-
dents are not called by their real names, but instead are called
Alpha, Beta, Gamma, Delta, Epsilon and Zeta.
Data Collection and Procedure
A case study protocol was worked out for the data collection
procedure, which was then sent to the respondents in advance.
The procedures were carried out so as to increase the reliability
of findings and to maintain the chain of evidence, in line with
the work of Yin (2003, 2009).
Existing survey data. Basic information about the respon-
dents and their universities was found out in advance, and more
information was gathered throughout the projects and their
completion. The experts’ reports from the benchmarking exer-
cises were used as additional data (Comba et al., 2010; Wil-
liams & Brown, 2009; Williams & Rotheram, 2010).
Additional data for this study were collected through in-
Table 1.
Respondents involved in the benchmarking projects, EADTU E-xcel-
lence+ and the ESMU e-learning benchmarking exercise.
Respondents E-xcellence+ e-learning
benchmarking exercise
A Alpha x
B Beta x
Γ Gamma x
Δ Delta x
Ε Epsilon x
Z Zeta x x
depth qualitative interviews or more as narratives recorded
through Adobe Connect. One interview was carried out at the
respondents’ office (in this case, two respondents were both
involved in the dialogue throughout the interview, which is also
the way in which they work and teach) and recorded with a
manual audio recorder. The interviews were typed out in verba-
tim. Following the case study protocol, open-ended questions
on the following main themes were used for the interviews in
order to capture the respondents own experiences:
Why did your university joined the benchmarking project?
What was your opinion of the process after its completion?
In your opinion, where there any drawbacks within the pro-
ject process?
Do you have any additional thoughts regarding the in-
volvement in the benchmarking project?
Data Analysis and Process
The existing survey data served to illustrate the case studies.
Subsequently, during the analysis process, the data sources
were analysed and integrated in order to contextualise the find-
ings (Creswell & Clarke, 2007) and to elucidate the issues re-
garding the value and impact of benchmarking on e-learning in
higher education. Each case was first analysed on an individual
basis, followed by cross-case analysis (Miles & Huberman,
1994; Yin, 2003, 2009). The qualitative data from the cases
were processed in the following way. In order to obtain a gene-
ral and holistic sense of the whole situation and to become fa-
miliar with the information in the cases, the recordings and the
transcriptions of the in-depth interviews were repeatedly lis-
tened to and read through case by case. Listening to the mate-
rial repeatedly reminded is of its meaning and content, but also
gave a sense of the feelings and emotions contained in the nar-
ratives. The essential statements which had been made i.e. re-
garding the significant issues (natural units) were distinguished
(Yin, 2003, 2009). The words were counted and a “wordle”3
was worked out (Figure 2). The text, case by case, was further
condensed into units of meaning in relation to the respondents’
expressions which served to highlight certain issues. Using this
method ensured that the findings stayed close to the original
data. The findings were aggregated or clustered according to
key issues, reflecting the value and impact of taking part in
2…covering a wide set of applications and pedagogical processes sup-
p
orted by ICT learning, such as web-based learning, computer-base
d
learning, virtual classrooms and digital collaboration with an added value
of increased accessibility, flexibility and interactivity.” (personal communi-
cation, ESMU workshop, 25 May 2009.
3Wordle a tool for generating “word clouds” from text that you provide, the
cloud give greater prominence to words that appear more frequently in the
source text.
E. OSSIANNILSSON
212
Figure 2.
An example of a Wordle made from the interview texts.
benchmarking exercises. The analyses produced a summarised
description of each case. The descriptions were intended to re-
flect the respondents’ narratives and core experiences as closely
as possible. The findings begin with an overview of the six
cases, and then the findings will be presented case by case.
Finally, a cross-case analysis, exploring similarities, differences
and variations, will be presented at the end of the findings.
Findings
Overview of the Cases
The six cases and all of the respondents were from research
intense universities and so-called traditional universities which
employ a blended approach. Three of them were in a process of
transformation, in that they were merging universities and fa-
culties, which was one of the reasons for participating in the
benchmarking exercises. They were also in the process of for-
mulating e-learning strategies. Three of the respondents were
from universities in which e-learning was not separated from
other forms of learning e.g. in policies and plans; however, they
strongly emphasised new technologies and e-resources through-
out the university. The universities had well-known interna-
tional reputations for e-learning and distance education.
In the report from the e-learning benchmarking exercise, the
experts noted that participation in benchmark exercises always
involves reflections on the part of the institutions involved on the
lessons learnt and on new approaches and methods that the
experience can be expected to bring. They suggested “...inte-
gration of benchmarking into strategic planning and to conduct
benchmarking exercises as a regular practice to support on-
going organisational evaluation and retaining a competitive
edge. Such a change agenda depends crucially on strong lead-
ership to set clear directions and ensure their implementation.
Highly-performing higher education institu tio ns use a variety of
tools, including benchmarking, to better understand their op-
erations and progress towards increased performance” (Wil-
liams & Rotheram, 2010: pp. 29-30). The experts also empha-
sised the important lesson that, in reality, the most important
work in the benchmarking process actually starts when the
benchmarking procedures are over and the implementation
phase begins i.e. when changes are made with regard to human
capital and implemented within the organisation.
In the report by Williams and Brown (2009), they pointed
out, in addition to their findings from the site visit, some prob-
lems with international benchmarking. Some of the problems to
be aware of are, for example, the fact that benchmarks often
relate to more than one point and thus require a multi-faceted
response and a team-based approach. An important issue which
they noted concerns the interpretation of languages and linguis-
tics, as well as cultural interpretations. In this case (E-xcel-
lence+), some benchmarks were also lacking e.g. library re-
sources and student involvement, which were reported back to
the EADTU. Williams and Brown (2009) also expressed that
university campus courses would gain a great deal if they had
the same transparency as the e-learning courses which were
involved in the benchmarking process. They emphasised the
importance of commitment at the management level in question
when starting a benchmarking exercise, as benchmarking is
about the implementation of results and (eventually) about
change.
The Six Cases
The findings from the cases are summarized and presented
below case by case. Each case starts with setting the scene fol-
lowed by their experiences expressed in their narratives.
Alpha
Setting the scene. Alpha expressed two reasons for joining
the project (the e-learning benchmarking exercise). One of the
reasons was the desire to be part of one of the leading universi-
ties in the country within the e-learning arena i.e. to measure
performance and to measure value for the money invested.
Some ranking aspects were also considered. Another reason
was to learn from others and to network for the purposes of
further collaboration.
Being a multi-campus university creates challenges when it
comes to staff and student support and the organisation of
competence development schemes for both administrative and
teaching staff in the field of Internet-based communication,
collaboration and learning. At the same time, the multi-campus
situation provides a natural driving force for the implementa-
tion of online technology that can help to save time and money
on transportation between campuses.
Experiences. Alpha expressed and referred to the ten sum-
marised values which were described above by van Vught et al.
(2008b). However, Alpha expressed that one of the most im-
portant values was possibly decision-making (i.e. during and
after a benchmarking exercise, when an institution receives
basic data and documentation). Other core values were per-
ceived as an increase in awareness and understanding at the
individual, collective and organizational levels. This is a sus-
tainable value, as it involves the internal involvement of stake-
holders within the organization. Alpha also expressed that they
had obtained both explicit documentation and also tacit know-
ledge and understanding which, due to the benchmarking exer-
cise, became explicit. In addition, Alpha articulated that when
one undertakes a benchmarking exercise with benchmarks to
relate to, and holds discussions and interviews with stake-
holders in an organization, a collective picture of the situation
is gained, which had a fairly large impact. Relationships were
strengthened and team members became closer to each other.
According to this respondent, benchmarking is very much a
question of involvement. This has resulted in spin-off effects in
other projects within the organization.
Alpha also expressed that one of the lessons learnt was that,
during the benchmarking itself, a great deal of attention was
devoted to gathering data, collaborative discussions, writing
reports and formulating action reports with a lot of valuable
E. OSSIANNILSSON 213
help received from partners and experts. Conversely, less atten-
tion and support was gained during the implementation phase.
Choosing stakeholders is another important issue, especially
with regard to organizational learning, when the management
needs to be involved. Alpha stressed the importance of com-
mitment at the managerial level. This is probably the most im-
portant step in terms of organizational learning and develop-
ment. Alpha felt that they came out relatively well in the
benchmarking exercise, both in terms of their self-evaluation
and also compared with the others. Accordingly, they learnt
that they needed to look even more at their practices, and that
improvement and consolidations could be made. Additional
lessons were to consider a more constructive approach to
learning in relation to e-learning, and that learning needs to be
more deeply embedded in educational processes, i.e. to focus
on how learning can be enriched by technology, rather than
how technology can enrich learning. The fact that a new project
had been created within the organization with the aim of im-
plementing some of the results of the benchmarking exercise
was an explicit result. The values of the exercise, for Alpha, lay
in extended networking with partners in the project, which had
led to spin-off effects such as new projects and collaborations.
Alpha emphasised the importance of benchmarking as a
method for quality assurance, as it involves self-evaluation and
learning through and with others. Nevertheless, it requires a
large amount of resources. Alpha put forward the idea that, in
this context, it is probably of value to consider the content and
the number of benchmarks. The drawbacks were that there were
too many extensive benchmarks. Another issue was the linguis-
tic sense, (the benchmarks were in English), but also in the
translated form, there were different cultural interpretations
among the partnership, but also within the organization.
Beta
Setting the scene. One of the main reasons why Beta joined
the project (the e-learning benchmarking exercise) was the
desire to be able to compare their e-learning efforts with those of
other universities. Other reasons to participate were a need to
gain new inspiration and insights, and also to learn from others
in a spirit of collaboration. Beta is also a multi-campus univer-
sity, and so the same arguments apply as in the case of Alpha.
Experiences. Beta expressed a whole range of benefits for
their university. On a personal level, Beta learnt a lot about
their organization. An unexpected advantage was that they be-
came aware of some new contacts, for strategic and practical
work, even within the organization. They also appreciated that
they gained a good overview of what they were actually doing
within the field of e-learning, as well as their strengths and
weaknesses, what could be improved and what targets to set for
the years ahead. Beta expressed this as follows: “I think it sur-
prised me how many different benefits on so many different
levels in the organization we actually gained. It was valuable to
focus on the entire organization and the future of e-learning in
the university. At a managerial level, the project report was
seen as important, as the university came out with good results.
This also showed that the resources allocated to the area of
e-learning at the university were meaningful. It focused their
attention on quality and the need to link e-learning to quality
management efforts and to secure the widespread use of
e-learning. Within Beta’s quality organization and quality com-
mittees, they ensured the involvement of stakeholders and the
dissemination of the project. With this level of organization and
commitment, it was easier to work with the lessons learnt.
Though, this process was not perfect from the very beginning,
but they caught up in many ways during the process. From the
start, Beta was under pressure to come out as one of the best
universities in the partnership.
Conversely, having gone into the process and learnt more
and more about the organization, Beta felt that “the lessons you
learn and the knowledge you gain in the process is perhaps the
most valuable part of the benchmarking process. Of course,
they were happy to point out their strengths, but also to record
that there were some points which needed improvement, “so
you do not just rest on your laurels”.
Beta also gave voice to the fact that the benchmarking pro-
cess and results motivated people to developments, either by
continuing with current developments or through innovation.
Beta articulated that there were some limitations to the ways in
which the benchmarks were formulated, in that they were
pragmatic and depended on a certain way of looking at e-
learning. Thus, Beta would recommend more open-ended ques-
tions and also more time to work through and revise the
benchmarks within the partnership. It should be more like a
joint effort and a collaborative approach. It was also suggested
that time frames could be added for the further development of
the process, i.e. time allocation for follow up processes.
In addition, Beta emphasised that the successful areas within
e-learning are mainly based on backup from top management,
with clear and obvious visions and strategies throughout the
whole organization. The development of teachers’ competences
within the fields of learning and blended learning are also of the
highest importance, together with pedagogical and technical
support. Well-implemented e-learning platforms and tools are
necessary. Knowledge sharing and the sharing of experiences
both nationally and internationally are furthermore of the ut-
most significance. Finally, Beta expressed that funding should
be provided for pilot projects or the most innovative projects
within a certain area.
Gamma
Setting the scene. Gamma expressed three main reasons for
participating in the project. First, they expressed that they were
in the middle of a major merging process, and that they were
also reducing the number of their faculties. Second, they ex-
pressed that there was a wide variety of e-learning standards
and procedures at the university. Getting involved in the ben-
chmarking exercise was therefore one obvious way in which to
gain an overview of what was going on and a way to com-
mence internal co-operation. Finally, their participation was a
way for them to generate a roadmap and to get some new ideas,
thanks to this international co-operation. They had no expecta-
tions regarding how well their participation would go or
whether or not they would succeed. Gamma is also a multi-
campus university, and so the same arguments apply as were
mentioned for Alpha and Beta.
Experiences. Gamma articulated that, in principle, bench-
marking had a great deal of value and a significant impact, as
suggested by van Vught et al. (2008). The value for Gamma
was not so much in learning with and from collaborators, as in
coming to know one’s own organisation, especially as they
were in the process of an extensive merger (so they did not
have any working practice to relate to). Therefore, the main
E. OSSIANNILSSON
214
impact upon Gamma was related to values and lesson learnt
regarding how they formulated themselves. Gamma expressed
that “benchmarking is a great tool to point out flaws, to explore
and spot missing areas and to raise attention. It is also a good
opportunity to work with best practice and it is also a method
to create inspiration. Gamma is in a phase of reorganization,
and due to this they had many ideas relating to the processes of
innovation. Gamma suggested that benchmarking ought to be
mandatory for universities, either on an annual basis or every
other year. Benchmarking should take place in different areas in
order to enhance quality and with the aim of being in the front-
line. In this case, benchmarking is a great method for educa-
tional movement; enhancing quality and working processes in
order to relate to best practice etc. in a more structured and
more professional manner. Universities need to have bench-
marks and indicators in order to speak the same language. In
addition, benchmarking allows universities to extend their net-
works and to collaborate on setting up new projects or bilateral
agreements (spin-off effects).
Gamma expressed some pitfalls of benchmarking, which of
can be turned into opportunities as and when they became visi-
ble. Some pitfalls are language and cultural differences, but
also the question of how to set benchmarks e.g. in this case,
they were rather unfashionable, so Gamma had to go above and
beyond in order to connect to their area and context. In order to
improve, it is essential to consider cultures, such as the univer-
sity’s own culture but also international perspectives. The dis-
advantages also include that benchmarking is time-consuming
and demands focus. However, Gamma expressed that if one
spends time and focuses on the work in a benchmarking exer-
cise, one will get more out of it and therefore gains more from
it: “What comes in goes out”.
Gamma also stated that they now focus more on learning and
how students interact with information technology (IT) and the
social media in their context. This raises questions about the
contextualization of learning and education. Therefore, in the
event that a benchmark exercise should be set up again, they
will consider these phenomena to a greater extent.
Delta
Setting the scene. Delta expressed that as they were selected
to take part in the benchmarking project (E-xcellence+), they
naturally appreciated the opportunity and wanted to join in.
They were also curious with regard to obtaining an awareness
of how far they had reached, even though they had established
an international reputation many years ago, and had many in-
ternational students enrolled. Second, they saw the advantages
of the process, such as obtaining an internal awareness and
knowledge about their own organization. The third driving force
was the fact that there was an opportunity to attain official in-
ternational recognition; this, of course, was a motivating fac-
tor. Another motivating factor was the future situation of the
university, as they from 2011 have to charge tuition fees for
students outside Europe. Competition will thus be more explicit,
and there will be an even stronger drive to ensure that students
get what they pay for. Being internationally distinguished is, in
this context, very significant.
Experiences. Delta was the first university in Europe to be
awarded the E-xcellence Associates label by EADTU as a con-
crete recognition of the high quality of their e-learning pro-
gramme. As a result of this, the level of their national and in-
ternational collaboration, as well as student recruitment, was
increased. The real impact of this will first be noted in a few
years when the first students (who are paying tuition fees) en-
roll and they are asked why they chose this programme instead
of others. In addition, students who are already enrolled are
impressed and appreciate being enrolled in a programme which
has been labelled as excellent. They have also articulated a
positive response to the educational culture and level of profes-
sionalism they have experienced. Delta also emphasised that
they had received an increased level of recognition for their
work at different levels within the university, which has re-
sulted in spin-off effects.
Delta stated that, in terms of their positive experiences, they
have had the opportunity to undertake the benchmarking and
internal processes, which increased their understanding and
knowledge. To have the chance to reflect and to share within
the centre, but also with colleagues within the university, was
very valuable, and made their work more effective. In addition,
it was valuable to perform a self-evaluation, as well as the
documentation and the experts’ report. This was also a way for
Delta to gain an awareness of the areas in which they fall short.
In addition, they had opportunities to review and improve their
routines.
Thus far, they have not been involved with or taken part in
the Excellence Associates club, but they will in the future and
can see the advantages of such a network. The roadmap that
they worked out has served as an inspiration and as a driving
force. It is not a formal action plan, but a working plan and an
inspirational document. They expressed that they have either
worked through some of the actions or at least considered them.
It is an honest document, referring to what they ought to do,
considering time restraints and financial opportunities. They
articulated that, during the process, they benefited from support
and commitment from the management team, and they also
stated that, without this support, it would be more or less im-
possible to carry out changes or improvements in relation to the
benchmarking exercise. Other issues were that the exercise was
so time-consuming due to the need for data gathering, internal
discussions and analyses that support and commitment were
needed. They expressed that they had experienced continuous
support, even now the real work has started with the imple-
mentation and transition phase.
Delta emphasised the importance of involvement and team-
work during the process. The benchmarking exercise was as
such also important due to the need to “attract attention, gather
and spread information and engage staff and students (three
core concepts”. Delta articulated that the benchmarks were
comprehensive, but maybe too comprehensive. On the other
hand, this contributed to discussions, interpretations and the
development of knowledge within their own educational culture
and context. If Delta were to undertake the exercise (or a simi-
lar exercise), again, they stated that it would be valuable to
estimate time requirements, and probably also to have a more
explicit overview of the process. The drawbacks of the exercise,
when one is sensitive to them, can be turned into opportunities.
Another issue is whether the university could or should market
and raise awareness of the benchmarking process to an even
greater extent.
Epsilon
Setting the scene. Epsilon indicated three incentives for their
E. OSSIANNILSSON 215
participation (in E-xcellence+ and the e-learning benchmark-
ing exercise). First, they expressed that they wanted to obtain
an awareness of what was state-of-the-art in the area. Second,
they wanted to learn from others and to obtain an awareness of
their progress in the area in an international context. Finally,
they expressed that participating was a way to obtain insights
and knowledge with an eye to the future.
They also expressed that their participation was a way of
gaining both internal and external processes. They stated that
benchmarking is an optimal method for raising awareness and
understanding of what is going on, both internally and exter-
nally. Both advantages and disadvantages become explicit, but
primarily and most importantly, potential areas for success and
where quality can be enhanced become visible within the com-
mon understanding in an organization.
Experiences. Epsilon expressed having found the exercise to
be valuable both internally and externally, and that lessons can
be learnt in and through both dimensions. For Epsilon, the pro-
jects had been extremely valuable. Benchmarking is a way of
watching what is happening in the surrounding world. Regard-
ing Epsilon’s participation, it can be stated that they found it to
be valuable on several levels; within departments, on the uni-
versity management level, and on national and international
levels. One example which was articulated several times in the
in-depth interview was the fact that a new programme is going
to be set up in collaboration with several other universities in
the country. With this programme, there is a need for high qua-
lity work to be carried out with new technologies, not just for
the purposes of content and inter-university collaboration, but
also to create a model and to become part of an IT society in the
twenty-first century. Even within other areas of the university,
schemes are being implemented as result of the benchmarking
projects in combination with the current discourse in this area.
Epsilon articulated that they found it interesting how many
outstanding things had happened throughout their faculties and
departments; it seems likely that these things have happened
due to the fact that Epsilon is a decentralised university. They
therefore expressed that freedom, enthusiasm and development
on demand are crucial for a reflective and growing organization.
The key is not always efficiency, as sometimes quality en-
hancement is more important. Both bottom-up strategies and
top-down strategies require flexible methods. Benchmarking
helps universities to choose a direction. At a research intensive
university, where the teachers and programmes are at a variety
of levels in terms of IT proficiency, from early beginners to
those who excel, there are a variety of demands and require-
ments; however, there needs to be something for everyone,
even at a central organizational level. The most important thing
at a large university is to develop maturity and “teaching envi-
ronments to learning environments to web environments and
also that it [e-learning, new technologies, social media etc.] is
embedded in [the] curriculum. This journey has to take time.
An awareness of maturity is of great importance, as if an or-
ganization is immature, then no development will happen.
There needs to be maturity, at both individuals and organiza-
tional levels. Each faculty/department has their own structures
for dealing with demands and cultures. This is good. In addition,
one has to make choices, and one has to be familiar with e-
learning and technology development. When maturity has been
reached, co-ordination can begin.
The fact that two of the programmes at Epsilon were awarded
the E-xcellence Associates label indicates that they have been
recognised both internally and externally. For the programmes
that were awarded the label and already have large groups of
international students from outside Europe, this award is very
valuable and will have a significant impact as they are facing a
new situation regarding tuition fees for students from outside
European Union. Of course, this is due to other programmes as
well. The E-xcellence Associates label supports the entire uni-
versity in its strategic work towards achieving consolidation,
internationalization and distinction, as universities are striving
to be world class. Epsilon stated that they valued all of the
documentation which had been a part of the benchmarking
projects, which they will now use for other purposes. One ex-
ample is that when the National Agency for Higher Education
conducted a survey of distance education in the country (Swe-
den), then the benchmarking reports supported the documenta-
tion required for this survey to a large extent.
Epsilon stressed the fact that e-learning needs to be user-
friendly, integrated and embedded into courses and educational
arenas. Epsilon stressed further the importance of freedom in a
large organization, as creativity grows and things happen when
there is freedom.
Zeta
Setting the scene. The main reason why Zeta chose to parti-
cipate in the project (E-xcellence+) was to ensure the quality of
its education and courses and to provide evidence that these
were of the highest possible quality. The main reasons for this
are due to student rights and student satisfaction. Other reasons
were of both an internal and external character. The internal
reasons were a desire to raise awareness of advantages and
disadvantages concerning range, support and implementation of
the e-learning arena at Zeta, and in addition student satisfaction,
and through the process find gaps for possible improvements.
The external reasons were mainly a desire for international
co-operation and networking, but also an attempt to learn from
and through others. As benchmarking is a form of self-evalua-
tion, it is more honest and the motivation to make changes is
thus higher, as any criticism tends to be more constructive.
There were also other reasons, such as the fact that each mem-
ber of the staff including the students need to feel appreciated
for their work. It is also important to note who the actors are
who initiated and took part in the benchmarking e.g. in this case,
the benchmarking was initiated by the EADTU, the National
Agency for Higher Education and the university itself. The
chance to attain the E-xcellence Associates label was also an
incentive to join the project.
Experiences. Zeta expressed that through the benchmarking
project, advantages and disadvantages concerning range, sup-
port and implementation of e-learning became visible and ideas
for changes were considered. The fact that Zeta earned the cer-
tification and the label was, of course, very encouraging, and
the label demonstrated that the university had been recognised
internally as well as in international collaboration and net-
working. In addition, more students are being recruited. This is,
of course, a triumph, especially when the university was co-
operating with countries that tend to distrust distance education.
However, Zeta expressed that they might be able to market
distance education to an even greater extent, but they already
have more than enough students (i.e. more than they get paid
E. OSSIANNILSSON
216
for). Naturally, this was an occasion for self-evaluation and this
influenced all staff and their work. Of course, the label as such
is not as important as the knowledge and understanding of the
work which the university is doing, as well as an awareness of
their shortcomings. Nobody is happy with just the award itself.
Zeta, therefore, articulated a very humble attitude towards its
recognition. However, two of the leading staff at Zeta awarded
the university a prize for excellence in teaching and pedagogy
in the following semester.
Zeta stated that the roadmap served as inspiration and not as
a dogmatic document. Some of the actions have already been
taken and some are being considered, due to limited time and
resources. They expressed that they have been supported by the
university, although perhaps not always at the closest adminis-
trative level i.e. the faculty level, probably due to a degree of
mistrust, as they are too far ahead and too innovative in their
course development and delivery. Maybe this had been the fact
anyway, if they had been innovative and therefore succeeded in
other areas, they would have experienced mistrust there as well.
Concerning the benchmarks, Zeta expressed that they were
particularly extensive and that they worked in teams in order to
get the staff and students involved. They used micro and mini
meetings to go through everything and to discuss different as-
pects of the process. Zeta gives voice to the idea that one of the
most important issues is flexibility at all levels and dimensions
in any attempt to satisfy students. The ultimate goal is to satisfy
students, and they frequently demand a high degree of flexibi-
lity. Satisfied students are important quality indicators. Flexible
and boundless education with regard to pathways, time (even
starting time), free resources, flexible materials/resources (texts,
audio, video) etc. is of paramount importance and forms the
foundation of the programme. Student evaluations have also
shown that this flexibility is crucial.
One explicit impact was an inspection of the learning plat-
form and the overall university administrative system in order
to make communication more flexible. Zeta stated that when
e-learning programmes begin, the most important aspect seems
to be the learning dimension. This is natural, but when courses
are scaled up, it is crucial that administrative systems commu-
nicate well.
Thus far, Zeta has limited experience of the E-xcellence As-
sociates club, due to limited time and human resources. Obvi-
ously, such initiatives are of the highest importance. Zeta’s
final comment was that universities which are striving for the
highest quality in an international field should certainly con-
sider benchmarking.
Cross-Case Findings
Overall, the findings indicated many similarities between the
universities, but also differences, or rather different ways of
expressing experiences across the cases. The cases illustrated
how different aspects interact to a varying extent and indicated
the value and impact of the international benchmarking exer-
cises which were conducted. In the section below, some of the
key areas are summarised.
Internal Processes and Involvement
The benchmarking allowed the teams to maintain their focus.
Involvement and shared responsibilities in the work with bench-
marking enabled not just the benchmarking itself, but also con-
tributed to commitment and appreciation among the co-workers
within the institution. The fact that advantages and disadvan-
tages in the field of e-learning became explicit supplied the
motivation to make the changes which led to the impact of the
scheme. In addition, the fact that the exercise was mainly a
self-evaluation motivated the institutions to examine even nega-
tive issues, which often could be turned to challenges.
In addition, the solid documentation which was collected, as
well as the knowledge and institutional awareness regarding e-
learning was important and led to a significant impact. The
institutions gained a lot from the solid work with its transparency,
which could even be used and valued in other contexts. The
awareness of the infrastructural support for e-learning, which
was one of the results of the benchmarking exercise, was made
explicit and this led to closer collaboration between the infra-
structural units in question, as well as further collaboration with
other faculties and departments, for example within pedagogical
areas and e-resources. Additional benefits were clearly expressed
regarding teamwork, creating a dialogue, policy making, quality
assurance and transparency within the organization.
Quality Enhancement
Benchmarking is an advanced method, but the cases have in-
dicated that it is quite easy to use with explicit opportunities for
self-reflection. Even though certain steps have to be taken, one
can set the structure within the partnership as either a smaller or
a larger exercise; one can also define the benchmarks, partners
and the time schedule. Nearly all of the cases reported that
enough time and support was allocated during the project period.
However, they emphasised that they misallocated time and
support for the reflection and implementation phases. This
process needs to be followed by reflections on the lessons learnt,
which can provide incentives for further development in the
institution. This may involve changes in structure, organisation
and resource allocation, which may require strong leadership at
all levels. Furthermore, the importance of integrating bench-
marking processes as a natural part of strategic quality assurance
work was emphasised. The value of continuously following up
earlier benchmarking exercises as well as committing to new
projects was highlighted. All of the respondents agreed on the
value of benchmarking as a method for quality assurance.
Management and Commitment
The cases showed, to a significant extent, the importance of
full support and commitment from all levels of the management
team during the whole benchmarking process. Such support is
necessary in order to maintain the focus on the project, to allow
staff and students to be involved, to work in an interdisciplinary
manner and award status for the ongoing work and dissemina-
tion during the process, but most of all with regard to data
gathering and reports i.e. the implementation phase, if changes
have to be made. In these cases, several departments were in-
volved in full. They were therefore able to focus on and discuss
common areas and processes, creating togetherness, trust, com-
mitment and involvement. This will certainly contribute in turn
to enriching the future employment situation and potential areas
for development.
Collaboration and Networking
When taking part in benchmarking exercises, universities
have the chance to meet other institutions that are facing the
same challenges as themselves. There are opportunities to ap-
proach one another, to learn from best practices and to take part
E. OSSIANNILSSON 217
in joint discussions on how to handle challenges. This can evolve
into continued benchmarking on a smaller scale, where two or
more universities from the group collaborate and formulate new
benchmarks. Such new benchmarks can help the institution to
improve important areas and can play a major role in continuing
quality assurance efforts. On a more practical level, the know-
ledge exchange between benchmarking participants can lead to
mutual inspiration and can help the individual universities to
understand and handle topical issues within the field of e-
learning.
In the benchmarking exercise, it became apparent that each
university possessed best practices within certain areas that were
of importance for all of the universities. These areas included
pedagogy, technology, strategy etc. In the benchmarking exer-
cise itself, there was unfortunately no time to study the best
practices of the other universities in depth.
However, the exercise itself provided a good overview of the
strengths of each individual university. Mutual trust and under-
standing has been gained, and future activities such as joint
applications and agreements for further co-operation will natu-
rally be established. A network has come into being, in which
participants can use each other as experts when knowledge of a
certain topic is needed.
Cultural Issues
As this was an international benchmarking initiative, English
was the chosen language. All of the respondents raised a prob-
lem with this, as none of them had English as native language.
It was not just the language and linguistics as such, but also
cultural aspects of the language and how it is used in different
contexts. The benchmarking was mainly carried out in teams,
and even within the teams there were differences in how certain
concepts were understood. Another issue was the somewhat
old-fashioned concepts regarding e-learning. Nearly all of the
cases work within a blended mode context. In addition, nearly
all of the universities in question are decentralised, and thus
there was not only one way of acting and performing.
Attitudes
Through the in-depth interviews, all of the cases voiced their
enthusiasm and positive feelings regarding the exercise itself,
but also regarding the methods. It was, for example, noted that
during a comparatively short time, many lessons had been
learnt, both about the institution themselves, but also about
wider perspectives including other universities and the sur-
rounding international context. Furthermore, it was articulated
that they had also gained a deep understanding and knowledge
in the area of e-learning through the benchmarks and indicators,
areas which to some extend seemed to be tacit knowledge. In
addition, they expressed their enthusiasm and interest in con-
tinuing to work, or as they put it, they had the energy to tackle
future challenges. All of the respondents expressed that it
would be useful to carry out the exercise again, but maybe on a
smaller scale with time to define their own benchmarks in a
more useful way.
Discussion
This study explored the experiences of respondents who had
taken part in an international benchmarking project. The inte-
grated case study findings provided critical information on the
value and impact of benchmarking in higher education. Most
importantly, this study showed the complexity of the value and
impact of taking part in such exercises. The complexity of
benchmarking e-learning in higher education is based on the
fact that values can refer to either immediate or more long-term
values. Furthermore, these values can apply to different aspects
and different levels, and to both individuals and organizations.
Moreover, some effects take years to evaluate.
One contribution of this study to our present knowledge and
understanding of benchmarking as a method seems to be a
recommendation that time, resources and maybe even the use of
experts and/or co-partners is allocated for the implementation
phase, as this is an important step in the benchmarking process.
All of the respondents in the study clearly emphasised the lack
of support they had received afterwards. Another contribution
is the apparent importance of commitment from managerial
levels. In the event that structural changes or other changes
have to be made, resources will probably need to be allocated
or re-allocated, and due to this it is extremely important to have
commitment from the leadership. This works both ways; if the
institution produced as “good result”, it is essential to commu-
nicate and market such benefits, internally as well as externally.
It is never easy to speak up for oneself, and so support for mar-
keting is often needed.
We are facing a paradigm shift within the twenty-first cen-
tury, and e-learning as a phenomenon was considered among
the respondents to have a very broad meaning. Therefore, the
learning dimensions were emphasised. Changes in technology
have to be considered in education, even through a social-con-
textual approach. This study shows, therefore, that benchmarks
must be chosen with care and that linguistic, cultural and con-
textual issues have to be taken into account. Even if this study
did not aim to focus on benchmarks as such, critical issues
within the area were articulated, in accordance with the success
factors described by the EADTU (Ubachs, 2008) and Ossian-
nilsson and Landgren (in press). The need for flexibility in all
of the concepts and dimensions was clearly highlighted. Even
the structure of the partnership is crucial, as are similarities and/
or differences within the partnership. How to choose bench-
marks that provide direction is one of the aims within the
benchmarking process. Trust, involvement, collaborative learning,
honesty and courage are of the utmost importance in bench-
marking processes for fruitful and sustainable work and col-
laboration. As mentioned above, there can be pitfalls, but
awareness and thoughtfulness can turn them into challenges.
Turning to the theoretical frame of reference, this study
showed, as the discourse on benchmarking also shows, that
benchmarking is about change, and thus lessons have to be
learnt from benchmarking processes (Moriarty, 2008; Moriarty
& Smallman 2009; Ossiannilsson, 2010, in press, 2011; Os-
siannilsson & Landgren, in press). Once again, as Moriarty &
Smallman (2009: p. 484) stated, “… the locus of benchmark-
ing lies between the current and desirable states of affairs and
contributes to the transformation process that realise[s] these
improvements”. In this study, the findings on the value and
impact of benchmarking as described by the respondents were
close to and in accordance with the ten good reasons to conduct
benchmarking as described by van Vught (2008b). Some more
critical issues were also touched upon, such as the significance
of the first and last phases of benchmarking; the identification
E. OSSIANNILSSON
218
and implementation phases. Furthermore, the value and impact
of tacit knowledge (Elliott & Stemler et al., 2011) were expli-
citly expressed. However, they were not only expressed; the
possibility of sharing this tacit knowledge, as well as know-
ledge and experiences, was articulated as a possible impact.
The case studies indicated the importance of support and man-
agement when making changes, as Sangra (2008) also argued.
Contributions to the transformation processes that realize
improvements require commitment, but also reflection. Through
the cross-case analysis, the respondents experiences in this
study indicated that a process of philosophical interpretation is
required, and that there is a need to modify the theoretical as-
sumptions concerning benchmarking. Høyrup (2004) argues,
for example, that there is a need for a learning organization to
reflect, but he distinguished between reflection and critical
reflection. He emphasised critical reflection in particular as one
of the core processes of organizational learning. He stated,
moreover, that critical reflection was necessary on both the
individual and organizational levels. Critical reflection includes
social, cultural and political aspects and concerns the ‘why’
(meaning) rather than questions about how to act or perform.
The meaning of reflection and the importance of reflective
practices for change was also clearly expressed in the study.
Another dimension of critical reflection can be seen as
Deleuze’s concept of “becoming” (cited in Giger, 2010). Be-
coming share the common meaning of becoming which even is
true in this context. Thus, the process of becoming is about
changes, just like benchmarking is about changes. Becoming,
as a more philosophical concept, has something to do with be-
coming us as individuals. The rhizome concept, also put for-
ward by Deleuze (cited in Giger, 2010), was described in brief
as connectivity and multiplicity. Both concepts by Deleuze are
new dimensions which are valuable for benchmarking and or-
ganizational change.
Reflections on this study also touch on limitations of the re-
search approach. Of course it would have been valuable to have
had all participating universities involved in the study. Probably
the findings had been somewhat more valid. On the other hand
the findings from the cases involved are seen as representative.
Within the six cases variations, but also similarities were found.
Conclusion
Benchmarking is a powerful strategic tool used to assist de-
cision-makers to improve the quality and effectiveness of or-
ganisational processes. This study showed that the quality of
e-learning has to be considered and must be embedded in suit-
able contexts. This study explicitly showed the value and im-
pact of benchmarking e-learning in higher education. In the
case studies, the respondents expressed that the benchmarking
process had a significant impacts i.e. the power to make changes.
Lessons from this study have to be taken into consideration.
Lessons on how benchmarking is conducted, on objectives, on
how it is organized and set up, how to choose partners and how
to choose benchmarks must be acknowledged. In addition, how
to generate commitment and allocate time and resources during
the entire benchmarking process, including the implementation
phase, has been emphasised in certain cases.
Some characteristics which may not be mentioned as often in
a positive light are the intentions, i.e. the meaning of the bench-
marking process as such. The involvement of individuals within
the organization creates an impact on both individual and or-
ganizational levels, not least through critical reflection. The
concepts of “rhizome” and “becoming” used in this study have
leant other and new dimensions to benchmarking. Thus, this
study is not only relevant to management, but to philosophy as
well.
The study shows that benchmarking, in line with national
and international quality boards/agencies, is a powerful tool for
making improvements to teaching and learning in higher edu-
cation.
Acknowledgements
I would like to express my thanks to the respondents who
participated in the E-xcellence+ project and/or in the e-learning
benchmarking exercise.
References
Bacsich, P. (2009a). Benchmarking e-Learning in United Kingdom
universities—The methodologies. In T. Mayes, D. Morrison, H.
Mellar, P. Bulen, & M. Oliver (Eds.), Transforming higher education
through technology enhanced learning (pp. 90-106). Bristol: The
Higher Academy.
Bacsich, P. (2009b). Benchmarking e-learning in United Kingdom
universities: Lessons learned from and for the international context.
Proceedings of the 23rd ICDE World Conference on Open Learning
and Distances Education Including the 2009 EADTU Annual Con-
ference. M-ICDE Conference Flexible Education for All: Open-
Global-Innovative. Maastricht.
Bates, T. (2009). Re: Using technology to improve the cost-effectivenes
of the academy. [Web log message]. URL (last checked 10 June
2011)
http://www.tonybates.ca/2009/10/10/using-technology-to-improve-th
e-cost-effectiveness-of-the-academy-part-1/
Bates, T. (2010). Re: Strategic thinking about e-learning. [Web log
message]. URL (last checked 10 June 2011)
http://www.tonybates.ca/2010/06/11/strategic-thinking-about-e-learn
ing/
Bonk, C. J. (2009). The world is open: How web technology is revolu-
tionizing education. San Francisco, CA: Jossey-Bass, A Wiley Im-
print.
Comba, V., Ossiannilsson, E., Landgren, L., Blok, R., Martiens, F. J.,
Kjaer, C., & Christensen, I.-M. (2010). A benchmarking exercise for
quality blended learning. A challenge for European universities in
the twentyfirst century. Proceedings of EADTU Annual Conference,
Zermatt, 55-75.
Conole, G. (2010). Learning design: Making practice explicit. Con-
nectED Conference 28 June, 2010. Sydney. URL (last checked 10
June 2011)
http://www.slideshare.net/grainne/connect-ed-conole
Creswell, J. W., & Clarke, P. (2007). Designing and conducting mixed
methods research. Thousand Oaks, CA: Sage Publications.
Ehlers, U.-D., & Pawlowski, J. (2006). Quality in European e-learning:
An Introduction. In U.-D. Ehlers and J. Pawlowski (Eds.), Handbook
on quality and standardization in e-learning (pp. 1-3). Berlin:
Springer. doi:10.1007/3-540-32788-6_1
Ehlers, U.-D., & Schneckenberg, D. (2010). Introduction: Changing
cultures in higher education. In U.-D. Ehlers and D. Schneckenberg
(Eds.), Changing cultures in higher education (pp. 1-14). Berlin:
Springer-Verlag. doi:10.1007/978-3-642-03582-1_1
European Commission (2009). The Bologna process 2020—The Euro-
pean higher education area in the decade. URL (last checked 10
June 2011)
E. OSSIANNILSSON 219
http://portal.bolognaexperts.net/files/Leuven_Louvain-la-Neuve_Co
mmunique_April_2009.pdf
Elliott, J. G., Stemler, S. E., Sternberg, R. J., Grigorenko, E. L., &
Hoffman, N. (2011). The socially skilled teacher and the develop-
ment of tacit knowledge. British Educational Research Journal, 1,
83-104. doi:10.1080/01411920903420016
Giger, P. (2010). Conversation and figuration from the horizontality of
the 2.0 decade. Ph.D. Thesis, Karlskrona: Blekinge Institute of
Technology.
Hämäläinen, K., Jessen, A., Kaartinen-Koutaniemi, M., & Kristoffersen,
D. (2003). Benchmarking in the improvements of higher education.
Helsinki: ENQA [The European Association for Quality Assurance
in Higher Education].
Høyrup, S. (2004). Reflection as a core process in organizational learn-
ing. The Journal of Workplace Learning, 8, 42-54.
Johnson, L., Smith, R., Willis, H., & Haywood, K. (2011). The 2011
horizon report. Austin, Texas: The New Media Consortium.
Johnson, M., & Seborg, D. (Eds.) (2007). Benchmarking concepts: An
introduction. Process control performance assessment benchmarking
concepts: An introduction. London: Springer.
McLoughlin, C., & Lee, M. J. W. (2008). The three P’s of pedagogy for
the networked society: Pesonalization, participation and productivity.
International Journal of Teaching and Learning in Higher Education,
1, 10-27.
Miles, M. B., & Huberman, M. A. (1994). Qualitative data analysis.
Los Angeles, California: Sage Publications Inc.
Moriarty, J. P. (2008). A theory of benchmarking. Lincoln: Lincoln
University.
Moriarty, J. P., & Smallman, C. (2009). En route to a theory on bench-
marking. Benchma r k ing: An International Journal, 4, 484-503.
NAHE [The Swedish National Agency for Higher Education] (2008).
E-learning quality: Aspects and criteria. Solna: Högskoleverket.
Ossiannilsson, E. (2010). Benchmarking e-learning in higher education.
Findings from EADTU’s E-xcellence+ project and ESMU’s bench-
marking exercise in e-learning. In M. Soinila, & M. Stalter (Eds.),
Quality assurance of e-learning (pp. 32-44). Helsinki: ENQA [The
European Association for Quality Assurance in Higher Education].
Ossiannilsson, E. (in press). Benchmarking on e-learning in universities:
Impacts and values, European perspectives. International Journal of
Management in Education. Special Issue on Virtual University.
Ossiannilsson, E. (2011). European experiences on the e-xcellence
benchmarking initiative by EADTU. In W. Aung (Ed.), iNEER Spe-
cial Volume: Innovations 2011—World innovations in engineering
education and research. Manuscript accepted for publication.
Ossiannilsson, E., & Landgren, L. (in press). Quality in e-learning—A
conceptual framework based on experiences from three international
benchmarking projects at Lund Univesity, Sweden. Journal of Com-
puter Assisted Learnin, Special Issue on Quality in e-learning.
O’Reilly, T., & Batelle, J. (2009). Web Squared: Web 2.0 Five Years
On. Special Report. URL (last checked 10 June 2011)
http://assets.en.oreilly.com/1/event/28/web2009_websqared-whitepa
per.pdf
Re.ViCa. (2009). Bibliography of benchmarking. URL (last checked 10
June 2011)
http://www.virtualcampuses.eu/index.php/Bibliography_of_benchma
rking
Sangra, A. (2008). Technologies in university: Models, problems and
challenges. Ph.D. Thesis, Tarragona: Universitas Rovira i Virgili.
Soinila, M., & Stalter, M. (Eds.) (2010). Quality assurance of e-le a r n i n g .
Helsinki: ENQA [The European Association for Quality Assurance
in Higher Education].
Schreurs, B. (Ed.) (2009). Critical success factors. Reviewing the vir-
tual campus phenomenon. The rise of large-scale e-Learning initia-
tives worldwide (pp. 59-84). Leuven: EuroPACE ivzw
Siemens, G. (2005). Connectivism: A learning theory for the digital age.
International Journal of Instructional Technology & Distance Lear-
ning, 1, URL (last checked 10 June 2011)
http://www.itdl.org/journal/jan_05/article01.htm..
Ubachs, G. (2009). E-xcellence quality assessment for e-learning.
Heerlen: EADTU [European Association of Distance Teaching Uni-
versities].
Van Vught, F., Brandenburg, U., & Burquel, N. (Eds.) (2008a).
Benchmarking in European higher education. Findings of a two year
EU-funded project. Brussels: ESMU [European Centre for Strategic
Management of Universities].
Van Vught, F., Brandenburg, U., & Burquel, N. (Eds.) (2008b). A prac-
tical guide. Benchmarking in European higher education. Brussels:
ESMU [European Centre for Strategic Management of Universities].
Williams, K., & Brown, T. (2009). E-xcellence site visit report, Lund
University. Heerlen: EADTU [European Association of Distance
Teaching Universities].
Williams, K., & Rotheram, B. (2010). Report. Benchmarking exercise
on e-learning. Brussels: ESMU [European Centre for Strategic
Management of Universities].
Yin, R. K. (2003). Case study research. Design and methods. Los An-
geles, California: Sage Publications, Inc.
Yin, R. K. (2009). Case study research. Design and methods. Thousand
Oaks, CA: Sage Publications, Inc.