Creative Education 2011. Vol.2, No.3, 208-219 Copyright © 2011 SciRes. DOI:10.4236/ce.2011.23029 Findings from European Benchmarking Exercises on E-Learning: Value and Impact Ebba Ossiannilsson Department of Industrial Engineering and Management, Oulu University, Oulu, Finland. Email: Ebba.Ossiannilsson@oulu.fi Received February 22nd, 2011; revised March 10th, 2011; accepted March 20th, 2011. Benchmarking is becoming a more and more commonly used method for quality assurance and enhance- ment in higher education today. The aim of this study was to explore the impact and value of benchmarking e-learning in universities. Two European benchmarking initiatives, E-xcellence+ through the European Associa- tion of Distance Teaching Universities [EADTU] and an e-learning benchmarking exercise through the Euro- pean Centre for Strategic Management of Universities [ESMU], were studied using an exploratory multiple case study method and cross-case analysis of the findings. The findings indicated the value and impact of a variety of as- pects of benchmarking e-learning on different levels, and stressed the need for a holistic and contextual ap- proach as both benchmarking and e-learning are complex and comprehensive phenomena. This study also shows that benchmarking, in line with national and international quality boards/agencies, is a powerful tool for making improvements to teaching and learning in higher education. Keywords: Benchmarking, Critical Success Factors, E-Learning, Quality Enhancement Introduction This paper focuses on research questions concerning the value and impact of benchmarking e-learning in higher educa- tion, which until now has remained a fairly unexplored area. This study uses as points for departure the experiences of two European benchmarking initiatives; E-xcellence+, which was carried out by the European Association of Distance Teaching Universities (EADTU) (Ossiannilsson & Landgren, in press; Ubachs, 2009; Williams & Brown, 2009) and the e-learning benchmarking exercise, carried out by the European Centre for Strategic Management of Universities (ESMU) (Comba et al., 2010; Ossiannilsson & Landgren, in press; Williams & Rotheram, 2010). Moriarty (2008) argued that although the literature on benchmarking is overwhelming, research and evidence on the value and impact of benchmarking is still missing. This re- search aims to contribute to the formation of a deeper under- standing and knowledge of the value and impact of bench- marking e-learning in higher education. This research does not claim to focus on benchmarks as such, and neither has this study been conducted using a special method for analysing impact. Major changes are taking place in European higher education today, and quality in education and research is the key to sup- porting development and innovation. In this context, enhancing the performance of universities and modernising university management must be on the agenda for all university leaders and decision-makers in Europe (Bates, 2010). A clear under- standing and transparency of the modes of operations and processes with the aim of continuous improvement is needed. Universities are facing new challenges in the twenty-first cen- tury, as they are required to be competitive not just in terms of their educational, social, managerial and technological aspects, but they are also called to work globally as drivers for innova- tion and to contribute to sustainable development. In this envi- ronment, in parallel with institutional, regional, national and international demands and challenges, improving the perform- ance and quality of universities has become more important than ever. At the same time, respect for individual students and accountability for the use of funding has to be taken into ac- count, in addition to contributing to economic growth (Euro- pean Commission, 2009; van Vught, 2008a, 2008b). Enhanced quality and increased transparency are strong driving forces behind competition in education and research. Ossiannilsson and Landgren (in press) showed in their study on three interna- tional benchmarking projects that the identification of success factors for e-learning and its implementation in the educational arena is of the utmost significance. Benchmarking has become a more commonly used method for quality assurance and enhancement in higher education, as it deals with identifying gaps and making changes, but also with improvement and successful implementation of new procedures and schemes (Bacsich, 2009a, 2009b; Ossiannilsson, 2010, 2011). In a recent study, Ossiannilsson (in press) advocated the use of benchmarking as a method and a powerful tool for mak- ing improvements to teaching and learning (concerning e- learning/blended learning) in the twenty-first century, and to support improved governance and management in higher edu- cation. The study showed that further research has to be done according to holistic perspectives in order to answer questions on the value and impact of benchmarking e-learning in higher education. These themes include: why is benchmarking used, what is benchmarking e-learning? When is benchmarking used? etc. Thus, this study is based on multiple case studies, together with cross-case analyses, in order to contribute to the existing knowledge and understanding of the value and impact of ben- chmarking e-learning in higher education. In the following section, the concepts of benchmarking and
E. OSSIANNILSSON 209 e-learning will be further elaborated in order to set the scene. After this, the projects will be described briefly, and this will be followed by a summary of the findings from the multiple case studies and the cross-case analysis, with the focus on value and impact. Finally, some reflections and conclusions will be pre- sented. The Concept of Benchmarking The concept of benchmarking has its origins in the manage- ment and business context. Nevertheless, it started in the pri- vate sector, and was introduced by the Xerox Corporation in the ways in which they conducted their successful development. Looking what others were doing, and especially what their competitors were doing, led Xerox to make changes internally in order to improve their quality and processes and to enable the company to gradually regain its market position. Their original definition of benchmarking was: “...a process for improving performance by constantly iden- tifying, understanding and adapting best practices and pro- cesses followed inside and outside the company and imple- menting the results. The main emphasis of benchmarking is on improving a given business operation or a process by exploit- ing ‘best practices’, not on ‘best performanc’. Simply put, benchmarking means comparing one’s organization [sic] or a part of it with that of the other companies.”1 Camp one of the most frequently cited scholars with regard to benchmarking (cited in Bacsich, 2009a; Hämäläinen, Jessen, Kaartinen-Koutaniemi, & Kristoffersen, 2003; Johnson & Seborg, 2007; Ossiannilsson, in press; ReVica, 2009; van Vught, 2008a), explored the benchmarking process using five continuous steps: determining what to benchmark, forming a benchmarking team, identifying benchmarking partners, col- lecting and analysing benchmarking information and finally taking action. The steps are illustrated in Figure 1. Benchmarking has developed into an essential tool for or- ganizations, and is regarded as a vital component of good man- agment practice. It is internationally respected, not just in busi- sses, organisations and management, where the concept came from, but also now in education and higher education (Moriarty, 2008). Moriarty and Smallman, (2009: p. 484) stated that “the ‘locus’ of benchmarking lies between the current and desirable states of affairs and contributes to the transformation processes that realise these improvements”. Furthermore, Moriarty (2008) discussed the fact that although the literature on benchmarking is overwhelming, research and evidence on the value and im- pact of benchmarking is still missing. The criticisms of bench- marking are, according to Moriarty (2008), based mainly on lack of information, difficulties with implementation and a lack of theory. He stressed that benchmarking may require another definition, and that benchmarking is intended to be a means towards the end of achieving a more desirable organisational state of affairs. Benchmarking may identify the changes which are necessary to achieve that end. The concept of change seems to be inherent in benchmarking. Benchmarking is, however, not just about change, but about improvement, or as Harrington (1995) put it, “all improvements is [sic] change, but not all change is improvement” (cited in Moriarty, 2008: p. 29). Mo- Figure 1. Illustration inspired by the benchmarking process described by Camp, 1989. riarty continued by stating that benchmarking is not just about making changes, as it is more about identification and success- ful implementation. Therefore, he suggested a provisional defi- nition: “Benchmarking is an exemplar-driven teleological proc- ess operating within an organization [sic] with the objective of intentionally changing an existing state of affairs into a supe- rior state of affairs.” (Moriarty, 2008: p. 30). Benchmarking is a rather new phenomenon in higher educa- tion, especially with regard to e-learning (Bacsich, 2009a; Os- siannilsson, in press; Ubachs, 2009; van Vught, 2008a, 2008b). Quality assurance, quality indicators, benchmarks and critical success factors for e-learning have neither been taken seriously nor incorporated into regular quality assurance procedures in higher education. These concepts have not been conceptualised, and are even sometimes taken as being synonymous. The con- cept of quality in e-learning has been discussed, considered and managed in a very disconnected manner, and it has not been embedded in learning and quality contexts (National Agency of Higher Education in Sweden (NAHE), 2008; Soinila & Stalter, 2010). Ossiannilsson also showed in earlier studies (2010, in press, 2011) that there is a lack of experience of the value and impact of benchmarking in higher education. The ESMU has worked with benchmarking in several pro- jects in different areas within higher education. The definition of benchmarking used by ESMU is: “Benchmarking is an internal organizational [sic] process which aims to improve the organization’s performance by learning about possible improvements of its primary and/or support processes by looking at these processes in other, bet- ter-performing organizations” (van Vught et al., 2008: p. 16). As shown in the definitions above, benchmarking is very much a process designed to enhance quality, to identify gaps and to bring about the implementation of changes. Benchmark- ing initiatives are often conducted as self-evaluations, including systematic data and gathering information from predefined benchmarks, as well as formulating roadmaps. The goal of 1Xerox Corporation (1979). Xerox. The Benchmarking Story.URL (last checked 10 June 2011) http://www.icmrindia.org/free%20resources/case- studies/xerox-benchmarking-2.htm
E. OSSIANNILSSON 210 benchmarking is to formulate, together with other colleagues, strengths and challenges for the purpose of improvement (Os- siannilsson, in press; van Vught et al., 2008b). The benefits of benchmarking were expressed by the ESMU in 10 statements: “self-assess institution, better understand the process, measure and compare, discover new ideas, obtain data to support deci- sion-making, set targets for improvement, strengthen institu- tional identity, enhance reputation, respond to national per- formance indicators and benchmarks [and] set new standards for the sector” (van Vught et al., 2008b). Participating in a benchmarking process can potentially lead to change in the area being investigated. In addition, an in- creased awareness, both individual and collective, of the or- ganisation itself occurs as a result of participation, which can be considered as a direct and substantial value (Ossiannilsson, in press, 2011). This awareness can often lead to reflections. Re- flections are crucial and can even be a method for organisa- tional change (Højrup, 2004; Ossiannilsson & Landgren, in press). The Concept of E-Learning Information, Communication and Technology (ICT) has be- come embedded in the social and economic sectors, and ought to be similarly embedded in the education and training systems. However, new and more efficient ways of operating, supporting pedagogical and organisational innovation need to be found. In the twenty-first century and with the new millennium learners existing in a digital world, the e-phenomenon has to be embed- ded in all learning and educational activities in order to push the boundaries in daily life, in a global sustainable environment (Bates, 2010; Bonk, 2010; Conole, 2010; Ehlers & Pawlowski, 2006). Hence, several scholars have argued that there is no longer a need for definitions, as e-learning has implications on a vast number of fields (Ehlers & Schneckenberg, 2010; John- son, Smith, Willis, & Haywood, 2011; O’Reilly & Batelle, 2009). However, research has shown that respect has to be given to critical success factors in educational environments in order to succeed in the field of e-learning and to benefit learn- ers, teachers and management (Ossiannilsson & Landgren, in press). McLoughlin and Lee (2008) explored success factors under the headings of personalisation, participation and produc- tivity. Bonk (2009) went a step further, and expressed this con- cept as ubiquitous learning (u-learning), with the focus in the ‘you perspective’ on personalisation and the learner’s rights and responsibilities. The EADTU emphasises within their E-excel- lence Associates label four success factors regarding e-learning, namely accessibility, interactivity, flexibility and personalisa- tion (Ubachs, 2009). Connectivism is also considered as an- other concept which is essential to success i.e. that knowledge is distributed across networks of connections, and learning therefore consists of the ability to construct and traverse these networks (Siemens, 2005). The concepts and success factors related to e-learning in the twenty-first century will surely change the current learning climate, and may have an impact on how benchmarking e-learning in higher education will be con- ducted in the future and the kinds of quality-related issues which matter (Ossiannilsson & Landgren, in press). Management is fundamental regarding the integration and implementation of ICT (Ubachs, 2008; Sangra, 2008). Sangra (2008) stated that the integration of technology, organisation and pedagogy is crucial for success, as all of the elements are needed to increase productivity and processes. New challenges for universities in the twenty-first century include bringing together all of the aspects of e-learning in a holistic framework, and perceiving these concepts in context (Ehlers & Schnecken- berg, 2010; NAHE, 2008; Soinila & Stalter, 2010). Bates (2009) argued for the need for experimentation, innovation and vision where there are challenges, in order to bring together three competing factors: increasing access; increasing quality or im- proving outcomes, and reducing costs. The Projects Two European benchmarking initiatives regarding e-learning were recently implemented. The first one was the E-xcellence+ project, carried out during 2008 through the EADTU (Ubachs, 2009). The second was an e-learning benchmarking exercise, carried out during 2009 by the ESMU in collaboration with the EADTU (Comba et al., 2010; Williams & Rotheram, 2010). The focus of this article is on these two projects. Neither the research, nor this paper, will primarily focus on single bench- marks, indicators, critical success factors or methodologies as such, but rather on the value and impact of benchmarking on e-learning. Both initiatives aimed to identify good practices with regard to e-learning by learning from other participants. The E-Xcellence+ Project The EADTU (Ubachs, 2009) initiated the E-xcellence project in 2004 as part of the e-learning programme on behalf of their members, who had indicated that specific e-learning criteria were missing from the current quality assurance systems in place in Europe. The E-xcellence instrument was developed in co-operation with The European University Association (EUA), United Nations Educational, Scientific and Cultural Organiza- tion (UNESCO) and The European Association for Quality Assurance in Higher Education (ENQA), in order to comple- ment the existing quality assurance systems in higher education, rather than to interfere with them. The project is more explored and investigated in more detail in research papers by Ossian- nilsson (2010, 2011) and by Ossiannilsson and Landgren (in press). The quality benchmarking assessment instrument which was developed covered pedagogical, organisational and techni- cal frameworks, with special attention being paid to accessibi- lity, flexibility and interactivity. After the piloting phase, and under the E-xcellence Associates label, personalisation was also highlighted. The instrument was based on a manual covering 33 benchmarks on e-learning, across three main areas: manage- ment (strategic management), products (curriculum design, course design and course delivery) and support (student and staff support), with indicators relating to benchmarks, guidance for improvement and references how to achieve excellent levels of performance. In addition, assessors’ notes were included to provide a more detailed description of the issues and ap- proaches, as well as the tools, comprising the online instrument, the quick scan and the full assessment. E-xcellence+ was pilot- ing during 2008 at local seminars (Ossiannilsson, 2011), and three universities carried out the full assessment, together with site visits. Several universities carried out the quick scan. Uni- versities who conducted the full assessment on the level of excellence and underwent site visits, worked out roadmaps and committed themselves to carrying out the benchmarking pro- cess for e-learning in higher education every other year awarded
E. OSSIANNILSSON 211 the E-xcelle nce Associates label. The E-Learning Benchmarking Exercise The e-learning benchmarking exercise, devised by the ESMU in co-operation with the EADTU (Comba et al., 2010; Williams & Rotheram, 2010), was carried out by nine European universi- ties during 2009-2010. This initiative combined the ESMU’s collaborative benchmarking approach through a comparison of the good practices of different universities with the EADTU’s more individualistic approach, as described above. The exercise started with the self-evaluated quick scan from the E-xcellence+ scheme (Ubachs, 2009). Furthermore, two workshops with in- ternational experts were carried out, and roadmaps for im- provement were designed by the partners. In order to find some kind of common point of departure within this partnership, e-learning was considered to be.2 Material and Methods In order trying to explore a multifaceted phenomenon in depth, this study used an exploratory multiple case study stra- tegy, as described by Yin (2003, 2009). A mixed-methods ap- proach was applied, utilising a combination of qualitative data sources and integrated methods for data analysis (Yin, 2003, 2009; Creswell & Clarke, 2007). The data were mainly pro- cessed using cross-case analysis, with embedded and multiple units of analyses. Respondents The cases for the current study were selected from both pro- jects: E-xcellence+ (three respondents) and the e-learning benchmarking exercise (four respondents). One of the respon- dents from the latter project took part in both projects (Table 1). The respondents from the E-xcellence+ project were chosen because they had been the first in Europe to be awarded the E-xcellence Associates label. All nine partners from the e- learning benchmarking exercise were invited to take part in the study, and five of them agreed (however, one agreed too late to be included in this paper). In the following work, the respon- dents are not called by their real names, but instead are called Alpha, Beta, Gamma, Delta, Epsilon and Zeta. Data Collection and Procedure A case study protocol was worked out for the data collection procedure, which was then sent to the respondents in advance. The procedures were carried out so as to increase the reliability of findings and to maintain the chain of evidence, in line with the work of Yin (2003, 2009). Existing survey data. Basic information about the respon- dents and their universities was found out in advance, and more information was gathered throughout the projects and their completion. The experts’ reports from the benchmarking exer- cises were used as additional data (Comba et al., 2010; Wil- liams & Brown, 2009; Williams & Rotheram, 2010). Additional data for this study were collected through in- Table 1. Respondents involved in the benchmarking projects, EADTU E-xcel- lence+ and the ESMU e-learning benchmarking exercise. Respondents E-xcellence+ e-learning benchmarking exercise A Alpha x B Beta x Γ Gamma x Δ Delta x Ε Epsilon x Z Zeta x x depth qualitative interviews or more as narratives recorded through Adobe Connect. One interview was carried out at the respondents’ office (in this case, two respondents were both involved in the dialogue throughout the interview, which is also the way in which they work and teach) and recorded with a manual audio recorder. The interviews were typed out in verba- tim. Following the case study protocol, open-ended questions on the following main themes were used for the interviews in order to capture the respondents own experiences: Why did your university joined the benchmarking project? What was your opinion of the process after its completion? In your opinion, where there any drawbacks within the pro- ject process? Do you have any additional thoughts regarding the in- volvement in the benchmarking project? Data Analysis and Process The existing survey data served to illustrate the case studies. Subsequently, during the analysis process, the data sources were analysed and integrated in order to contextualise the find- ings (Creswell & Clarke, 2007) and to elucidate the issues re- garding the value and impact of benchmarking on e-learning in higher education. Each case was first analysed on an individual basis, followed by cross-case analysis (Miles & Huberman, 1994; Yin, 2003, 2009). The qualitative data from the cases were processed in the following way. In order to obtain a gene- ral and holistic sense of the whole situation and to become fa- miliar with the information in the cases, the recordings and the transcriptions of the in-depth interviews were repeatedly lis- tened to and read through case by case. Listening to the mate- rial repeatedly reminded is of its meaning and content, but also gave a sense of the feelings and emotions contained in the nar- ratives. The essential statements which had been made i.e. re- garding the significant issues (natural units) were distinguished (Yin, 2003, 2009). The words were counted and a “wordle”3 was worked out (Figure 2). The text, case by case, was further condensed into units of meaning in relation to the respondents’ expressions which served to highlight certain issues. Using this method ensured that the findings stayed close to the original data. The findings were aggregated or clustered according to key issues, reflecting the value and impact of taking part in 2“…covering a wide set of applications and pedagogical processes sup- orted by ICT learning, such as web-based learning, computer-base learning, virtual classrooms and digital collaboration with an added value of increased accessibility, flexibility and interactivity.” (personal communi- cation, ESMU workshop, 25 May 2009. 3Wordle a tool for generating “word clouds” from text that you provide, the cloud give greater prominence to words that appear more frequently in the source text.
E. OSSIANNILSSON 212 Figure 2. An example of a Wordle made from the interview texts. benchmarking exercises. The analyses produced a summarised description of each case. The descriptions were intended to re- flect the respondents’ narratives and core experiences as closely as possible. The findings begin with an overview of the six cases, and then the findings will be presented case by case. Finally, a cross-case analysis, exploring similarities, differences and variations, will be presented at the end of the findings. Findings Overview of the Cases The six cases and all of the respondents were from research intense universities and so-called traditional universities which employ a blended approach. Three of them were in a process of transformation, in that they were merging universities and fa- culties, which was one of the reasons for participating in the benchmarking exercises. They were also in the process of for- mulating e-learning strategies. Three of the respondents were from universities in which e-learning was not separated from other forms of learning e.g. in policies and plans; however, they strongly emphasised new technologies and e-resources through- out the university. The universities had well-known interna- tional reputations for e-learning and distance education. In the report from the e-learning benchmarking exercise, the experts noted that participation in benchmark exercises always involves reflections on the part of the institutions involved on the lessons learnt and on new approaches and methods that the experience can be expected to bring. They suggested “...inte- gration of benchmarking into strategic planning and to conduct benchmarking exercises as a regular practice to support on- going organisational evaluation and retaining a competitive edge. Such a change agenda depends crucially on strong lead- ership to set clear directions and ensure their implementation. Highly-performing higher education institu tio ns use a variety of tools, including benchmarking, to better understand their op- erations and progress towards increased performance” (Wil- liams & Rotheram, 2010: pp. 29-30). The experts also empha- sised the important lesson that, in reality, the most important work in the benchmarking process actually starts when the benchmarking procedures are over and the implementation phase begins i.e. when changes are made with regard to human capital and implemented within the organisation. In the report by Williams and Brown (2009), they pointed out, in addition to their findings from the site visit, some prob- lems with international benchmarking. Some of the problems to be aware of are, for example, the fact that benchmarks often relate to more than one point and thus require a multi-faceted response and a team-based approach. An important issue which they noted concerns the interpretation of languages and linguis- tics, as well as cultural interpretations. In this case (E-xcel- lence+), some benchmarks were also lacking e.g. library re- sources and student involvement, which were reported back to the EADTU. Williams and Brown (2009) also expressed that university campus courses would gain a great deal if they had the same transparency as the e-learning courses which were involved in the benchmarking process. They emphasised the importance of commitment at the management level in question when starting a benchmarking exercise, as benchmarking is about the implementation of results and (eventually) about change. The Six Cases The findings from the cases are summarized and presented below case by case. Each case starts with setting the scene fol- lowed by their experiences expressed in their narratives. Alpha Setting the scene. Alpha expressed two reasons for joining the project (the e-learning benchmarking exercise). One of the reasons was the desire to be part of one of the leading universi- ties in the country within the e-learning arena i.e. to measure performance and to measure value for the money invested. Some ranking aspects were also considered. Another reason was to learn from others and to network for the purposes of further collaboration. Being a multi-campus university creates challenges when it comes to staff and student support and the organisation of competence development schemes for both administrative and teaching staff in the field of Internet-based communication, collaboration and learning. At the same time, the multi-campus situation provides a natural driving force for the implementa- tion of online technology that can help to save time and money on transportation between campuses. Experiences. Alpha expressed and referred to the ten sum- marised values which were described above by van Vught et al. (2008b). However, Alpha expressed that one of the most im- portant values was possibly decision-making (i.e. during and after a benchmarking exercise, when an institution receives basic data and documentation). Other core values were per- ceived as an increase in awareness and understanding at the individual, collective and organizational levels. This is a sus- tainable value, as it involves the internal involvement of stake- holders within the organization. Alpha also expressed that they had obtained both explicit documentation and also tacit know- ledge and understanding which, due to the benchmarking exer- cise, became explicit. In addition, Alpha articulated that when one undertakes a benchmarking exercise with benchmarks to relate to, and holds discussions and interviews with stake- holders in an organization, a collective picture of the situation is gained, which had a fairly large impact. Relationships were strengthened and team members became closer to each other. According to this respondent, benchmarking is very much a question of involvement. This has resulted in spin-off effects in other projects within the organization. Alpha also expressed that one of the lessons learnt was that, during the benchmarking itself, a great deal of attention was devoted to gathering data, collaborative discussions, writing reports and formulating action reports with a lot of valuable
E. OSSIANNILSSON 213 help received from partners and experts. Conversely, less atten- tion and support was gained during the implementation phase. Choosing stakeholders is another important issue, especially with regard to organizational learning, when the management needs to be involved. Alpha stressed the importance of com- mitment at the managerial level. This is probably the most im- portant step in terms of organizational learning and develop- ment. Alpha felt that they came out relatively well in the benchmarking exercise, both in terms of their self-evaluation and also compared with the others. Accordingly, they learnt that they needed to look even more at their practices, and that improvement and consolidations could be made. Additional lessons were to consider a more constructive approach to learning in relation to e-learning, and that learning needs to be more deeply embedded in educational processes, i.e. to focus on how learning can be enriched by technology, rather than how technology can enrich learning. The fact that a new project had been created within the organization with the aim of im- plementing some of the results of the benchmarking exercise was an explicit result. The values of the exercise, for Alpha, lay in extended networking with partners in the project, which had led to spin-off effects such as new projects and collaborations. Alpha emphasised the importance of benchmarking as a method for quality assurance, as it involves self-evaluation and learning through and with others. Nevertheless, it requires a large amount of resources. Alpha put forward the idea that, in this context, it is probably of value to consider the content and the number of benchmarks. The drawbacks were that there were too many extensive benchmarks. Another issue was the linguis- tic sense, (the benchmarks were in English), but also in the translated form, there were different cultural interpretations among the partnership, but also within the organization. Beta Setting the scene. One of the main reasons why Beta joined the project (the e-learning benchmarking exercise) was the desire to be able to compare their e-learning efforts with those of other universities. Other reasons to participate were a need to gain new inspiration and insights, and also to learn from others in a spirit of collaboration. Beta is also a multi-campus univer- sity, and so the same arguments apply as in the case of Alpha. Experiences. Beta expressed a whole range of benefits for their university. On a personal level, Beta learnt a lot about their organization. An unexpected advantage was that they be- came aware of some new contacts, for strategic and practical work, even within the organization. They also appreciated that they gained a good overview of what they were actually doing within the field of e-learning, as well as their strengths and weaknesses, what could be improved and what targets to set for the years ahead. Beta expressed this as follows: “I think it sur- prised me how many different benefits on so many different levels in the organization we actually gained. It was valuable to focus on the entire organization and the future of e-learning in the university.” At a managerial level, the project report was seen as important, as the university came out with good results. This also showed that the resources allocated to the area of e-learning at the university were meaningful. It focused their attention on quality and the need to link e-learning to quality management efforts and to secure the widespread use of e-learning. Within Beta’s quality organization and quality com- mittees, they ensured the involvement of stakeholders and the dissemination of the project. With this level of organization and commitment, it was easier to work with the lessons learnt. Though, this process was not perfect from the very beginning, but they caught up in many ways during the process. From the start, Beta was under pressure to come out as one of the best universities in the partnership. Conversely, having gone into the process and learnt more and more about the organization, Beta felt that “the lessons you learn and the knowledge you gain in the process is perhaps the most valuable part of the benchmarking process”. Of course, they were happy to point out their strengths, but also to record that there were some points which needed improvement, “so you do not just rest on your laurels”. Beta also gave voice to the fact that the benchmarking pro- cess and results motivated people to developments, either by continuing with current developments or through innovation. Beta articulated that there were some limitations to the ways in which the benchmarks were formulated, in that they were pragmatic and depended on a certain way of looking at e- learning. Thus, Beta would recommend more open-ended ques- tions and also more time to work through and revise the benchmarks within the partnership. It should be more like a joint effort and a collaborative approach. It was also suggested that time frames could be added for the further development of the process, i.e. time allocation for follow up processes. In addition, Beta emphasised that the successful areas within e-learning are mainly based on backup from top management, with clear and obvious visions and strategies throughout the whole organization. The development of teachers’ competences within the fields of learning and blended learning are also of the highest importance, together with pedagogical and technical support. Well-implemented e-learning platforms and tools are necessary. Knowledge sharing and the sharing of experiences both nationally and internationally are furthermore of the ut- most significance. Finally, Beta expressed that funding should be provided for pilot projects or the most innovative projects within a certain area. Gamma Setting the scene. Gamma expressed three main reasons for participating in the project. First, they expressed that they were in the middle of a major merging process, and that they were also reducing the number of their faculties. Second, they ex- pressed that there was a wide variety of e-learning standards and procedures at the university. Getting involved in the ben- chmarking exercise was therefore one obvious way in which to gain an overview of what was going on and a way to com- mence internal co-operation. Finally, their participation was a way for them to generate a roadmap and to get some new ideas, thanks to this international co-operation. They had no expecta- tions regarding how well their participation would go or whether or not they would succeed. Gamma is also a multi- campus university, and so the same arguments apply as were mentioned for Alpha and Beta. Experiences. Gamma articulated that, in principle, bench- marking had a great deal of value and a significant impact, as suggested by van Vught et al. (2008). The value for Gamma was not so much in learning with and from collaborators, as in coming to know one’s own organisation, especially as they were in the process of an extensive merger (so they did not have any working practice to relate to). Therefore, the main
E. OSSIANNILSSON 214 impact upon Gamma was related to values and lesson learnt regarding how they formulated themselves. Gamma expressed that “benchmarking is a great tool to point out flaws, to explore and spot missing areas and to raise attention. It is also a good opportunity to work with best practice and it is also a method to create inspiration”. Gamma is in a phase of reorganization, and due to this they had many ideas relating to the processes of innovation. Gamma suggested that benchmarking ought to be mandatory for universities, either on an annual basis or every other year. Benchmarking should take place in different areas in order to enhance quality and with the aim of being in the front- line. In this case, benchmarking is a great method for educa- tional movement; enhancing quality and working processes in order to relate to best practice etc. in a more structured and more professional manner. Universities need to have bench- marks and indicators in order to speak the same language. In addition, benchmarking allows universities to extend their net- works and to collaborate on setting up new projects or bilateral agreements (spin-off effects). Gamma expressed some pitfalls of benchmarking, which of can be turned into opportunities as and when they became visi- ble. Some pitfalls are language and cultural differences, but also the question of how to set benchmarks e.g. in this case, they were rather unfashionable, so Gamma had to go above and beyond in order to connect to their area and context. In order to improve, it is essential to consider cultures, such as the univer- sity’s own culture but also international perspectives. The dis- advantages also include that benchmarking is time-consuming and demands focus. However, Gamma expressed that if one spends time and focuses on the work in a benchmarking exer- cise, one will get more out of it and therefore gains more from it: “What comes in goes out”. Gamma also stated that they now focus more on learning and how students interact with information technology (IT) and the social media in their context. This raises questions about the contextualization of learning and education. Therefore, in the event that a benchmark exercise should be set up again, they will consider these phenomena to a greater extent. Delta Setting the scene. Delta expressed that as they were selected to take part in the benchmarking project (E-xcellence+), they naturally appreciated the opportunity and wanted to join in. They were also curious with regard to obtaining an awareness of how far they had reached, even though they had established an international reputation many years ago, and had many in- ternational students enrolled. Second, they saw the advantages of the process, such as obtaining an internal awareness and knowledge about their own organization. The third driving force was the fact that there was an opportunity to attain official in- ternational recognition; this, of course, was a motivating fac- tor. Another motivating factor was the future situation of the university, as they from 2011 have to charge tuition fees for students outside Europe. Competition will thus be more explicit, and there will be an even stronger drive to ensure that students get what they pay for. Being internationally distinguished is, in this context, very significant. Experiences. Delta was the first university in Europe to be awarded the E-xcellence Associates label by EADTU as a con- crete recognition of the high quality of their e-learning pro- gramme. As a result of this, the level of their national and in- ternational collaboration, as well as student recruitment, was increased. The real impact of this will first be noted in a few years when the first students (who are paying tuition fees) en- roll and they are asked why they chose this programme instead of others. In addition, students who are already enrolled are impressed and appreciate being enrolled in a programme which has been labelled as excellent. They have also articulated a positive response to the educational culture and level of profes- sionalism they have experienced. Delta also emphasised that they had received an increased level of recognition for their work at different levels within the university, which has re- sulted in spin-off effects. Delta stated that, in terms of their positive experiences, they have had the opportunity to undertake the benchmarking and internal processes, which increased their understanding and knowledge. To have the chance to reflect and to share within the centre, but also with colleagues within the university, was very valuable, and made their work more effective. In addition, it was valuable to perform a self-evaluation, as well as the documentation and the experts’ report. This was also a way for Delta to gain an awareness of the areas in which they fall short. In addition, they had opportunities to review and improve their routines. Thus far, they have not been involved with or taken part in the Excellence Associates club, but they will in the future and can see the advantages of such a network. The roadmap that they worked out has served as an inspiration and as a driving force. It is not a formal action plan, but a working plan and an inspirational document. They expressed that they have either worked through some of the actions or at least considered them. It is an honest document, referring to what they ought to do, considering time restraints and financial opportunities. They articulated that, during the process, they benefited from support and commitment from the management team, and they also stated that, without this support, it would be more or less im- possible to carry out changes or improvements in relation to the benchmarking exercise. Other issues were that the exercise was so time-consuming due to the need for data gathering, internal discussions and analyses that support and commitment were needed. They expressed that they had experienced continuous support, even now the real work has started with the imple- mentation and transition phase. Delta emphasised the importance of involvement and team- work during the process. The benchmarking exercise was as such also important due to the need to “attract attention, gather and spread information and engage staff and students (three core concepts”. Delta articulated that the benchmarks were comprehensive, but maybe too comprehensive. On the other hand, this contributed to discussions, interpretations and the development of knowledge within their own educational culture and context. If Delta were to undertake the exercise (or a simi- lar exercise), again, they stated that it would be valuable to estimate time requirements, and probably also to have a more explicit overview of the process. The drawbacks of the exercise, when one is sensitive to them, can be turned into opportunities. Another issue is whether the university could or should market and raise awareness of the benchmarking process to an even greater extent. Epsilon Setting the scene. Epsilon indicated three incentives for their
E. OSSIANNILSSON 215 participation (in E-xcellence+ and the e-learning benchmark- ing exercise). First, they expressed that they wanted to obtain an awareness of what was state-of-the-art in the area. Second, they wanted to learn from others and to obtain an awareness of their progress in the area in an international context. Finally, they expressed that participating was a way to obtain insights and knowledge with an eye to the future. They also expressed that their participation was a way of gaining both internal and external processes. They stated that benchmarking is an optimal method for raising awareness and understanding of what is going on, both internally and exter- nally. Both advantages and disadvantages become explicit, but primarily and most importantly, potential areas for success and where quality can be enhanced become visible within the com- mon understanding in an organization. Experiences. Epsilon expressed having found the exercise to be valuable both internally and externally, and that lessons can be learnt in and through both dimensions. For Epsilon, the pro- jects had been extremely valuable. Benchmarking is a way of watching what is happening in the surrounding world. Regard- ing Epsilon’s participation, it can be stated that they found it to be valuable on several levels; within departments, on the uni- versity management level, and on national and international levels. One example which was articulated several times in the in-depth interview was the fact that a new programme is going to be set up in collaboration with several other universities in the country. With this programme, there is a need for high qua- lity work to be carried out with new technologies, not just for the purposes of content and inter-university collaboration, but also to create a model and to become part of an IT society in the twenty-first century. Even within other areas of the university, schemes are being implemented as result of the benchmarking projects in combination with the current discourse in this area. Epsilon articulated that they found it interesting how many outstanding things had happened throughout their faculties and departments; it seems likely that these things have happened due to the fact that Epsilon is a decentralised university. They therefore expressed that freedom, enthusiasm and development on demand are crucial for a reflective and growing organization. The key is not always efficiency, as sometimes quality en- hancement is more important. Both bottom-up strategies and top-down strategies require flexible methods. Benchmarking helps universities to choose a direction. At a research intensive university, where the teachers and programmes are at a variety of levels in terms of IT proficiency, from early beginners to those who excel, there are a variety of demands and require- ments; however, there needs to be something for everyone, even at a central organizational level. The most important thing at a large university is to develop maturity and “teaching envi- ronments to learning environments to web environments and also that it [e-learning, new technologies, social media etc.] is embedded in [the] curriculum”. This journey has to take time. An awareness of maturity is of great importance, as if an or- ganization is immature, then no development will happen. There needs to be maturity, at both individuals and organiza- tional levels. Each faculty/department has their own structures for dealing with demands and cultures. This is good. In addition, one has to make choices, and one has to be familiar with e- learning and technology development. When maturity has been reached, co-ordination can begin. The fact that two of the programmes at Epsilon were awarded the E-xcellence Associates label indicates that they have been recognised both internally and externally. For the programmes that were awarded the label and already have large groups of international students from outside Europe, this award is very valuable and will have a significant impact as they are facing a new situation regarding tuition fees for students from outside European Union. Of course, this is due to other programmes as well. The E-xcellence Associates label supports the entire uni- versity in its strategic work towards achieving consolidation, internationalization and distinction, as universities are striving to be world class. Epsilon stated that they valued all of the documentation which had been a part of the benchmarking projects, which they will now use for other purposes. One ex- ample is that when the National Agency for Higher Education conducted a survey of distance education in the country (Swe- den), then the benchmarking reports supported the documenta- tion required for this survey to a large extent. Epsilon stressed the fact that e-learning needs to be user- friendly, integrated and embedded into courses and educational arenas. Epsilon stressed further the importance of freedom in a large organization, as creativity grows and things happen when there is freedom. Zeta Setting the scene. The main reason why Zeta chose to parti- cipate in the project (E-xcellence+) was to ensure the quality of its education and courses and to provide evidence that these were of the highest possible quality. The main reasons for this are due to student rights and student satisfaction. Other reasons were of both an internal and external character. The internal reasons were a desire to raise awareness of advantages and disadvantages concerning range, support and implementation of the e-learning arena at Zeta, and in addition student satisfaction, and through the process find gaps for possible improvements. The external reasons were mainly a desire for international co-operation and networking, but also an attempt to learn from and through others. As benchmarking is a form of self-evalua- tion, it is more honest and the motivation to make changes is thus higher, as any criticism tends to be more constructive. There were also other reasons, such as the fact that each mem- ber of the staff including the students need to feel appreciated for their work. It is also important to note who the actors are who initiated and took part in the benchmarking e.g. in this case, the benchmarking was initiated by the EADTU, the National Agency for Higher Education and the university itself. The chance to attain the E-xcellence Associates label was also an incentive to join the project. Experiences. Zeta expressed that through the benchmarking project, advantages and disadvantages concerning range, sup- port and implementation of e-learning became visible and ideas for changes were considered. The fact that Zeta earned the cer- tification and the label was, of course, very encouraging, and the label demonstrated that the university had been recognised internally as well as in international collaboration and net- working. In addition, more students are being recruited. This is, of course, a triumph, especially when the university was co- operating with countries that tend to distrust distance education. However, Zeta expressed that they might be able to market distance education to an even greater extent, but they already have more than enough students (i.e. more than they get paid
E. OSSIANNILSSON 216 for). Naturally, this was an occasion for self-evaluation and this influenced all staff and their work. Of course, the label as such is not as important as the knowledge and understanding of the work which the university is doing, as well as an awareness of their shortcomings. Nobody is happy with just the award itself. Zeta, therefore, articulated a very humble attitude towards its recognition. However, two of the leading staff at Zeta awarded the university a prize for excellence in teaching and pedagogy in the following semester. Zeta stated that the roadmap served as inspiration and not as a dogmatic document. Some of the actions have already been taken and some are being considered, due to limited time and resources. They expressed that they have been supported by the university, although perhaps not always at the closest adminis- trative level i.e. the faculty level, probably due to a degree of mistrust, as they are too far ahead and too innovative in their course development and delivery. Maybe this had been the fact anyway, if they had been innovative and therefore succeeded in other areas, they would have experienced mistrust there as well. Concerning the benchmarks, Zeta expressed that they were particularly extensive and that they worked in teams in order to get the staff and students involved. They used micro and mini meetings to go through everything and to discuss different as- pects of the process. Zeta gives voice to the idea that one of the most important issues is flexibility at all levels and dimensions in any attempt to satisfy students. The ultimate goal is to satisfy students, and they frequently demand a high degree of flexibi- lity. Satisfied students are important quality indicators. Flexible and boundless education with regard to pathways, time (even starting time), free resources, flexible materials/resources (texts, audio, video) etc. is of paramount importance and forms the foundation of the programme. Student evaluations have also shown that this flexibility is crucial. One explicit impact was an inspection of the learning plat- form and the overall university administrative system in order to make communication more flexible. Zeta stated that when e-learning programmes begin, the most important aspect seems to be the learning dimension. This is natural, but when courses are scaled up, it is crucial that administrative systems commu- nicate well. Thus far, Zeta has limited experience of the E-xcellence As- sociates club, due to limited time and human resources. Obvi- ously, such initiatives are of the highest importance. Zeta’s final comment was that universities which are striving for the highest quality in an international field should certainly con- sider benchmarking. Cross-Case Findings Overall, the findings indicated many similarities between the universities, but also differences, or rather different ways of expressing experiences across the cases. The cases illustrated how different aspects interact to a varying extent and indicated the value and impact of the international benchmarking exer- cises which were conducted. In the section below, some of the key areas are summarised. Internal Processes and Involvement The benchmarking allowed the teams to maintain their focus. Involvement and shared responsibilities in the work with bench- marking enabled not just the benchmarking itself, but also con- tributed to commitment and appreciation among the co-workers within the institution. The fact that advantages and disadvan- tages in the field of e-learning became explicit supplied the motivation to make the changes which led to the impact of the scheme. In addition, the fact that the exercise was mainly a self-evaluation motivated the institutions to examine even nega- tive issues, which often could be turned to challenges. In addition, the solid documentation which was collected, as well as the knowledge and institutional awareness regarding e- learning was important and led to a significant impact. The institutions gained a lot from the solid work with its transparency, which could even be used and valued in other contexts. The awareness of the infrastructural support for e-learning, which was one of the results of the benchmarking exercise, was made explicit and this led to closer collaboration between the infra- structural units in question, as well as further collaboration with other faculties and departments, for example within pedagogical areas and e-resources. Additional benefits were clearly expressed regarding teamwork, creating a dialogue, policy making, quality assurance and transparency within the organization. Quality Enhancement Benchmarking is an advanced method, but the cases have in- dicated that it is quite easy to use with explicit opportunities for self-reflection. Even though certain steps have to be taken, one can set the structure within the partnership as either a smaller or a larger exercise; one can also define the benchmarks, partners and the time schedule. Nearly all of the cases reported that enough time and support was allocated during the project period. However, they emphasised that they misallocated time and support for the reflection and implementation phases. This process needs to be followed by reflections on the lessons learnt, which can provide incentives for further development in the institution. This may involve changes in structure, organisation and resource allocation, which may require strong leadership at all levels. Furthermore, the importance of integrating bench- marking processes as a natural part of strategic quality assurance work was emphasised. The value of continuously following up earlier benchmarking exercises as well as committing to new projects was highlighted. All of the respondents agreed on the value of benchmarking as a method for quality assurance. Management and Commitment The cases showed, to a significant extent, the importance of full support and commitment from all levels of the management team during the whole benchmarking process. Such support is necessary in order to maintain the focus on the project, to allow staff and students to be involved, to work in an interdisciplinary manner and award status for the ongoing work and dissemina- tion during the process, but most of all with regard to data gathering and reports i.e. the implementation phase, if changes have to be made. In these cases, several departments were in- volved in full. They were therefore able to focus on and discuss common areas and processes, creating togetherness, trust, com- mitment and involvement. This will certainly contribute in turn to enriching the future employment situation and potential areas for development. Collaboration and Networking When taking part in benchmarking exercises, universities have the chance to meet other institutions that are facing the same challenges as themselves. There are opportunities to ap- proach one another, to learn from best practices and to take part
E. OSSIANNILSSON 217 in joint discussions on how to handle challenges. This can evolve into continued benchmarking on a smaller scale, where two or more universities from the group collaborate and formulate new benchmarks. Such new benchmarks can help the institution to improve important areas and can play a major role in continuing quality assurance efforts. On a more practical level, the know- ledge exchange between benchmarking participants can lead to mutual inspiration and can help the individual universities to understand and handle topical issues within the field of e- learning. In the benchmarking exercise, it became apparent that each university possessed best practices within certain areas that were of importance for all of the universities. These areas included pedagogy, technology, strategy etc. In the benchmarking exer- cise itself, there was unfortunately no time to study the best practices of the other universities in depth. However, the exercise itself provided a good overview of the strengths of each individual university. Mutual trust and under- standing has been gained, and future activities such as joint applications and agreements for further co-operation will natu- rally be established. A network has come into being, in which participants can use each other as experts when knowledge of a certain topic is needed. Cultural Issues As this was an international benchmarking initiative, English was the chosen language. All of the respondents raised a prob- lem with this, as none of them had English as native language. It was not just the language and linguistics as such, but also cultural aspects of the language and how it is used in different contexts. The benchmarking was mainly carried out in teams, and even within the teams there were differences in how certain concepts were understood. Another issue was the somewhat old-fashioned concepts regarding e-learning. Nearly all of the cases work within a blended mode context. In addition, nearly all of the universities in question are decentralised, and thus there was not only one way of acting and performing. Attitudes Through the in-depth interviews, all of the cases voiced their enthusiasm and positive feelings regarding the exercise itself, but also regarding the methods. It was, for example, noted that during a comparatively short time, many lessons had been learnt, both about the institution themselves, but also about wider perspectives including other universities and the sur- rounding international context. Furthermore, it was articulated that they had also gained a deep understanding and knowledge in the area of e-learning through the benchmarks and indicators, areas which to some extend seemed to be tacit knowledge. In addition, they expressed their enthusiasm and interest in con- tinuing to work, or as they put it, they had the energy to tackle future challenges. All of the respondents expressed that it would be useful to carry out the exercise again, but maybe on a smaller scale with time to define their own benchmarks in a more useful way. Discussion This study explored the experiences of respondents who had taken part in an international benchmarking project. The inte- grated case study findings provided critical information on the value and impact of benchmarking in higher education. Most importantly, this study showed the complexity of the value and impact of taking part in such exercises. The complexity of benchmarking e-learning in higher education is based on the fact that values can refer to either immediate or more long-term values. Furthermore, these values can apply to different aspects and different levels, and to both individuals and organizations. Moreover, some effects take years to evaluate. One contribution of this study to our present knowledge and understanding of benchmarking as a method seems to be a recommendation that time, resources and maybe even the use of experts and/or co-partners is allocated for the implementation phase, as this is an important step in the benchmarking process. All of the respondents in the study clearly emphasised the lack of support they had received afterwards. Another contribution is the apparent importance of commitment from managerial levels. In the event that structural changes or other changes have to be made, resources will probably need to be allocated or re-allocated, and due to this it is extremely important to have commitment from the leadership. This works both ways; if the institution produced as “good result”, it is essential to commu- nicate and market such benefits, internally as well as externally. It is never easy to speak up for oneself, and so support for mar- keting is often needed. We are facing a paradigm shift within the twenty-first cen- tury, and e-learning as a phenomenon was considered among the respondents to have a very broad meaning. Therefore, the learning dimensions were emphasised. Changes in technology have to be considered in education, even through a social-con- textual approach. This study shows, therefore, that benchmarks must be chosen with care and that linguistic, cultural and con- textual issues have to be taken into account. Even if this study did not aim to focus on benchmarks as such, critical issues within the area were articulated, in accordance with the success factors described by the EADTU (Ubachs, 2008) and Ossian- nilsson and Landgren (in press). The need for flexibility in all of the concepts and dimensions was clearly highlighted. Even the structure of the partnership is crucial, as are similarities and/ or differences within the partnership. How to choose bench- marks that provide direction is one of the aims within the benchmarking process. Trust, involvement, collaborative learning, honesty and courage are of the utmost importance in bench- marking processes for fruitful and sustainable work and col- laboration. As mentioned above, there can be pitfalls, but awareness and thoughtfulness can turn them into challenges. Turning to the theoretical frame of reference, this study showed, as the discourse on benchmarking also shows, that benchmarking is about change, and thus lessons have to be learnt from benchmarking processes (Moriarty, 2008; Moriarty & Smallman 2009; Ossiannilsson, 2010, in press, 2011; Os- siannilsson & Landgren, in press). Once again, as Moriarty & Smallman (2009: p. 484) stated, “… ‘the locus’ of benchmark- ing lies between the current and desirable states of affairs and contributes to the transformation process that realise[s] these improvements”. In this study, the findings on the value and impact of benchmarking as described by the respondents were close to and in accordance with the ten good reasons to conduct benchmarking as described by van Vught (2008b). Some more critical issues were also touched upon, such as the significance of the first and last phases of benchmarking; the identification
E. OSSIANNILSSON 218 and implementation phases. Furthermore, the value and impact of tacit knowledge (Elliott & Stemler et al., 2011) were expli- citly expressed. However, they were not only expressed; the possibility of sharing this tacit knowledge, as well as know- ledge and experiences, was articulated as a possible impact. The case studies indicated the importance of support and man- agement when making changes, as Sangra (2008) also argued. Contributions to the transformation processes that realize improvements require commitment, but also reflection. Through the cross-case analysis, the respondents experiences in this study indicated that a process of philosophical interpretation is required, and that there is a need to modify the theoretical as- sumptions concerning benchmarking. Høyrup (2004) argues, for example, that there is a need for a learning organization to reflect, but he distinguished between reflection and critical reflection. He emphasised critical reflection in particular as one of the core processes of organizational learning. He stated, moreover, that critical reflection was necessary on both the individual and organizational levels. Critical reflection includes social, cultural and political aspects and concerns the ‘why’ (meaning) rather than questions about how to act or perform. The meaning of reflection and the importance of reflective practices for change was also clearly expressed in the study. Another dimension of critical reflection can be seen as Deleuze’s concept of “becoming” (cited in Giger, 2010). Be- coming share the common meaning of becoming which even is true in this context. Thus, the process of becoming is about changes, just like benchmarking is about changes. Becoming, as a more philosophical concept, has something to do with be- coming us as individuals. The rhizome concept, also put for- ward by Deleuze (cited in Giger, 2010), was described in brief as connectivity and multiplicity. Both concepts by Deleuze are new dimensions which are valuable for benchmarking and or- ganizational change. Reflections on this study also touch on limitations of the re- search approach. Of course it would have been valuable to have had all participating universities involved in the study. Probably the findings had been somewhat more valid. On the other hand the findings from the cases involved are seen as representative. Within the six cases variations, but also similarities were found. Conclusion Benchmarking is a powerful strategic tool used to assist de- cision-makers to improve the quality and effectiveness of or- ganisational processes. This study showed that the quality of e-learning has to be considered and must be embedded in suit- able contexts. This study explicitly showed the value and im- pact of benchmarking e-learning in higher education. In the case studies, the respondents expressed that the benchmarking process had a significant impacts i.e. the power to make changes. Lessons from this study have to be taken into consideration. Lessons on how benchmarking is conducted, on objectives, on how it is organized and set up, how to choose partners and how to choose benchmarks must be acknowledged. In addition, how to generate commitment and allocate time and resources during the entire benchmarking process, including the implementation phase, has been emphasised in certain cases. Some characteristics which may not be mentioned as often in a positive light are the intentions, i.e. the meaning of the bench- marking process as such. The involvement of individuals within the organization creates an impact on both individual and or- ganizational levels, not least through critical reflection. The concepts of “rhizome” and “becoming” used in this study have leant other and new dimensions to benchmarking. Thus, this study is not only relevant to management, but to philosophy as well. The study shows that benchmarking, in line with national and international quality boards/agencies, is a powerful tool for making improvements to teaching and learning in higher edu- cation. Acknowledgements I would like to express my thanks to the respondents who participated in the E-xcellence+ project and/or in the e-learning benchmarking exercise. References Bacsich, P. (2009a). Benchmarking e-Learning in United Kingdom universities—The methodologies. In T. Mayes, D. Morrison, H. Mellar, P. Bulen, & M. Oliver (Eds.), Transforming higher education through technology enhanced learning (pp. 90-106). Bristol: The Higher Academy. Bacsich, P. (2009b). Benchmarking e-learning in United Kingdom universities: Lessons learned from and for the international context. Proceedings of the 23rd ICDE World Conference on Open Learning and Distances Education Including the 2009 EADTU Annual Con- ference. M-ICDE Conference Flexible Education for All: Open- Global-Innovative. Maastricht. Bates, T. (2009). Re: Using technology to improve the cost-effectivenes of the academy. [Web log message]. URL (last checked 10 June 2011) http://www.tonybates.ca/2009/10/10/using-technology-to-improve-th e-cost-effectiveness-of-the-academy-part-1/ Bates, T. (2010). Re: Strategic thinking about e-learning. [Web log message]. URL (last checked 10 June 2011) http://www.tonybates.ca/2010/06/11/strategic-thinking-about-e-learn ing/ Bonk, C. J. (2009). The world is open: How web technology is revolu- tionizing education. San Francisco, CA: Jossey-Bass, A Wiley Im- print. Comba, V., Ossiannilsson, E., Landgren, L., Blok, R., Martiens, F. J., Kjaer, C., & Christensen, I.-M. (2010). A benchmarking exercise for quality blended learning. A challenge for European universities in the twentyfirst century. Proceedings of EADTU Annual Conference, Zermatt, 55-75. Conole, G. (2010). Learning design: Making practice explicit. Con- nectED Conference 28 June, 2010. Sydney. URL (last checked 10 June 2011) http://www.slideshare.net/grainne/connect-ed-conole Creswell, J. W., & Clarke, P. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications. Ehlers, U.-D., & Pawlowski, J. (2006). Quality in European e-learning: An Introduction. In U.-D. Ehlers and J. Pawlowski (Eds.), Handbook on quality and standardization in e-learning (pp. 1-3). Berlin: Springer. doi:10.1007/3-540-32788-6_1 Ehlers, U.-D., & Schneckenberg, D. (2010). Introduction: Changing cultures in higher education. In U.-D. Ehlers and D. Schneckenberg (Eds.), Changing cultures in higher education (pp. 1-14). Berlin: Springer-Verlag. doi:10.1007/978-3-642-03582-1_1 European Commission (2009). The Bologna process 2020—The Euro- pean higher education area in the decade. URL (last checked 10 June 2011)
E. OSSIANNILSSON 219 http://portal.bolognaexperts.net/files/Leuven_Louvain-la-Neuve_Co mmunique_April_2009.pdf Elliott, J. G., Stemler, S. E., Sternberg, R. J., Grigorenko, E. L., & Hoffman, N. (2011). The socially skilled teacher and the develop- ment of tacit knowledge. British Educational Research Journal, 1, 83-104. doi:10.1080/01411920903420016 Giger, P. (2010). Conversation and figuration from the horizontality of the 2.0 decade. Ph.D. Thesis, Karlskrona: Blekinge Institute of Technology. Hämäläinen, K., Jessen, A., Kaartinen-Koutaniemi, M., & Kristoffersen, D. (2003). Benchmarking in the improvements of higher education. Helsinki: ENQA [The European Association for Quality Assurance in Higher Education]. Høyrup, S. (2004). Reflection as a core process in organizational learn- ing. The Journal of Workplace Learning, 8, 42-54. Johnson, L., Smith, R., Willis, H., & Haywood, K. (2011). The 2011 horizon report. Austin, Texas: The New Media Consortium. Johnson, M., & Seborg, D. (Eds.) (2007). Benchmarking concepts: An introduction. Process control performance assessment benchmarking concepts: An introduction. London: Springer. McLoughlin, C., & Lee, M. J. W. (2008). The three P’s of pedagogy for the networked society: Pesonalization, participation and productivity. International Journal of Teaching and Learning in Higher Education, 1, 10-27. Miles, M. B., & Huberman, M. A. (1994). Qualitative data analysis. Los Angeles, California: Sage Publications Inc. Moriarty, J. P. (2008). A theory of benchmarking. Lincoln: Lincoln University. Moriarty, J. P., & Smallman, C. (2009). En route to a theory on bench- marking. Benchma r k ing: An International Journal, 4, 484-503. NAHE [The Swedish National Agency for Higher Education] (2008). E-learning quality: Aspects and criteria. Solna: Högskoleverket. Ossiannilsson, E. (2010). Benchmarking e-learning in higher education. Findings from EADTU’s E-xcellence+ project and ESMU’s bench- marking exercise in e-learning. In M. Soinila, & M. Stalter (Eds.), Quality assurance of e-learning (pp. 32-44). Helsinki: ENQA [The European Association for Quality Assurance in Higher Education]. Ossiannilsson, E. (in press). Benchmarking on e-learning in universities: Impacts and values, European perspectives. International Journal of Management in Education. Special Issue on Virtual University. Ossiannilsson, E. (2011). European experiences on the e-xcellence benchmarking initiative by EADTU. In W. Aung (Ed.), iNEER Spe- cial Volume: Innovations 2011—World innovations in engineering education and research. Manuscript accepted for publication. Ossiannilsson, E., & Landgren, L. (in press). Quality in e-learning—A conceptual framework based on experiences from three international benchmarking projects at Lund Univesity, Sweden. Journal of Com- puter Assisted Learnin, Special Issue on Quality in e-learning. O’Reilly, T., & Batelle, J. (2009). Web Squared: Web 2.0 Five Years On. Special Report. URL (last checked 10 June 2011) http://assets.en.oreilly.com/1/event/28/web2009_websqared-whitepa per.pdf Re.ViCa. (2009). Bibliography of benchmarking. URL (last checked 10 June 2011) http://www.virtualcampuses.eu/index.php/Bibliography_of_benchma rking Sangra, A. (2008). Technologies in university: Models, problems and challenges. Ph.D. Thesis, Tarragona: Universitas Rovira i Virgili. Soinila, M., & Stalter, M. (Eds.) (2010). Quality assurance of e-le a r n i n g . Helsinki: ENQA [The European Association for Quality Assurance in Higher Education]. Schreurs, B. (Ed.) (2009). Critical success factors. Reviewing the vir- tual campus phenomenon. The rise of large-scale e-Learning initia- tives worldwide (pp. 59-84). Leuven: EuroPACE ivzw Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology & Distance Lear- ning, 1, URL (last checked 10 June 2011) http://www.itdl.org/journal/jan_05/article01.htm.. Ubachs, G. (2009). E-xcellence quality assessment for e-learning. Heerlen: EADTU [European Association of Distance Teaching Uni- versities]. Van Vught, F., Brandenburg, U., & Burquel, N. (Eds.) (2008a). Benchmarking in European higher education. Findings of a two year EU-funded project. Brussels: ESMU [European Centre for Strategic Management of Universities]. Van Vught, F., Brandenburg, U., & Burquel, N. (Eds.) (2008b). A prac- tical guide. Benchmarking in European higher education. Brussels: ESMU [European Centre for Strategic Management of Universities]. Williams, K., & Brown, T. (2009). E-xcellence site visit report, Lund University. Heerlen: EADTU [European Association of Distance Teaching Universities]. Williams, K., & Rotheram, B. (2010). Report. Benchmarking exercise on e-learning. Brussels: ESMU [European Centre for Strategic Management of Universities]. Yin, R. K. (2003). Case study research. Design and methods. Los An- geles, California: Sage Publications, Inc. Yin, R. K. (2009). Case study research. Design and methods. Thousand Oaks, CA: Sage Publications, Inc.
|