J. Service Science & Management, 2010, 3 : 33 -44
doi:10.4236/jssm.2010.31004 Published Online March 2010 (http://www.SciRP.org/journal/jssm)
33
Evaluating a Student MIS in the Tertiary Education
Sector: Addressing Functional-Operational
Misalignment through Continuous Participative
Evaluation
Marian Carcary
Kemmy Business School, University of Limerick, Limerick, Ireland.
Received September 16th, 2009; revised November 2nd, 2009; accepted December 7th, 2009.
ABSTRACT
The Information and Communications Technology (ICT) evaluation literature spans several decades. ICT evaluation
approaches range from objective and positivistic to subjective and interpretive. While positivistic approaches have
dominated the past, there is increa sing recognition of the va lue of interpretivist metho ds and the need for ongoing pro-
ject evaluation. Formative con tinuous participative evalua tion (CPE) offers several benefits in terms of project control,
enhanced stakeholder relationsh ip s and benefit realisa tion; nonetheless th is is often ignored in practice. There is a pau-
city of ICT evaluation within the Higher Education sector. The 14 Irish Institutes of Technology (IoTs) recently under-
went an extensive transformation of their ICT systems, through a nationwide implementation of a suite of integrated IS.
This research study, centred on the evaluation of the Student MIS implementation was interpretive in nature; case stud-
ies were conducted in five IoTs. This paper focuses specifically on one issue uncovered through the research i.e. the
misalignment between the Student MIS and the IoTs requirements. The paper proposes a set of guidelines for address-
ing this issue throug h focusing on the th eoretical underp innings of CPE and its impo rtance for organisational learnin g
and benefit realisation.
Keywords: ICT Investment Management, ICT Evaluation, Formative Evaluation, Continu ous Participative Evaluation,
MIS, ICT in Tertiary Education
1. Introduction
ICT evaluation research has attracted the interests of
academics and practitioners for several decades. Evalua-
tion approaches can be viewed along a spectrum that
ranges from objective, rational, positivist approaches to
subjective, interpretive approaches [1]. The former have
historically dominated ICT evaluation research [2], at the
expense of contextual issues. However, in recent years,
interpretive approaches have gained a stronger foothold.
There is growing consensus that evaluation needs to be
formative in nature i.e. ongoing throughout the project
lifecycle [3–6]. Gemmell and Pagano (2003) [7] suggest
that organisations would benefit from moving towards a
continuous evaluation approach that was integrated into
the project management process. Arguments for con-
tinuous evaluation include project control, adaptability,
Value realization and improved relationships [8]. The
formative approach seeks to explore all project issues,
identify unexpected impacts, assess the degree to which
the project proceeds as planned and provide feedback to
improve project performance. It focuses on complex in-
teractions between people and technology, and considers
the system’s short-term and intermediate effects and its
influence o n stakeholder s.
Many authors have argued that ICT evaluation needs
to be dynamic and continuous across the project lifecycle
(see for example [9–13]). This is necessary in order for
what Willcocks (1992) [14] terms the “islands of evalua-
tion” to become integrated and for continuous improve-
ment opportunities to be identified. A particular type of
formative evaluation is Continuous Participative Evalua-
tion (CPE). Continuous participative approaches for ex-
ample Remenyi et al’s (1997) [15] Active Benefit Reali-
sation (ABR1), promote stakeholder involvement, moti-
vation and commitment and im prove IC T deci si on-making

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
34
through establishing dialogue with all affected parties.
Cordoba’s (2009) [16] development of a methodological
framework to support the process of IS planning empha-
sises continuous identification of concerns from stake-
holders; it promotes continuous dialogue, listening and
mutual collaboration between participants and facilitates
critical reflection in the exploration of possibilities for
improvement. CPE is integrated into organisational d aily
activities with a continuous focus on stakeholders, ICT
quality and evolving business objectives. Stakeholder
participation helps improve functionality, helps identify
and manage intangible impacts, stimulates organisational
learning, focuses attention on issues that impair ICT
success, helps maximise ICT benefit realisation and re-
duces the culture gap between business and ICT domains.
Throughout the 1980s and 1990s the concept of an
“evaluation party” [17] emerged to emphasise the im-
portance of stakeholder views.
Despite its usefulness, formative continuous participa-
tive evaluation is often not conducted in practice. Will-
cocks et al. (1999) [18] suggested that few companies
evaluate strategic, business, end-user and technical per-
formance in an integrated manner throughout the sys-
tem’s lifecycle. Evaluation declines following project
feasibility assessment; it is poorly linked across lifecycle
stages; and little learning for future ICT investments is
derived. Further, Hillam and Edwards (2001) [19] stated
that user perception is rarely considered. Lack of in-
volvement can result in decreased commitment or nega-
tive attitudes. For example, Markus (2004) [20] claimed
that approximately 75% of organisational change efforts
driven by technology fail because of negative stakeholder
reactions to work practice, business process and tech-
nology changes.
This paper discusses the need for formative continuous
participative evaluation mechanisms in addressing some
of the problems resultant from a large-scale standard
Student MIS implementation in the Irish Institutes of
Technology (IoTs). To date, there is a paucity of evalua-
tion research in the tertiary edu cation sector [21]. Further
research is required in this area as the ICT systems that
support student administrative operations represent the
means for competitive parity with or advantage over
other educational establishments. This paper discusses
one of the key findings that were distilled from the Stu-
dent MIS evaluation processes undertaken – the issue of
functional-operational misalignment. It proposes a set of
guidelines to help the IoT sector address this issue
through focusing on the importance of formative CPE
approaches. The evidence suggests that the approaches
adopted in the implementation and operationalisation of
the Student MIS did not emphasise the importance of
evaluation exercises; however, this paper suggests that
future benefit potential realisation can be enhanced
through adopting formative CPE methods.
2. The MIS Project in the IoT Sector
A proposal to investigate a collaborative acquisition of a
MIS for the Irish IoT sector was initially raised in
1991/1992. This system sought to support new modes of
education delivery, support IoT administrative operations,
improve services to all stakeholders, streamline work-
flow and improve organisational communications and
competitiveness. The project involved representatives
from the Department of Education and Science (DoES)
and the Council of Directors of the Irish IoTs. The sys-
tem was selected by a team of 12 representatives from 8
IoTs. However, the mechanism used in tender evaluation
was not sufficiently in-depth or no t applied with the nec-
essary rigour; hence it was felt by many stakeholders that
the system selection decision was a misguided one. The
initiative resulted in the implementation of a suite of in-
tegrated Information Systems (IS) for library, human
resources, finance and student management functions in
15 Institutions. These systems were rolled out to the IoT
sector in a series of implementation waves between 2000
and 2006. This paper focuses primarily on the impact of
the Student MIS.
In an attempt to maintain a common national standard
system design, a central project team was responsible for
all system development. Any required system changes
were managed through a central system change request
process. Hence, the majority of IoTs did not have system
development autonomy. While the central team offered
the benefits of resource efficiency, development of spe-
cialisations, access to scarce resources etc, it was felt that
the change request process to the central project team for
expost system development was unnecessarily bureau-
cratic. Changes refused by the central team, on the basis
that they were not required by the majority of IoTs, re-
sulted in some sites using functionality they found to be
unsuitable. Because any tailoring was designed to meet
common requirements, the changes made were often
compromises on specific needs. The bureaucracy of sys-
tem change requests resulted in IoTs deviating from the
common system standard through in-house developments
that were adhoc and nonstandard across the sector.
1ABR shifts the focus away from technical issues towards stakeholde
r
involvement. Each stakeholder level plays a co-creation and coevolu-
tionary role in systems development and in achieving results, and is
responsible for ICT management. In the process, the primary issues and
evaluation information are summarised in documents called pictures-a
business picture and a supporting financial picture and project picture.
Evaluation consists of three phases-setting the course, formative evalua-
tion, and moving forward after feedback; and seven key activi-
ties-initialisation, production of pictures, agreement to proceed, systems
development, evidence collection, participative evaluation and devel-
opment of updated pictures. This evaluation process continues in an
iterative manner.

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
35
Commissioning of the Student MIS across the various
implementation waves resulted in considerable work for
both the central team and individual IoTs. There was a
perceived lack of preparation for system commissioning
which gave rise to negative stakeholder perceptions and
detracted from staff willingness to embrace the system.
System operationalisation resulted in multiple problems;
these were linked to the system’s complexities, lack of
system configuratio n, lack of IoT pr eparation , and lo ss of
IoT control to the central structure. Further, in many IoTs,
the project teams established for system implementation
were disbanded/scaled back too early, hence there were
little resources to support problems or exploit system
capabilities. However, following a period of system use,
staff’s increased familiarity with the system led to the
realisation of a not insignificant flow of benefits. Bene-
fits included for example improved structure, data stan-
dards and data access; job and management related bene-
fits; improved quality procedures; and staff empower-
ment; and the Student MIS also served as a platform for
future IoT IC T developments.
3. The Student MIS Explained
This Student MIS was oriented towards the US market
and was anticipated to cater for all stages of a student’s
interaction with an IoT. The system had comprehensive
functionality for cour se and subject management, student
data, admissions application processing, student registra-
tion, maintenance grants payments, accounts receivable
and fees assessment, examinations and academic history,
student progression and graduation. Details of its mod-
ules are discussed in Table 1:
Figure 1 provides an overview of how these Student
MIS modules were integrated. Firstly, biodemographic
data is captured to facilitate admissions applications
processing. These applications may come from the CAO
or be made directly to the IoT. The applications are veri-
fied against the catalogue of existing courses. Following
acceptance of a course place, the individual’s data is used
to create a student record. All students are registered at
the individual subject level; this is made possible by the
breakdown of subjects into their respective delivery
modes based on the ACS. This registration is required for
student fee assessment, maintenance grants payments and
for examination results processing. All results are trans-
ferred to an academic history archive; they become the
basis for student graduation from a completed pro-
gramme or progression to the following academic year,
where a new student record is created.
4. Research Methodology
The research methodology used in evaluating the impact
of the Student MIS was interpretive in nature. The inter-
pretive paradigm offered the opportunity to develop an
in-depth understanding of the ICT system’s impact; it
facilitated the capture of contextual depth and detailed,
nuanced descriptions; and avoided the unproblematic,
value-free view of organisations associated with positiv-
ist approaches. The study’s research methodology is out-
lined in Figure 2. The case study was the selected re-
search method and was based on data collected from five
sources-organisational websites, project documentation,
newspaper articles, independent reports and semi-struc-
tured interviews.
Table 1. Student MIS modules
Catalogue The database of courses and subjects offered by an Institution
Approved Course
Schedule (ACS) The listing of subjects linked to a course; it outlines teaching hours, examination and continuous as-
sessment components
Sections and Blocks Breaks down subjects into specific delivery modes, such as lecture, practical and tutorial hours. Stu-
dents are registered on these sections. A block is a grouping of sections by stage
CAO Interface Enables download of student data from the Central Applications Office website. The CAO is the body
responsible for recordi ng all student applica tions to third level educa tion instituti ons in Ireland
Admissions Stores applicant data and processes applications. It supports both CAO and direct admission applicants
General Person Captures biographical and demogra ph i c da t a
General Student Records current and historical student data. The record is originally created when an applicant accepts
a course place offered
Registration Records registration information for a given term
Maintenance Grants Facilitates grant payments to a student’s bank account
Accounts Receivable Tracks financial transactions such as student fee assessment, contract assignment, payments, and re-
ceipt, invoice and jour n a l p roduction
Examinations Facilitates student exam results and production of relevant reports
Academic History Enables historical academic results to be recorded in the Student MIS
Web for Faculty Empowers academics to enter student results, and view class and student details. Grades are broken
into their component parts through electronic gradebook functionality
Letter Generation Enables selected data to be merged with predefined letters
Graduation Produces graduate lists and records da ta relating to graduation ceremonies

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
36
Figure 1. Functionality and operations of the student MIS
Figure 2. Research methodology
Case studies were conducted within five IoTs. Pur-
posive sampling was used in case site selection as this
sampling strategy ensures that key research themes are
addressed and that diversity in each category is explored.
The five case sites were selected due to their diversity in
a number of respects. They participated in different im-
plementation waves, were geographically dispersed and
differed in their student population sizes and academic
programme offerings. The following points give a brief
synopsis of the five case sites:

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
37
 Site One was one of the first IoTs to implement
the student MIS. This site had significant in-
house MIS team capabilities, who were respon-
sible for system implementation, support and
had considerable autonomy in system develop-
ment.
 Site Two was a member of the first implementa-
tion wave. Due to difficulties experienced with
the central change request process, its in-house
MIS unit also developed additional functionality
to meet end-user needs.
 Site Three was a member of the second imple-
mentation wave. This IoT was smaller than the
previous two sites and had more limited techni-
cal resources.
 Site Four was the smallest IoT examined in this
study and was also a member of the second im-
plementation wave. Its project team experienced
a number of personnel changes during the im-
plementation effort.
 Site Five was a member of the final implemen-
tation wave. It experienced difficulties in re-
sourcing a dedicated project team and its initial
system start-up date was delayed. At the time of
research, the system was used to a limited exten t
and work was ongoing in implementing core
functionality.
Wi thin those IoTs, 49 semi-structured interviews were
carried out between 30 November 2005 and 24 May
2006 with senior management personnel, MIS team per-
sonnel and system end users. The selected informants
were closely involved in the ICT project and had
in-depth knowledge of the subject area. Each interview
lasted between 60 and 90 minutes, was recorded with
the informants’ permission and was later transcribed.
The informants were g iven the opportunity to verify the
transcripts prior to analysis. Further, the supporting
documentation was valuable in corroborating the evi-
dence collected in the semi-structured interviews. It pro-
vided a means of triangulation in th at it supplied specific
details, and helped to augment and substantiate the in-
terview data. The data analysis process is outlined in
Figure 3.
Data analysis was carried out using a variant of Glaser
and Strauss’s (1967) [22] grounded theory method. GT is
one of the most widely used qualitative frameworks in
business and management studies [23]. This GT analysis
was supported by a Computer Aided Qualitative Data
Analysis Software (CAQDAS) package called N-vivo.
The N-vivo package facilitates efficient data indexing
and management, and supports analysis through for ex-
ample relationship and model ex ploration. As outlined in
Figure 3, the interview transcripts were initially im-
ported into this software. Examination of these tran-
scripts led to key words/ideas being identified and these
were coded using N-vivo. Groupings of these codes that
contain similar content are referred to as concepts in GT.
As coding progressed, it became apparent that many con-
cepts were related and these were reclassified into a se-
Figure 3. Qualitative data analysis

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
38
Table 2. Degree of system exploitation
IoT’s Degree of System Exploitation
System Capabilities Site One Site Two Site Three Site Four Site Five
Catalogue Partly Exploited Partly ExploitedExploited Partly Exploited Exploited
ACS Unexploited Partly ExploitedPartly ExploitedPartly Exploited Partly Exploited
Sections & Blocks Partly Exploited Explo i ted Partly ExploitedPartly Exploited Exploited
CAO Interface Exploited Partly ExploitedPartly ExploitedPartly Exploited (Was being implemented)
Admissions Partly Exploited Partly ExploitedPartly ExploitedPartly Exploited Partly Exploited
General Person Exploited Exploited Partly ExploitedPartly Exploited Partly Exploited
General Student Partly Exploited Exploited Partly ExploitedPartly Exploited Partly Exploited
Registration Exploited Partly ExploitedPartly ExploitedPartly Exploited Partly Exploited
Maintenance Grants Exploited Exploited Exploited Exploited Unexploited
Accounts Receivable Unexploited Partly ExploitedPartly ExploitedPartly Exploited Unexploited
Examinations (Was being implemented) Partly ExploitedPartly ExploitedPartly Exploi ted (Was being implemented)
Academic History (Was being implemented) Exploited Exploited Exploited Unexploited
Web for Faculty (Was being implemented) Unexploited Partly Expl oitedUnexploited Unexploited
Letter Generation Unexploited Unexploited Unexploited Unexploited Unexploited
Graduation Unexploited Unexploited Unexploited Unexploited Unexploited
ries of categories (i.e. broad groups of similar content
that are later used to generate theory) and related sub-
categories using N-vivo’s hierarchical tree structure. This
organised related concepts in relation to the overall re-
search and facilitated greater understanding of the body
of evidence through examining the key themes. Memo
creation to clarify ideas and identify relationships be-
tween categories, constant concept comparison and itera-
tive reflection on what was already coded were important
steps in this coding process.
The key concepts and categories identified through
N-vivo coding, as well as important details from the
other four sources of case study evidence were synthe-
sised into a detailed cross-case primary narrative of the
Student MIS project. Narratives play an important role in
the social world; they are a form of knowledge and
communication [24], as complex situations can be better
understood in story format. Hence, they enable a re-
searcher to shape various interview stories into a coher-
ent account of the key themes. Through significant re-
flection on the primary narrative, it was reduced to the
principal findings or themes. Reflection on the primary
narrative involved considering three questions: “what
does the text say?” “why does the text say what it does?”,
and “what is my understanding of what is taking place?”.
This approach was useful in providing a conceptual
separation of thre e ways of examining the primary narra-
tive and in expanding my interpretation over a series of
stages. The processes involved in distillation of the key
findings involved both creativity and flexibility. Dia-
grammatic representation was important in understanding
the phenomenon’s diversity and in exploring relation-
ships and complex processes.
5. Findings-the Issue of Functional
Operational Misalignment
Findings on several different aspects of the project were
uncovered including system selection; system develop-
ment for the Irish IoTs; system commissioning; ex-post
performance in the early years; and ex-post performance
at the time of research. These findings are discussed by
Carcary (2009) [21]. This paper focuses on one of those
findings i.e. the misalignment between system capabili-
ties and IoT requirements. The main challenge facing the
IoTs was the misalignment between what the system was
used for at the time of research and what it was capable
of delivering, and between IoT requirements and the ex-
tent to which they were met. As explained in the follow-
ing sections, IoTs use of the student MIS fell short of
system potential; hence, IoTs were not leveraging the
system’s potential advantages. Sub-optimal system usage
was partly due to the system’s inability to meet some IoT
requirements.
5.1 An Evaluation of System Functional
Capability Exploitation
Benefit realisation from the student MIS was sub-optimal
within the IoTs. The system offered functionality for all
aspects of student administration, yet many of its capa-
bilities remained unused. For example, several infor-
mants within Site Three suggested that system usage at
the time of research was “a fire fighting” exercise and
was “just touching the tip of the iceberg”. One end-user
estimated functionality exploitation to be 35%-40%,

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
39
while a project team member suggested 50%-60%.
Within Site Two, only the baseline functionality was
implemented; work was ongoing in streamlining opera-
tional processes. Site Five’s exploitation was limited to
course set-up and registrations functionality, and signifi-
cant work was required to advance the knowledge base.
Site Four’s system usage was equated to “scratching the
surface”; estimates of exploitation ranged between 5%-
10% according to the ICT Manager and 70%-75% ac-
cording to a project team member. Despite being the first
to implement the system, Site One had according to the
system’s development team leader used only 10% of the
baseline functionality. For example, examination-related
modules were only being introduced at the time of in-
formant interviewing. Hence, in all IoTs, informants
suggested that the system was not fully exploited. Table
2 provides a synopsis of the degree of IoT’s system ex-
ploitation. This suggests that certain modules were ex-
ploited, partially exploited or completely untouched.
The variances experienced in system functional capa-
bility exploitation can be linked to a number of factors:
 Implementation Wave: From Site Five’s per-
spective, limited system exploitation was partly
due to its postponed implementation deadline.
However in the other case study sites, the sys-
tem was stabilised between three and five years.
Those IoTs had a longer timeframe to exploit
the system but had not taken full advantage of
this.
 Staff Knowledge and Resources: Across all IoTs,
reasons for sub-optimal system exploitation
were related to limited knowledge of further
system capabilities and the time required to
achieve system familiarity. This was linked to
the scaling back of project teams and the associ-
ated training problem. Site Two explicitly cited
lack of resources as key in failing to explore the
system’s fuller potential and implement further
modules.
 Ability to Meet Requirements: The inability of
some modules to meet requirements and lack of
tailoring for those sites under the central struc-
ture impacted on the degree of system exploita-
tion. The central structure had a phased ap-
proach for system development, which was re-
strictive for urgent IoT requirements.
5.2 An Evaluation of the Degree of IoT
Operational Requirements Met
Across case study sites, the efficiency with which re-
quirements were met was questionable. Many baseline
modules did not meet Site One’s requirements and, as
stated, resulted in numerous in-ho use customisations and
development projects. The system involved significant
work for Site One’s administrative staff; however in
comparison to legacy applications, all informants agreed
that it had substantially improved student record man-
agement. The degree of requirements met by the baseline
system within Site Two was estimated by a managerial
figure at 20%. The functionality was too generic and
deficient in reporting capabilities; hence the system was
used primarily for data storage while all reporting, ma-
nipulating and controlling was done in-house. Within
Site Three, the required further development of the Stu-
dent MIS had not taken place. Hence, the system re-
mained bespoke towards full-time students and was in-
flexible in dealing with any other applican t types. A pro-
ject team member of Site Four regarded it as meeting
85%-90% of requirements, but suggested a complete
suite of integrated products would be more beneficial.
Site Five also encountered some limitations. Approxi-
mately 30% was regarded by one team member as un-
suited to the Irish tertiary educational system. Table 3
outlines the degree of requirements met by the various
modules; it highlights that modules either met all re-
quirements, some requirements or were unsuited.
5.3 Functional-Operational Misalignment
The functional-operational misalignment uncovered im-
pacted the extent to which administrative staff could ef-
fectively use the system. IoTs ability to address this issue
depended on a number of factors:
 Site autonomy over the Student MIS: Site One
had greater control over its system development.
Much functionality used on a day-to-day basis
was developed in-house. This was in contrast to
the other case study sites that had little local
autonomy. They were restricted in their ability
to customise the system due to the control exer-
cised by the central structure.
 Degree of adherence to the common national
system standard versus in-house development:
As stated, IoTs under the central structure’s di-
rective were tied to a common national standard .
However, the central structure’s development
time for mission critical reports and for applica-
tions outside the full-time student population
was slow and based on priority demand. Further,
development was generic rather than customised
to IoT specifications. Hence, all case study sites
diverged to some extent from the common stan-
dard. This was primarily in developing external
reporting capabilities.
 Degree of in-house development resource capa-
bility: A lack of resources was apparent in Site
Two, Three and Four due to the scaling back of
project teams. This impacted on:
- IoT’s ability to advance the system for pro
jects such as modularisation and semesteri-
sation, and tailor it for different student co-

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
40
Table 3. Degree of IoT requirements addresse d
Degree of IoT Requirements Met
System Capabilities Site One Site Two Site Three Site Four Site Five
Catalogue Unsuited Met Past Require-
ments Partly Met Re- quirements Met Requirements Met Require- ments
ACS Unsuited Met Past Require-
ments Met RequirementsMet Past Require- ments Met Requi re- ments
Sections & Blocks Partly Met
Requirements Met Require- ments Met Requirement sMet Requirements Met Require- ments
CAO Interface Met Requirements Met RequirementsM et RequirementsMet Requirements (Unknown2)
Admissions Met Require-
ments Met RequirementsPartly Met Re- quirements Met Requirements Met Require- ments
General Person Met Require-
ments Partly Met Re- quirements Met RequirementsPartly Met Re- quirements Met Require- ments
General Student Unsuited Partly Met Re-
quirements Met RequirementsPartly Met Re- quirements Met Require- ments
Registration Partly Met
Requirements Partly Met Re- quirements Partly Met Re- quirements Partly Met Re- quirements Partly Met Re- quirements
Maintenance
Grants Partly Met Requirements Partly Met Re- quirements Partly Met Re- quirements Partly Met Re- quirements (Unknown)
Accounts Receivable Unsuited Unsuited
Partly Met Re-
quirements Unsuited Unsuited
Examinations Partly Met
Requirements Partly Met Re- quirements Partly Met Re- quirements Partly Met Re- quirements (Unknown)
Academic History (Unknown) Met RequirementsMet RequirementsMet Requirements (Unknown)
Web for Faculty Met Require-
ments (Unknown) Partly Met Re- quirements (Unknown) Met Require- ments
Letter Generation Unsuited Unsuited Unsuited Unsuited Unsuited
Graduation (Unknown) Unsuited Unsuited Unsuited (Unknown)
Figure 4. Addressing functional-operational misalignment
2The compatibility of a system module with IoT requirements was unknown when the module had not been exploited or examined by MIS per-
sonnel

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
41
horts;
- Time lags for report development;
- The degree to which unutilised modules
could be further exploi t e d;
- The degree to which staff could be trained
in further system capabilities and to which
more efficient work practices could be
found.
Those IoTs with greater resources were better posi-
tioned to address the above four points.
5.4 Towards CPE-Addressing Functional-
Operational Misalignment
The following model provides a visual depiction of func-
tional-operational misalignment. It puts forward a num-
ber of guidelines/steps that IoTs should consider in ad-
dressing this issue. These guidelines draw on the theo-
retical underpinni ngs of format i ve CP E.
The following five steps provide an explanation of
Figure 4.
Step One. Organisational requirements constantly
evolve. Examples of such changes in the Irish IoTs in-
clude:
 Changes in education delivery modes. These in-
clude the introduction of semesterisation, modu-
larisation, e-learning and blended learning, and a
move towards distance learning through exploit-
ing the virtual campus concept. Each IoT had a
specific timeframe for embracing these changes.
 Changes in student profiles. Examples of non-
standard students include students with subject
exemptions, Socrates students, repeat students,
ACCS students, Fáilte Ireland and FÁS appren-
tice students, adult learners and students under-
taking more than one course.
Determining functional-operational alignment requires
IoTs to iteratively establish what their requirements are.
Figure 5. Determining organisational requirements
This process is outlined in Figure 5. Key stakeholder
groups need to be involved. In the Student MIS project
these stakeholders include senior management who make
academic policy decisions and administrative staff who
use the system.
Step Two. System capabilities also evolve. Changes to
the Student MIS capab ilities in the Irish IoTs are due to:
 Improvements and upgrades from the system vendor;
 Developments by the central structure or in- house
MIS personn el.
Determining functional-operational alignment requires
IoTs to iteratively review what the system capabilities
are. This process is outlined in Figure 6. Key stake-
holders need to be involved. In the Student MIS project,
these stakeholders include MIS personnel who under-
stand the system’s changing functionality. Such changes
also need to be promoted to administrative staff to in-
crease their awareness of the functionality available.
Step Three. The output of the previous two steps is
examined in a requirements-capability match analysis
(Figure 7). It investigates the degree to which organisa-
tional-operational requirements and system functional
capabilities converge. Each operational requirement is
examined against relevant system components to identify
how well it is supported. In totality, this exercise o utlines
the extent to which the system meets IoT requirements.
Each system module is also examined to determine its
degree of utilisation. In totality, this ou tlines the ex tent to
which the IoT exploits system capabilities. This process
should be documented in the format of detailed descrip-
tions, which would better help in identifying steps for
remedial action.
Step Four. The outcome of the requirements- capabil-
ity match analysis (Figure 8) will be a state of func-
tional-operational alignment and/or a state of func-
tional-operational misalignment. Functional-operational
Figure 6. Determining system capabilities

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
42
Figure 7. Requirements-capability match analysis
Figure 8. Functional-operational alignment or misalignment
alignment arises when all existing functionality is used
and all organisational requirements are met by available
functionality. Functional-operational misalignment oc-
curs when existing functionality is unused and organisa-
tional requirements are not met by available fun ctionality.
Achieving an ideal match between requirements and ca-
pabilities is complex. The misalignment experienced
across all IoT case study sites was unsurprising given the
system’s US orientation, the common national standard,
the phased approach for system development and chang-
ing IoT requirements.
Step Five. Areas of misalignment uncovered are ad-
dressed. As outlined in Figure 9, this includes:
 System development to better match organisa-
tional requirements. This involves development
by the central project team for general IoT sec-
tor needs. It also requires a degree of in-house
tailoring to meet site specific demands.
 Resource investment to exploit necessary unuti-
lised functionality. Resource investment for
further system exploitation is important only
where unused functionality would be of benefit
to the organisation. This includes reinstating
Figure 9. Addressing functional-operational misalignment
functional team members to identify unexploited
capabilities and to train and educate users in the
broader system functionality.
The goal of this activity is to achieve greater conver-
gence between organisational requirements and system
capabilities, with a view to improving organisational-
system compatibility/performance and realisation of
greater system benefit potential.
The above guidelines reflect in five steps how the
functional-operational misalignment issue may be im-
proved. This model may be viewed as an alternative con-
tinuous participative ICT investment evaluation tech
nique that may be iteratively applied throughout the sys-
tem’s lifecycle. Through examining the alignment be-
tween requirements and capabilities, the IoTs can high-
light problematic areas, identify steps for corrective ac-
tion, set future performance targets and track progress by
involving k ey stakeholder gr o u ps.
6. Discussion and Conclusions
ICT investment evaluation is important in identifying
and addressing the misalignment that exists between the
standard student system’s functional capabilities and the
diverse operational requirements of the multiple autono-
mous IoTs. Functional-operational misalignment results
from the inability of organisations to exploit system ca-
pabilities. There is also an issue regarding whether such
a system meets organisational needs. Prior to this re-
search, the impact of functional-operational misalign-
ment in leveraging system benefit potential was not
clearly addressed in the literature. Some researchers dis-
cussed the match between system capabilities and or-
ganisational requirements. For example Avram (2001)
[25] highlighted “definitional mismatch”, i.e. the degree
of risk faced by an organisation in terms of how well
functional requirements meet operational needs; and
Saastamoinen (2005) [26] suggested that the most diffi-
cult factor in ICT evaluation is the match between sys-
tem and processes. However, previous research had not
explored this issue in-depth or suggested how it may be
addressed.
In this respect, this paper expands the body of ex-
isting research. It examined how CPE offers the op-
portunity to understand and exploit the unused function

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
43
ality in a standard system through involving key stake-
holder groups. This not only increases stakeholder
awareness of evolving technological capabilities and
organisational requirements but also promotes a more
favourable acceptance to organisational changes through
fostering their commitment and motivation. It can lead to
enhanced stakeholder relationships through recognising
the impact of organisational change and the interactions
that exist between people and technology. Hence,
through a continuous participative evaluation process
that emphasises dialogues and stakeholder feedback,
there is potential for improving functional-operational
alignment and this is important for enhanced system
benefit r eal isatio n.
The author’s development of this CPE approach is in
line with a growing volume of ICT evaluation research
that recognises the value of more formative evaluation
methods over rational, objective approaches that do not
capture contextual depth or address important human
complexities. For example, Gemmell and Pagano (2003)
[7], who conducted ICT evaluation research in the
Higher Education sector argued for the importance of
continuous evaluation throughout the project lifecycle.
Similarly, several researchers in other sectors discuss the
importance of formative methods in identifying the un-
expected impacts of ICT investment; monitoring pro-
gress, increasing control and improving project perform-
ance; identifying continuous improvement opportunities;
improving relationships with stakeholders; and increas-
ing benefit realisation (see for example [3–6,8–13]). As
evident in the author’s discussion in Section 5.4, the most
important aspect in addressing the functional-operational
misalignment issue is the promotion of stakeholder in-
volvement; enabling more effective ICT decisions on
how to develop the system and staff capabilities through
ongoing dialogue with affected parties. This is the un-
derlying fundamental principle of widely cited CPE
methods such as Remenyi et al’s (1997) [15] ABR and
Cordoba’s (2009) [16] IS Planning Framework. The ap-
proach developed in this paper is a further useful step in
enhancing this CPE body of research in that it addresses
the misalignment issue, which was not previously ex-
plored in-depth. It further adds to the limited body of
evaluation research in the Higher Education Sector.
7. Avenues of Further Research
This study has taken a small step towards addressing the
paucity of ICT evaluation research in HEIs and in pro-
viding greater understanding of the challenges in opera-
tionalising a standard ICT system in organisations with
diverse requirements. However, there are many opportu-
nities for further research in this sector.
 The research findings may be examined in a
wider context. Interviews may be conducted
with informants in the ten other Institutions.
This would provide a basis for comparing issues
across the IoT sector and for confirming or ex-
tending this study’s findings. It would also de-
termine the finding’s inferential transferability
to a broader set t ing.
 Secondly, the study’s scope could be expanded
to include the viewpoints of the central project
team and the DoES. Due to the nature of their
Student MIS project involvement; it is probable
that their project perceptions differ from those
directly impacted by system introduction. If this
were found to be true, it would make for an in-
teresting examination of the diversity in social
groups’ perceptions regarding large-scale MIS
implementations.
 Thirdly, the model for resolving functional-
operational misalignment could be iteratively
applied across the IoT sector. This CPE ap-
proach would provide for greater stakeholder
involvement in determining evolving IoT re-
quirements and system capabilities, and in
evaluating the degree of functional-operational
alignment. Further, it would focus IoT activi-
ties on problematic areas in order to improve
system utilisation and benefit realisation. This
project would be longitudinal; however, through
setting various benchmarks, it would determine
the model’s usefulness in optimising ICT per-
formance.
 Finally, the research was confined to the Irish
education market. Comparative research in other
countries would be of benefit. This would de-
termine whether issues, similar to those experi-
enced in the Irish IoTs, were encountered in im-
plementing and operationalising large-scale
standard ICT systems, and how such issues
were overcome. In this respect it might be in-
teresting to look to th e UK or even the USA for
comparisons.
REFERENCES
[1] M. Wilson and D. Howcroft, “Power, politics and persua-
sion in IS evaluation: A focus on ‘relevant social
groups’,” Journal of Strategic Information Systems, Vol.
14, pp. 17–43, 2005.
[2] W. Chen and R. Hirschheim, “A paradigmatic and meth-
odological examination of information systems research
from 1991 to 2001,” Information Systems Journal, Vol.
14, pp. 197–235, 2004.

Evaluating a Student MIS in the Tertiary Education Sector: Addressing Functional-Operational
Misalignment through Continuous Participative Evaluation
44
[3] S. Cronholm and G. Goldkuhl, “Strategies for informa-
tion systems evaluation-six generic types,” Electronic
Journal of Information Systems Evaluation, Vol. 6, No. 2,
pp. 65–74, 2003.
[4] C. Miskelly, A. Moggridge, C. Stephenson, and M. Wil-
liams, “Formative and interpretive approaches to the
evaluation of community focused web-fronted projects,”
Proceedings of the 11th European Conference on Informa-
tion Technology Evaluation, Amsterdam, The Nether-
lands, Academic Conferences, Reading, pp. 281–288, 11–
12 November, 2004.
[5] A. Roudsari, E. J. Berridge, M. Vodel, and E. Carson,
“Design and evaluation of ICT-based patient information
systems to support the management of chronic diseases,”
Proceedings of the 10th European Conference on Informa-
tion Technology Evaluation, Madrid, Spain, MCIL, Read-
ing, pp. 547–556, 25–26 September, 2003.
[6] N. Shiratuddin and M. Landoni, “Evaluation of content
activities in children’s educational software,” Proceedings
of the 8th European Conference on Information Technol-
ogy Evaluation, Oxford, United Kingdom, MCIL, Read-
ing, pp. 79–87, 17–18 September, 2001.
[7] M. Gemmell and R. Pagano, “A post-implementation
evaluation of a student information system in the UK
higher education sector,” Electronic Journal of Informa-
tion Systems Evaluation, Vol. 6, No. 2, 2003.
[8] N. Doherty and L. McAulay, “The relationship between
the ex ante and ex post information systems evaluation:
Reflections from the literature and the practice of evalu-
ating e-commerce investments,” Proceedings of the 9th
European Conference on Information Technology Eva-
luation, Paris, France, MCIL, Reading, pp. 129–136,
15–16 July, 2002.
[9] E. Berghout and M. Nijland, “Full lifecycle management
and the IT management paradox,” In Remenyi, D. and
Brown, A. (eds.), Make or Break Issues in IT Management,
Butterworth Heinemann, Oxford, pp. 77–107, 2002.
[10] N. Doherty and M. King, “The treatment of organisa-
tional issues in systems development projects: The impli-
cations for the evaluation of information technology in-
vestments,” Electronic Journal of Information Systems
Evaluation, Vol. 4, No. 1, 2001.
[11] B. Farbey, F. Land, and D. Targett, “Evaluating invest-
ments in IT: Findings and a framework,” In Willcocks,
L.P. and Lester, S. (eds.), Beyond the IT Productivity
Paradox, Wiley, Chichester, pp. 183–215, 1999.
[12] B. Farbey, F. Land, and D. Targett, “Moving IS evalua-
tion forward: Learning themes and research issues,” Jour-
nal of Strategic Information Systems, Vol. 8, No. 2, pp.
189–207, 1999.
[13] Z. Irani, A. M. Sharif, and P. E. D. Love, “Linking
knowledge transformation to information systems evalua-
tion,” European Journal of Information Systems, Vol. 14,
pp. 213–228, 2005.
[14] L. P. Willcocks, “Evaluating information technology
investments: Research findings and reappraisal,” Journal
of Information Systems, Vol. 2, No. 3, pp. 243–268, 1992.
[15] D. Remenyi, M. Sherwood-Smith, and T. White, “Achie-
ving maximum value from information systems,” Wiley,
Chichester, 1997.
[16] J. R. Cordoba, “Critical reflection in planning information
systems: A contribution from critical systems thinking,”
Information Systems Journal, Vol. 19, No. 2, pp. 123–147,
2009.
[17] V. Serafeimidis and S. Smithson, “Information systems
evaluation as an organisational institution experiences
from a case study,” Information Systems Journal, Vol. 13,
pp. 251–274, 2003.
[18] L. P. Willcocks, G. Fitzgerald, and M. Lacity, “To out-
source IT or not? Research on economics and evaluation
practice,” In Willcocks, L. P. and Lester, S. (eds.), Be-
yond the IT Productivity Paradox. Wiley, Chichester, pp.
293–333, 1999.
[19] C. E. Hillam and H. M. Edwards, “A case study approach
to evaluation of Information Technology/Information
Systems (IT/IS) investment evaluation processes within
SMEs,” Electronic Journal of Information Systems
Evaluation, Vol. 4, No. 1, 2001.
[20] M. L. Markus, “Technochange management: Using IT to
drive organisational change,” Journal of Information
Technology, Vol. 19, No. 1, pp. 4–20, 2004.
[21] M. Carcary, “ICT evaluation in the Irish higher education
sector,” Electronic Journal of Information Systems Evalua-
tion, accepted for publication in Vol. 12, No. 2, 2009.
[22] B. G. Glaser and A. L. Strauss, “The discovery of
grounded theory: Strategies for qualitative research,” Al-
dine De Gruyter, New York, 1967.
[23] M. Alvesson and K. Sköldberg, “Reflexive methodol-
ogy-new vistas for qualitative research,” Sage Publica-
tions, London, 2000.
[24] B. Czarniawska, “Narratives in social science research-
introducing qualitative methods,” Sage Publications, Thou-
sand Oaks, London, 2004.
[25] G. Avram, “Evaluation of investments in ecommerce in
the Romanian business environment,” Electronic Jour-
nal of Information Systems Evaluation, Vol. 4, No. 1,
2001.
[26] H. Saastamoinen, “Exception-based approach for infor-
mation systems evaluation: The method and its benefits to
information systems management,” Electronic Journal of
Information Systems Evaluation, Vol. 8, No. 1, pp. 51–60,
2005.