ABSTRACT

Conservation projects have limited resources and an ever expanding to-do list, presenting a significant leadership challenge. Mid-term evaluations can be informative tools to check how short term activities and resources are achieving long term conservation outcomes. This research study involved a programme evaluation of a successful species recovery project in Mauritius, using a systems-thinking approach to conservation management, and utilising a Theory of Change to assess the effectiveness of short term activities on long term impacts. This systematic method of evaluation gave greater clarity on resource planning, performance indicators and supporting processes, with observations that could be incorporated into ongoing plans. Such an approach could be used by funding organisations or by local management teams to review project performance without the need for a comparator, extensive benchmark data, nor a prescriptive management standards framework.

Keywords:

Conservation Biology, Systems Thinking, Evaluation, Theory of Change, Effectiveness

1. Introduction

Leadership is an important tool for advancing the field of wildlife conservation. Although the study and research of leadership methods have been prevalent in many industries throughout the last century, conservation biology is a relatively new sector in which a significant proportion of operating organisations (government, NGO or research institutions) has only emerged since the 1980’s. As such, the sector is short of informative research on leadership, and there is a general lack of an understanding of the practices which relate to its effective management (Bruyere, 2015; Manolis et al., 2009) . Many projects and organisations have slipped into traditional ways of management thinking, with excessive bureaucracy and slow, hierarchical decision making (Black, Groombridge, & Jones, 2011) . There have been recent advances, however, and a turn towards vision building, adaptive management and systems thinking (Black & Copsey, 2014; Bruyere, 2015) .

Conservation biology is a “crisis discipline”, with quick reactions often resulting in entire species being saved from extinction (Soulé, 1985) . Saving a species from extinction is a complex problem, with many interlinking relationships between ecosystem inhabitants. The scarcity of financing and resource available should lead us to evaluate the effectiveness of conservation programmes, and there have been calls in recent years for more frequent programme evaluations, either to learn from successful interventions or to correct ineffective activities (Cullen & White, 2013; Ferraro & Pattanayak, 2006; Howe & Milner-Gulland, 2012b; Kapos et al., 2008; Margoluis, Stem, Salafsky, & Brown, 2009; Sutherland, Pullin, Dolman, & Knight, 2004) . Project evaluations are important for program effectiveness and accountability (Howe & Milner-Gulland, 2012a; Lamoreux et al., 2014) . Time frames for species conservation programs depend on the life cycle and species generation length, but typically run over ten or twenty year lifecycles with funding most commonly renegotiated on a shorter time scale of 1 - 5 years. It can take several decades before a positive impact is observed in the target population, which can present project leaders with additional challenges when securing funding in the short term (Black, Groombridge, & Jones, 2011) . The nature of the conservation sector means that leaders also need to be skilled in partnership building, conflict management, acting quickly with little information, and continuously learning from results, in order to be effective in achieving biodiversity goals (Black, Groombridge, & Jones, 2013; Black, 2015; Bruyere, 2015; Manolis et al., 2009; Martin et al., 2012) .

Systematic reviews are now common practice in the discipline, with an evidence based approach to outcomes and scientific reasoning for each conservation activity put in place, but this does not always extend to the management processes of a project. There is a broad selection of frameworks and methodologies regularly used in practice; from counterfactual outcomes, to standard frameworks and specially developed scorecards. Some are suitable for mid-term project evaluations, but many focus on a ranking system which is intended for comparison, or aim to establish a standard for a particular organisation. Processes such as resource management, leadership and planning can be as impactful as the scientific activities themselves when it comes to overall effectiveness of a species or habitat recovery programme. These supporting processes are often not considered when reviewing the success of a conservation project, but some existing methodologies, such as the Cambridge Conservation Forum Scorecard, IUCN-WCPA Management Effectiveness Evaluation Framework, WWF Organisational Assessment Framework and Conservation Excellence Model do consider management factors (Black & Groombridge, 2010; Kapos et al., 2009; Leverington, Hockings, Pavese, Lemos Costa, & Courrau, 2008; WWF, 2003; Young et al., 2014) .

The theory of change (TOC) model describes cause and effect, and can be used to build strategy for development or values-based projects (Jenks, Vaughan, & Butler, 2010; Wholey, Hatry, & Newcomer, 2010) . Regular activities or processes are linked with outputs and longer term impacts. The TOC can also be used for programme evaluations and to develop performance indicators, as a way of visualising an intervention or project as a complete system (Rogers, 2008) . This “systems thinking” approach to project evaluation aims to build an understanding of the factors that influence project effectiveness, in a method similar to that used by Bottrill et al. for the evaluation of conservation initiatives in Samoa (2011) . Systems thinking considers interrelationships between entities or processes within a complex environment, with leaders acting upon system processes and enabling employees a greater degree of autonomy (Black, Groom- bridge, & Jones, 2011; Senge & Sterman, 1990) . A systems thinking approach to conservation management can address some shortcomings of the traditional bureaucratic management style which is still prevalent in the discipline (Black & Copsey, 2014) .

The Mauritius Fody (Foudia rubra) is a small forest bird that is endemic to Mauritius, it being found no-where else on the planet having evolved separately on this island in the Indian Ocean. The species was under threat from habitat loss, invasive predators and introduced competitors, and is classified as endangered by the IUCN (Cristinacce et al. 2009; Birdlife International, 2015) . The Fody is highly endangered, and an important endemic species in Mauritius such that the senior management of the Mauritian Wildlife Foundation have dedicated a large amount of time and resource to ensure its survival.

The Fody has been the subject of conservation efforts since 2002-2003, under a management program carried out by the Mauritian Wildlife Foundation (MWF) and with most activity taking place on the small predator-free islet of Ile aux Aigrettes, a restored island approximately 800 m off the south-east coast of Mauritius (Cristinacce, Garrett, Cole, Tatayah, & Jones, 2010) . The conservation management intervention was initially very intensive, taking place through to 2006 with captive rearing of chicks and release of adult birds on Ile aux Aigrettes to establish a secondary location for the species in Mauritius (Cristinacce et al., 2008; Ferriere & Baboorun, 2014, 2015; Hotopp, 2012; Mccready, 2013) . Population monitoring, nest management techniques and supplementary feeding have been carried out over the life of the project, but other intensive management activities have now been reduced.

Additional food is prepared by project staff and made available to the wild population on Ile aux Aigrettes, as the practice is found to increase the general health of adult birds and reduce the likelihood of death in juveniles, while allowing them to tolerate smaller areas of habitat (Garrett, Jones, Cristinacce, & Bell, 2007; Robb, McDonald, Chamberlain, & Bearhop, 2008) . The population on Ile aux Aigrettes is constantly monitored, and a team attends the project year-round with a staff varying between two and five. The project has been successful, with the subpopulation growing eight fold since the initial translocation in 2004 (see Figure 1). The species was down-listed from

a critically endangered status in 2009 due to increased population stability resulting from actions taken by MWF (Birdlife International, 2015) . The maturity of the project makes it suitable for a mid-programme evaluation, because sufficient time has passed for ecological results to be realised, with potential for lessons to be learned from its successful strategy.

This paper explores a way of evaluating the project system, to assess the potential effectiveness of a single conservation solution by applying a logic model rather than a framework, and tying short term activities with intended impacts. This could reaffirm the current resourcing and strategy, or instead highlight gaps. Put simply, logic models can be used during the life of the project to check whether “what has been planned” is still “what is happening”, to ensure a cohesive conservation solution.

2. Material and Methods

A mixed-methods approach to data collection was used. A TOC model was drawn up from information given in MWF internal reports and through semi-structured interviews with field staff (Mauritian Wildlife Foundation, 2012; 2013; 2014; 2015; Taplin, Clark, Collins, & Colby, 2013) .

Semi structured interviews were used as a tool to gather information for the TOC model. Interviews were conducted on a sample group of seven project staff, comprising a mix of field staff, coordinators and members of senior management. Five interviews took place in person at the start of September 2015, and two interviews were conducted over email during September 2015. An interview guide was followed and each began with an explanation of the study purpose. Notes were taken but no sound recorder was used, similar to the approach taken by Black, Meredith & Groombridge for their appraisal of a field conservation project in East Africa (2011) . The interview scripts were coded and then analysed for three key areas; understanding of project objectives, personal development and the impact of experience in the role. Interviews were coded for project aims that were correctly indicated by the participants, and how these had been communicated to them. Communication of these objectives was either through official channels, for example at an induction or training session, or had been inferred by the member of staff themselves. The participants’ attitudes to personal development and self-led learning were analysed, by coding the length of time since reading about a biodiversity topic and their interest in carrying out research. The impact of experience was coded through length of time in role, attitudes to their own level of experience and the perceived importance of formal training or inductions. Coding of the scripts was straight-forward to carry out due to the design of each interview, which resulted in short statements from each participant on particular topics.

Interviews with project coordinators were used to estimate the amount of time that staff generally spent on each activity during the month, with seasonal differences taken into consideration. Activities were split into two categories-those which were directly related to conservation processes, and supporting activities. The Conservation Excellence Model was used as a reference for semi structured interview scripts. The model has previously been used to evaluate projects by providing a focus on management and project process rather than conservation activities alone (Black & Groombridge, 2010). It was used to ensure that interview questions covered process aspects of the project as well as conservation actions, and to act as a framework so that comparable questions were asked to project staff at different seniority levels.

Analyses of internal reports were carried out, with a particular focus on utilising data that is regularly collected but isn’t part of the conservation work. The days worked by each individual staff member over the life of the project were taken from monthly internal report documents and entered into a spreadsheet. The data were recorded as a percentage of each month worked. For example, an employee working three weeks of a four-week month was represented in that month with a value of 0.75, whereas a complete month was recorded as 1. This follows a methodology similar to 'full time equivalent’ reporting used in business and education (OECD, 2007) . If the employee was absent for the whole month, such as for a holiday or secondment to another project, then the data was recorded as 0. After the employee had left the project, the cell was left blank. Absence was recorded for staff returning another season, with the assumption that knowledge from prior seasons was retained.

The annual nesting activity for the Mauritius Fody was also collated from internal reports. This information gave an indication of the busiest periods for the project, and showed how the workload had increased as the population grew. This was compared to the time spent on each type of activity.

The inputs, constraints, processes, outputs, outcomes and impacts for the TOC model were mapped in a flow chart, using information from staff interviews and internal reports. A draft was sent to one of the project managers for review, to ensure its accuracy. The intended impacts were gathered in conversation with MWF senior management. The weighting of each activity according to time invested in them was added to the TOC chart following analysis of this data. It was used to build a general understanding of the project dynamics, and link current conservation activities with desired outcomes in order to assess whether improvements could be made.

The evaluation took approximately four weeks to complete. One week was taken for data collection through interviews and report analysis. Three weeks were taken for analysis and refinement of the method, but this was part of a degree programme and included a literature review. This time frame demonstrates some of the usual barriers to conservation evaluation; shortage of funds or resource, and the cost of independent review (Bottrill et al., 2011) . Data collection time could be greatly reduced if internal documents were already in a more suitable format for analysis, such as a spreadsheet or database, or if the study was conducted by persons already familiar with the project. Semi structured interviews could be carried out with fewer individuals if the singular purpose is to populate a TOC model and not to assess the staff members understanding of long term objectives. An exercise in simply mapping out the program activities and linking them to impacts could take less than a day when carried out by an individual already familiar with the project.

3. Results

The programme evaluation produced several key outcomes; a summary of current activity with resourcing requirements, a detailed linear TOC model, and results specific to the Mauritius Fody case study.

3.1. Case Study Outcomes and Impacts

The number of hours spent on conservation activities that directly contributed to long term impacts on the target species was found to be 47% of time available (see Figure 2 and Figure 3). Appendix 1 shows the full TOC model as defined for the Mauritius Fody on Ile aux Aigrettes.

Other conservation activities led to management and process outputs, or to further knowledge of the ecosystem in which the species exists. Approximately 27% of time is taken up with conservation activities such as data collection, population monitoring and gathering knowledge of the ecosystem. Supporting activities and personal time comprised 25% of that available. An analysis of staffing availability that considers annual leave, sickness and other absences is not included here, with time available considered to be after these absences.

3.2. Case Study Staff Experience

A number of observations were made surrounding staff hands on experience and the potential impacts of knowledge retention on productivity. The “cumulative average hands on experience” was calculated per month, being the average amount of time that the staff on site at the start of month had been working on the project.

Staff development and on-the-job knowledge was discussed with managers and coordinators during the semi structured interviews. It was found that they considered new employees to be competent after three months on the project, which we will define as the individual’s capacity to perform their tasks accurately and without supervision. It was also thought that employees generally reached peak performance after five months, which included the ability to perform more technical and sensitive tasks such as bird ringing or nest accesses.

Analysis of the staff resource data shows that the minimum level of experience is often not reached by all members of the team, due to some short volunteer placements during the year. A summary for the last four project years is shown in Figure 4, with the low season of April to June excluded. Short term volunteers, who attend for three months or fewer, might not be sufficiently experienced to assist competently at the peak times of breeding activity between November and January. Longer term staff might not reach peak performance before Fody breeding season due to the timing of volunteer recruitment, which could impact human productivity and the quality or precision of information.

4. Discussion

4.1. Setting Priorities and Linking Activities to Outcomes

27% of current activity is providing information about the ecosystem or species, but has no measurable direct impact on the survivorship of the Mauritius Fody. This information is key to the management of the species and hence is an important part of the project, however if there was a shortage of financing or staff available to carry out the work, the activities in this area could be the ones to reduce. Conservation activities have already had the required effect on the Fody population of Ile aux Aigrettes, so this could mean that confidence can be taken in the success of the existing strategy, allowing population monitoring to reduce and for resource to be focussed on the next stages of the project.

Systems behaviour charts have an emerging use in conservation, drawing on established control chart methods used in commerce and industry. Systems behaviour charts and decision triggers offer conservation leaders new insights into the behaviour of a system and allow them to monitor specific thresholds that prompt them to act if the system enters an undesirable state (Black, 2015; Cook, de Bie, Keith, & Addison, 2016) . Limits can be set for observable trends in order to detect declines in otherwise stable endangered species populations, and these methods have already been implemented in several countries (Addison, de Bie, & Rumpff, 2015; Timko & Innes, 2009) . Such methods allow conservation leaders and project managers to focus their attentions elsewhere while reducing management operations or dedicated resource (Cook et al., 2016) .

4.2. Process Indicators

The TOC model can be used to generate performance indicators. In this study, measures of outputs and outcomes already defined by the project were used; a change in the IUCN Red List classification after a few years indicated that the project was achieving its aims (Birdlife International, 2015) . The wild Mauritian Fody population is also stable enough that it can withstand the deliberate capture and relocation of some individual birds (enacted between 2008 and 2011) without ill effect, which can enable a new sub-population of the birds to be established at a different site (Birdlife International, 2015) . These events would have informed the management team that their activities were having the desired effect. Now that the project has reached stability, other indicators might be required in order to measure the performance and process effectiveness of the programme. Black, Groombridge & Jones similarly encouraged conservation leaders to focus on the immediate needs of the species and ecosystem, applying knowledge to set priorities that are appropriate to the current situation (2011) .

Collecting additional information about the ecosystem also could provide indicators about changes to the environment or emerging risks to the species, which is particularly important for an endangered species. The opportunity to carry out an evaluation in this way is only possible because of the abundant data collected, so this information feeds back into a refinement of management processes, however any time spent on data collection is only useful if all the data is used. Data collection at MWF follows a rough “80:20 rule”, using a methodology popularised by Joseph Juran (1951) . MWF aim to maximise the amount of information provided about the system by examining crucial elements that can explain around 80% of species population dynamics at that time. This assumes that accurately describing the remaining 20% of the system requires increasingly more effort with diminishing knowledge returns. An important question that arises from this evaluation is whether or not the same results could be achieved by reducing the time spent monitoring it. A recent study carried out on the Pink Pigeon (Nesoenas mayeri) in Mauritius found that monitoring effort for the species could be reduced, without impacting the accuracy of population figures (Concannon, 2014) . Black and Copsey note that a conscious effort should be made by leaders in conservation to overcome the need to seek more data rather than take action (2014) .

An increase in activities directly impacting the target species, however, means that less time is available for species studies or other areas of development. More resource dedicated to primary conservation activities might not lead to any improvement in the population stability, or there might not be anything further that could be done to improve the situation.

These results could be incorporated into annual planning reviews, to assist with resource management or for conservation activity planning. The information might also be provided to external funding bodies to demonstrate an efficient use of resources. Whereas the Mauritius Fody recovery project has been shown to link activities and resource allocations to desired long term outcomes, this can be contrasted with findings in Samoa where 41% of conservation programmes under evaluation were found to make only a partial contribution to their primary objectives (Bottrill et al., 2011) .

4.3. Knowledge and Effectiveness

Managers of the Mauritius Fody project face a potential trade-off between an individual staff member’s development and overall organisational productivity, in that short term volunteers on the project can sometimes be available only long enough to become competent. After these individuals have reached peak performance, they often leave the organisation and move to another. This benefits the individual in that they can gather a wealth of experience, but it unfortunately means that the organisation spends a large amount of time training staff whom then leave the project once they begin to excel in their tasks.

Many conservation organisations are volunteer led, and this brings many benefits, however the organisation itself may have more to gain by lengthening volunteer contracts in order to make the most of staff at their peak performance. Similarly, it could be worthwhile to assess when staff have become expert at a task, and to align this with peaks of seasonal activity. The Fody project is busiest during the breeding season from November to January and much of the activity during this season, such as bird ringing or nest access, requires a significant amount of training before an individual can carry them out without supervision. The recruitment process at MWF currently has new volunteers starting a six-month placement in July or early August, but if this was moved forward by one month then their team members might be able to find more nests or take part in more skilled tasks at this busy time. This in turn could relieve pressure and workload from the more experienced and senior staff members, however it would mean hiring staff when there is not much activity. A review of bird breeding activity would be required each year in order to monitor whether this peak season was moving back or forwards, to ensure that staffing patterns were adjusted appropriately.

The project coordinator was one of the strongest positive influences on the Mauritius Fody project, providing most of the experience and continuity over the periods evaluated in this study. The average experience per month was significantly reduced when the project coordinator was excluded from this analysis. When there is a change to the individual in this role, it could result in a sizable reduction in productivity for the project. The project coordinator might also be the only employee who is sufficiently experienced to carry out the more technical tasks, whilst also needing to supervise the newer team members. The implications of this are that the other team members have less autonomy and more workload is burdened upon the team coordinator.

4.4. Evaluation Process Methods

The initial evaluation strategy included several data collection methods which proved informative, and some that yielded less insight into the successes of the case study project. The TOC model, semi structured interviews, and review of internal reports were the basis of most process insights. There was a large amount of data available for analysis, because there is an established management process for recording information on all aspects of day to day activities, with regular and well defined reporting requirements. A review of publications based on data collected from the MWF Mauritius Fody project was also carried out, but the results were not used as part of the project evaluation. The impact factors of such publications or the number of citations per paper might be used, as a measure of external impact of the organisation.

The linear TOC used in this program evaluation doesn’t show all of the feedback processes. Some outcomes feed back into knowledge of the system and effect the input activities, and so the case study model is not complete. Further ecological studies and analysis of management processes would be needed to provide knowledge of more complex feedback loops.

5. Conclusion

Mid-term evaluations can be used to check against project milestones, to benchmark against other programmes or to simply think about the project from a fresh perspective. They can be used to identify areas of slack resource, or activities which could be scaled down in order to improve the overall effectiveness. The value of staff experience and the ways in which individuals learn while working their role is an important consideration for leaders because these factors have a direct impact on the effectiveness of their workforce. It can be difficult for individuals to get all the experience they need while workloads are not at a peak, but these times allow them to become comfortable in their everyday tasks, even if this leads to idle time.

The Theory of Change is a useful tool to assist leaders in understanding the importance of institutional knowledge, investment in experience and ecosystem knowledge. Conservation managers can develop a TOC to better explain interim outputs, set performance indicators and to relate these to impacts that are realised over the long term. A TOC could be devised by leaders of other conservation projects through a similar process of internal document review and by carrying out semi structured interviews with their staff to ensure that the programme aims are communicated and understood. Prioritisation of work is important to leaders, and particularly those in the field of conservation, because resources can be limited and decisions about which action to take first will commonly lead to a worsening situation in the areas where no action is taken.

Once established, the TOC model can be used in regular planning activities to help set priorities, and so that mid-term evaluations become part of management process. The conservation actions of a project may have regular reviews, perhaps prompted by annual funding decisions, but the management processes of conservation programmes can also benefit from the type of in-depth evaluation which this study has described.

Acknowledgements

We thank the staff at Mauritian Wildlife Foundation, particularly the interviewees and Mauritius Fody project team. In addition, we thank the three anonymous reviewers for their helpful comments on earlier drafts of this manuscript.

Cite this paper

Stebbings, E., Copsey, J., Tatayah, V., Black, S. A., Zuël, N., & Ferriere, C. (2016) Applying Systems Thinking and Logic Models to Evaluate Effectiveness in Wildlife Conservation. Open Journal of Leadership, 5, 70-83. http://dx.doi.org/10.4236/ojl.2016.53007

References