Abstract

Purpose: The implementation of routine outcome measurement was initiated as a quality improvement initiative in a unit delivering intensive functional rehabilitation for people with lower limb amputation. Two years post-implementation, completion rates remained low which raised the need to gain an in-depth understanding of the factors that might impact the systematic use of Outcome Measures (OMs). Method: A qualitative exploratory study embedded in the ongoing quality improvement initiative was designed. Data were gathered through a focus group with members of a multidisciplinary rehabilitation team. A deductive content analysis was performed using Consolidated Framework for Implementation Research (CFIR) as a guide to explore factors that impact routine outcome measurement. Results: Respondents perceived OMs as valid and offering clear advantages in clinical practice. At the organizational level, lack of fit with clinical practice, loss of project leaders and lack of clear management directives had negative repercussions on the use of OMs. Conclusion: Our results suggest that a dedicated project leader throughout the implementation process and effective communication may contribute to bypassing barriers associated to practice changes leading to a more systematic use of OMs among clinicians.

Keywords

Implementation, Outcome Measures, Quality Improvement, Rehabilitation, Health Care Professionals

1. Introduction

People who have undergone a Lower Limb Amputation (LLA) are impacted in functional, psychological and social areas of their life. Rehabilitation programs for people with LLA encompass many interdisciplinary interventions whose common goals are the improvement of mobility, functioning, and personal independence to achieve optimal social participation. Clinicians providing rehabilitation services for people with LLA increasingly need to use Outcome Measures (OMs) to demonstrate that the interventions they provide are effective and reach the needs of the person [1] -[3] . However, outcome measurements for persons with LLA raise a dual problem. First, despite the availability of generic and specific instruments [4] [5] on various rehabilitation outcomes, there is currently no consensus on which OMs to use upon discharge from inpatient rehabilitation [3] [6] -[8] . Nevertheless, mobility with and without prosthesis, functional independence, prosthetic profile and health-related quality of life are among the most commonly evaluated domains [2] [3] [6] . Second, the routine use of OMs among rehabilitation professionals is rarely implemented effectively [9] . Deathe [7] conducted a survey among Canadian rehabilitation professionals and found that a non-standardized checklist of daily activities was the most common OM used upon discharge of persons with LLA. They also reported that among those who used some formal measures, only 33% integrated self-reporting measures into their evaluation process. Depending on the context, multiple determinants can act as either facilitators or barriers in influencing the level of implementation of routine outcome measurements. Some of these determinants might be on a professional or team level, the organizational level, or they might arise because of the inherent characteristics or attributes of the OM being implemented [9] [10] . Our article explores the issues related to outcome measurement adoption in greater depth.

2. Context of the Study

In 2003, a specialized urban rehabilitation hospital in Canada started a quality improvement initiative in a unit delivering intensive functional rehabilitation for people with LLA. This quality initiative emerged from the need to objectively capture the ultimate rehabilitation goals—optimal quality of life and social participation. At that time, these outcomes were not assessed during or at the end of the rehabilitation process. Acknowledging this situation, a multidisciplinary committee (including the program manager, the clinical coordinator and one representative from each of the following disciplines: physiotherapy, occupational therapy, social work and psychology) was formed to review the OMs used with the LLA population and to obtain a consensus among therapists on the selection of valid and reliable measures to assess the rehabilitation outcomes. The selected measures were implemented into clinical practice as part of a research project and their benefits were examined [11] [12] . At the end of the project, the committee revised the selected OMs and agreed to maintain a subset of these OMs in their clinical practice. The implementation of this subset of OMs began in 2009. The process was led by the clinical coordinator.

Description of the Selected Outcome Measures

The rehabilitation professionals decided to implement one generic and two condition-specific OMs. 1) The assessment of Life Habits (LIFE-H) [13] —a validated and reliable questionnaire on social participation that evaluates a person’s difficulty and the assistance they require while performing life habits. 2) The Prosthetic Profile of the Amputee (PPA) [14] —a LLA specific questionnaire developed to determine predisposition, enabling and facilitating factors potentially related to the use of prosthesis. 3) The Prosthesis Evaluation Questionnaire (PEQ) [15] evaluates the person’s functioning and the quality of life of people with a lower limb prosthesis. All questionnaires were self-administrated or administrated by interview by clinicians during admission, discharge and during their three month follow-up (the three month follow-up is part of the routine rehabilitation process).

Throughout the implementation process, from 2009 to 2011, usage rates of the selected OMs remained low varying from 2.7% (PPA) to 20% (PEQ) with noticeable variation over time. For instance, usage rates of the PPA, administrated at discharge and at the three month follow up, went from 2.7% in 2009 to 0 in 2010 and increased to 8.4% in 2011. Given the inconsistent use of the OMs, a qualitative approach appeared best suited to explore the factors that may affect the use of OMs in this rehabilitation setting. More specifically, the objective of our study was to identify what were the main individual and organisational factors that influenced the systematic use of selected OMs among a multidisciplinary rehabilitation team working with people with LLA.

3. Methods

We conducted a qualitative exploratory study and embedded it in the ongoing quality improvement initiative of the LLA program. We convened a focus group so as to gain an in-depth understanding of the factors that impede or facilitate the implementation of routine outcome measurement. In the context of this study, the focus group was identified as the best method to explore and clarify views, and also to share different experiences according to each participant discipline and practice context regarding the use of OMs [16] . The focus group approach was also favored for its capacity to solicit group interaction and maximize the ability to leverage a broad range of experiences and perspectives. In this particular case, the clinicians knew each other and worked as a stable team for more than 5 years, so it was a “natural” means of gathering data by reproducing the group meetings where clinicians are used to communicating with each other. This allowed the research team to make observations that approximate naturally occurring events [16] . Furthermore, observation and analysis of the group’s dynamics allowed the research team to highlight convergent and divergent opinions and shared experiences [16] [17] . The focus group was led by one of the authors (LP) and a research assistant, both experienced at conducting focus groups. The two interviewers were not involved in the LLA program. One acted as a moderator (LP), facilitating the discussion, and one as an observer, documenting interactions and the general atmosphere. The focus group was audio-recorded and transcribed verbatim. The discussion guide was developed based on current knowledge of determinants of routine outcome measurement [9] and barriers and facilitators involved in the implementation of innovations in health care [10] . Open-ended questions were used to facilitate discussion and prompts were used when certain topics were not specifically addressed by the participants. Consent was obtained at the beginning of the focus group.

3.1. Participant Selection

All the professionals on the multidisciplinary team and the clinical coordinator (n = 16) who work in the LLA program and have some experience in the use of the selected OMs since they were implemented were eligible to participate in the study. A convenience sample was used to recruit participants. The program manager communicated to the team members the invitation to participate in the focus group. Clinicians who expressed interest were then contacted by the first author and invited to participate in a two-hour focus group. A total of five professionals representing all disciplines and one clinical coordinator volunteered to participate.

3.2. Data Analysis and Conceptual Framework

The verbatim was verified for accuracy by the primary author. Data was imported into QDA Miner 3.2.3 (qualitative analysis software) and a deductive content analysis was performed. This type of approach is appropriate when the structure of the analysis is performed on an existing theory or model [18] [19] . Our structured analysis was guided using a predefined coding scheme based on constructs taken from the Consolidated Framework for Implementation Research (CFIR) [10] . This framework, recently developed by Damshroder [10] , is the result of a synthesis of existing implementation theories and includes a list of constructs that are known to influence the implementation of interventions. The CFIR describes five major domains, each of which contains several constructs that interact in complex ways that influence the effectiveness of implementation. For instance, the outer setting domain refers to the economic, political, social and legal framework within an organization’s operations. The inner setting domain is the medium through which an intervention must navigate to be disseminated and implemented effectively. Individuals involved or affected by an innovation are also part of the internal context. The characteristics of the intervention and the process of implementation must also be considered as they can positively or negatively influence the level of implementation of an intervention. In the present project, we used the CFIR as a guide to classify factors that might have impacted routine outcome measurements and that help explore the interrelations between these factors.

We reviewed and coded the data according to these domains. New codes were added to describe additional prominent factors that emerged during the analysis and that did not match the pre-defined domains and constructs of the CFIR. Factors were then defined as being a facilitator or a barrier to the use of OMs. The multidisciplinary team served as the analysis unit. The initial coding was done by the primary author, an experienced physical therapist with an understanding of the contextual and practical environments of professionals of the LLA program. Regular consultations were held among researchers during the coding and interpretation of the data to ensure its appropriateness. Disagreements were discussed by comparing the verbatim with the operational definitions of the various constructs and a consensus was reached.

Since complete objectivity is impossible in social research, several methods were used to increase the confirmability of our results. The authors, met regularly throughout the analysis phase to discuss progress and findings and all results were reviewed by the two authors to minimize any interpretation bias. Furthermore, to assure dependability, a research protocol was written prior to study, the focus group was audio-recorded and transcribed, software was used to code the transcript and document coding decisions; and a codebook with codes’ definition was produced and used when analyzing the transcript.

3.3. Ethical Considerations

The research protocol was approved by a research center ethics review board (465-1109).

4. Results

All five domains of the CFIR were raised by participants and influenced the systematic use of OMs in clinical practice. However, not all constructs within each domain of the CFIR were discussed by respondents. Identified constructs and their contextualized definition are given in Table 1.

An overview of factors and their impacts (barrier or facilitator) on the use of OMs are given in Figure 1.

4.1. General Impressions

The general atmosphere was pleasant. We felt that clinicians spoke freely about their experience with measurement tools and reached a deeper level of reflection as the interview progressed. Participants appreciated being part of the quality initiative process and being provided with an opportunity to express their experience as individuals and as a team.

4.2. Intervention Characteristics

All respondents were convinced of the strength and quality of the OMs and were aware that the selected tools had proper psychometric properties i.e. were valid and reliable. All perceived some usefulness to the systematic use of OMs. For example, respondents reported that OMs allowed them to assess the effectiveness of rehabilitation interventions and the quality of care. Respondents also saw the benefit of using OMs as a means to compare organizations with LLA programs. They also provided a collective set of clinical data about patients and enabled the creation of patient profiles:

Each OM contains things, I think we do in our practice and that will be evaluated anyway. I think that if we take each of the questionnaires in the database, in OT, in psychology and social work, in PT, it appropriately describes the patient rehabilitation pathway and provides a great picture of the patient.

The relative advantage of OMs was discussed in terms of clinical value. For some respondents, the use of certain OMs highlighted patients’ perceptions and the relative importance they gave to specific rehabilitation needs and thus helped them identify the patient’s rehabilitation goals more accurately. For other respondents, OMs allowed them to gain information that otherwise would not have emerged from the standard rehabilitation processes and to add precision to the current assessments:

Sometimes there are things that will stand out [with the OMs] that we will not get in the overall assessment.

Another respondent adding: Yes, the patient’s perception of their level of independence is different from our perception of their independence. With the LIFE-H, it was really like seeing their perception and how they evaluated themselves, I thought it was important. It allowed them to assess their own priorities versus the rehabilitation priorities.

Concerns were expressed with the applicability of OMs, particularly for patients with visual and cognitive impairments. All agreed with a respondent when she mentioned: “The patient really needs to understand that it has to be well done” and another respondent added: “What we need is questionnaires that are more user-friendly; we need questionnaires that are easier for the patient to understand, ones that are standardized and that can be self-administered.” One strategy used by clinicians to administrate the tools for patients with visual impairments was to adapt those tools to fit their patients’ needs by administrating the questionnaires by interview instead of self-administration as it was designed to be administered.

^*Constructs that emerged during results analysis that was not initially included in the CFIR.

4.3. Characteristics of Individuals

Motivation fluctuated throughout the project. In fact, all participants reported that their motivation to use the OMs decreased with the absence of feedback about patient changes between admission and discharge. Everyone

agreed with the comment made by one respondent: “Because there is no feedback afterwards, there is no incentive to do it. We want objective data to compare patients over time, but if there is no comparison afterwards, it becomes somewhat meaningless.” Poor knowledge of the goals underlying the quality improvement initiative and more specifically the implementation of OM contributed to the low motivation among a specific professional discipline and was seen as a major impediment to the use of OMs: “I think, for a project to continue running, it is important to have a consensus. To have a consensus, I think it is important that everyone understands the background and the goal of a project.” However, at the end of the interview, respondents expressed an interest in revising and summarizing the project.

Attitudes to use of the OMs were generally positive and were associated with the perceived relative advantage of using OMs in their clinical practice. For example, a respondent mentioned: “When I completed the OM with the patients—when I had time—I thought it was worth it. And if it could provide data on the patient progress, it was satisfying.” Another respondent adding: “These tools are an excellent source of information and therefore, it is important to keep filling them out.”

4.4. Implementation Process

Participants highlighted several gaps in the implementation process. Most mentioned not being aware whether there was formal or informal planning to implement the OMs. While some respondents remembered a consultation process across and within disciplines to select the OMs, others did not recall any consultation or participation in the choice of OMs: “I do not know if we were consulted or not, but we don’t see it as our project.” Some respondents remembered meetings where guidelines on administering OMs and their use was explained and written documents were provided. However, they also expressed concern with the long delays between the consultation, implementation and feedback processes. Every one of the respondents expressed the need to revise the use rules of the OMs at least once a year to ensure their standardization across clinicians. Furthermore, all clinicians were critical of the lack of formal planning by the organization regarding the training and commitment of new employees for the use of OMs:

Well, it was not requested [training a new employee]. We had a new [naming the type of professional] in the program last year and I certainly didn’t give it to her [how to use the OMs], I haven’t informed her. I don’t know if it would have been my role to inform her actually. I don’t think so.

Another gap that emerged in the discussion was the loss of implementation leaders (program manager and clinical coordinator) who have been relocated to other positions in the organization. Clinicians were critical of the fact that following this loss, there were no formally appointed leaders and they perceived this as a lack of organizational support. They all expressed the need for a project manager (formally appointed leader) and a champion to support the use of the OMs: “They all left at the same time, there was a vacuum.” A respondent specifies: “Yes, we need a project manager” and another one added “Someone who can fire up the team”.

Feedback on the data collected was mentioned as an influence on the use of OMs. Clinicians reported receiving feedback only once from the manager during an annual program meeting. This feedback provided strong but temporary motivation to use OMs during the few weeks following that event:

Wow, it was so great. We really should continue to administer it [the OMs], it provided us with so much information. We left our annual meeting feeling really motivated, but then reality set in, the way we usually do things comes back, and we have to prioritize, unfortunately.

4.5. Inner Setting

Readiness for implementation was expressed by respondents in terms of the commitment of key people and training on use of the OMs. Training and familiarization with the OMs was perceived as adequate at the beginning of the implementation. However, the absence of follow-up and systematic training of new employees led to the perception that the program did not remain in a state of readiness throughout the project. In fact, respondents felt that training the newly hired professionals was highly dependent upon the importance attributed to the OM by the professional’s mentor.

Respondents did not feel tension to change their practice nor did they perceive that there was a clinical need to introduce OMs in their clinical activities, as a respondent reported: “It’s not like we wanted to save the program. I don’t think [outcome measurement] was a common concern of the clinicians.” There was also a lack of awareness about the project’s goals. Even several years after use of the OMs, many clinicians reported not having a sufficient understanding as to why and for whom this project was running. This was perceived by all respondents as a major reason not to use the OMs. A respondent mentioned: “It has to be meaningful. I embarked on the project without really knowing where all this was going.” Compatibility or fit with clinical practice was discussed extensively and was a major factor that negatively impacted the use of OMs. All the respondents perceived the use of OMs as extra work and time, especially for patients with visual impairments. Furthermore, many expressed that time spent documenting OMs was considered of less value than time spent providing therapy to patients:

When patients were able to complete the questionnaires, I gave it to them and they came back to see me if they had questions. Questions left unanswered, were discussed with the patient. So I thought it was not a waste of time because that day, we did not miss any therapy sessions and the patient did not see it as a waste of their time. We had our data. But we certainly can’t do the questionnaires for all patients. For those who can, it’s a winning situation.

Another respondent specified: “One reason for not administering the OMs is because it takes away from a patient’s clinical time.”

A major communication gap occurred at various levels (horizontally among professionals and vertically between professionals and the managerial team) and throughout the implementation process. Goals and objectives were not clearly transmitted over time to newcomers nor was feedback about the value of patient’s outcomes from a quality improvement initiative. This meant there was a lack of adhesion and cohesion by clinicians regarding the project: “When we don’t understand all the aspects or facets of a project, it makes it more likely that we will set it aside when our workload increases.”

4.6. Outer Setting

One external factor had an impact on systematic use of OMs: the legal context in which professionals practice. All the clinicians belong to professional associations that define various professional rules and charting is one of the defined and legislated activities. Respondents perceived conflicting issues between documenting OMs and following the charting rules of their respective professional associations. Respondents felt the use of OMs led to duplicate information, hence a loss of effectiveness: “We cannot exclude some aspects of our clinical report because patients have been assessed with the LIFE-H, there are requirements by our professional associations; the OM cannot replace what we have to do for our clinical assessment.”

5. Discussion

The aim of this study was to gain an in-depth understanding of the factors that influence the use of OMs in the clinical practice of rehabilitation health professionals. In our study, various factors acted in a complex and interrelated way to impede or facilitate the use of OMs. One main barrier was the lack of applicability of OMs to all patients with LLA. This issue was reported in the literature as being one of the three main reasons for not using standardized OMs in physical therapy practice. In a survey of physical therapists, 52% did not use any standardized OMs, and one of the three major reasons reported for doing so was that OMs were too difficult for patients to complete independently [20] . In another study, physical therapists reported not using OMs because they could not be administrated with patients with cognitive deficits or language barriers [21] .

Lack of time was another major barrier raised by respondents. The time issue seems to be a universal or shared barrier to the use of OMs [20] -[24] , however, this is reported and experienced differently across studies. Use of OMs can be perceived as time consuming due to the analyses, calculation, and scoring [20] and because of the search for the best measures, its administration and the subsequent discussion of results with patients and team members [23] . In our study, time was related to the planning of the assessment and the administration of OMs to patients for whom self-administration was not possible. Furthermore, this was seen as a loss of time dedicated to direct patient care, which is perceived by clinicians to be of higher priority than outcome assessment. This brings up the issue of competing priorities and reframes the perceived usefulness of OMs versus their concrete clinical use. The perception of “missing” clinical time when administrating OMs is a strong indicator that OMs were not integrated into the clinician’s practice but rather administered because it was asked for, or because it was part of the quality program initiative. It also reflects the relatively small advantage of OMs in comparison to traditional practice. Not using the information provided by OMs to set rehabilitation goals or to plan treatments, probably contributed to their decreased use [20] [25] [26] . The lack of appropriation for OMs in clinical practice is also reflected in the respondents’ comments about duplicating information i.e. clinicians see OMs as a duplication of the information they currently document. This might be related to their poor understanding or knowledge of the goals and benefits of OMs. Although clinicians have explicitly named the different goals of OMs (e.g. assessing the effectiveness of rehabilitation interventions), they had difficulty distinguishing clinical measures from OMs and understanding that one does not replace the other and that the same information, at some point in the patient’s pathway, can be used either as a clinical measure or an outcome measure. Results similar to ours, (on the level of knowledge of clinicians), has also been found in other studies [23] [25] .

While clinicians generally expressed a positive attitude to outcome measurement during the focus group, they were unable to transpose this into the systematic use of OMs. The relatively poor advantage and the lack of fit with current practice (time and duplication issues) overtook their positive attitude. This kind of paradox is not unusual and has been reported in other studies [27] -[29] .

As pointed out in an exhaustive review [30] on the implementation of health care innovations, the organizational context has an important influence on individual decisions to adopt and use an innovation. Over the course of our study, several major gaps (particularly in the inner or organizational context) greatly explained the low implementation levels of OMs among the professionals. One was the loss of the program’s manager and the project’s leader who acted also as the project’s champion. The adoption and routine use of an innovation is more likely to succeed if key individuals such as champions or opinion leaders support the innovation and if they, in turn, are supported by their organization [30] . In this project, change in management resulted in the loss of clear management directives on the use of OMs: clinicians did not know if they should continue to use the OMs. A final major gap was the lack of communication processes both at the team level (between clinicians of the same discipline) and at the program level (between managers and clinicians) throughout the implementation process. Effective communication processes are crucial at the beginning of the implementation process to rally individuals to the project and develop a common understanding of the project goals. This type of communication must continue throughout implementation to provide regular updates and feedback to those involved. Adding an organisational perspective to the relative advantage of OMs could act as an incentive to clinicians. In our study, clinicians did not perceive the need to change their practice. Effective communication early on in the implementation process to emphasize the importance of the project for the organization as well the specific organizational and professional goals of using OMs would have been helpful. Particular attention should have been given to the development of formal and informal communication networks within and across all hierarchical levels. This would have fostered a sense of cohesion and shared vision, and helped to develop the synergy necessary for the optimal participation of all stakeholders.

6. Study Strengths and Limitations

The use of a conceptual framework as the CFIR to guide the analysis is one of the main strengths of our study. One limit of this study is related to the study design. In fact, we conducted a single focus group in a single point of time and thus, our results reflect the perception of the respondents at that particular time and could be different if the focus group has been carried at another point in time. Given that the implementation process lasted more than 3 years and led to an unsuccessful implementation, it is possible that participants were more likely to remember what went wrong and inflate the role played by the barriers. However, this overestimation could actually be of benefit to organisations wishing to implement OMs as it would impose rigorous and thoughtful planning for a successful implementation process. Notwithstanding these limitations, because the analysis was guided by a conceptual framework, the selected case was a typical rehabilitation setting and the respondents represented the type of professionals usually involved in the rehabilitation of people with LLA, the results of this study might be applicable to other multidisciplinary teams in rehabilitation settings.

7. Conclusion

The implementation of OMs across several professional disciplines is a challenging task. Identifying and understanding the factors that impact implementation proved to be difficult. The use of qualitative theory-driven methods was particularly relevant in the context of our study as it allowed for a deeper understanding and a sound explanation of the use (or disuse) of OMs in a systematic way among a multidisciplinary team. Focusing implementation strategies predominantly on the inner setting and implementation process domains of the CFIR might also have contributed to bypassing certain barriers associated with the characteristics of OMs and professionals and thereby favour the increased use of OMs among clinicians. Future research should target longitudinal designs and repeated measurements to assess factors and process changes throughout the implementation.

Acknowledgements

We gratefully acknowledge the involvement of all study participants.

Declaration of Interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This study was funded by the OPPQ/REPAR. The funding body has no role in the design, collection, analysis, and interpretation of data; in the writing of the manuscript; or in the decision to submit the manuscript for publication. Dr. Poissant is supported by the “Fonds de la recherche en santé du Québec”. Diana Zidarov has a doctoral scholarship from the “Fonds de la recherche en santé du Québec”.

References

List of Abbreviations

LLA: Lower Limb Amputation OM: Outcome Measure LIFE-H: Assessment of Life Habits PPA: Prosthetic Profile of the Amputee PEQ: Prosthesis Evaluation Questionnaire CFIR: Consolidated Framework for Implementation Research