Received 28 August 2014; revised 28 September 2014; accepted 28 October 2014

ABSTRACT

Introduction: A laboratory’s ability to consistently produce high-quality and reliable results hinges on adopting laboratory standards that guide daily practices to ensure steady quality improvement. Although assessment is an extremely rewarding exercise in health care quality improvement processes, it is always considered very time consuming and expensive in developing world settings. A quarterly internal audit was conducted in 25 FHI360 supported Antiretroviral Treatment laboratories in the North West of Nigeria which can surely provide reference for other countries. Methodology: A checklist adapted from the World Health Organization/African Regional Office laboratory accreditation checklist was used to quantitatively evaluate 7 quality essentials (QEs). A team composed of technical staff from FHI360, State Ministry of Health and facility laboratory heads, conducted the audits, developed and monitored intervention plans. Information obtained with the checklist was captured in excel, validated and imported into Grappa Prism software version 5.0 for analysis. Results: Most (92%) facilities were at secondary level with (8%) at tertiary level. The mean total score on all QEs across the facilities was 63.34 ± 9.77 in quarter (Q) 1, 68.8 ± 10.91 in Q2, 72.59 ± 8.02 in Q3 and 72.72 ± 9.16 in Q4 (p ≤ 0.0001). The most improved QE through Q1-Q4 was organization and personnel (32.2%), while signage/bench top reference had an 18.6% point decline. In ranking facilities based on differences of total scores between Q4 and Q1, Kachia General Hospital was the highest with 27 point increase. Considering the mean percentage score for all quarters per facility, 4 had ≥ 80%, 19 had between 60% - 80% and 2 had ˂60%. The total non-conformities cited for QI-Q4 were 185, 100, 78 and 64 respectively with highest recorded in internal and external quality control and the least in facility and safety. Conclusion: We recorded some improvement in most QEs confirming the benefits of internal audits, reviews and follow-up. However, much more is needed in terms of technical assistance, capacity building, mentorship, and commitment at facility and state level to meet minimum acceptable laboratory quality standards.

Keywords:

Laboratory, Internal Audits, Quality Essentials, Non-Conformities, Nigeria

1. Introduction

Modern medical practice is increasingly dependent on reliable clinical laboratory services [1] . Yet in Nigeria, clinical laboratories are grossly inadequate with weak infrastructure [2] and Quality Management Systems (QMS) are uncommon like in other developing countries [3] [4] . Many laboratories are faced with numerous challenges including improper record keeping, documentation of procedures and lack of internal auditing. Family Health International (FHI) 360 with funding from the President’s Emergency Plan for AIDS Relief (PEPFAR) through United State of Agency for International Development (USAID) and Global Fund to fight AIDS and Tuberculosis and Malaria (GFATM) supported the Government of Nigeria to strengthen laboratories in 26 Public Health facilities in North West region of Nigeria for comprehensive Antiretroviral therapy (ART) services from 2005 to 2012. The key activities and support provided to strengthen the laboratories include, renovation of infrastructure, provision of alternate power sources and running water, procurement and installation of equipment, training of human resource, supply of commodities and reinforcement of the supply chain and logistics management systems. Others include establishment of the culture of planned preventive maintenance (PPM), introduction of laboratory quality management systems, registration for proficiency testing programs, development of policies, SOP, job aids, reporting and monitoring tools as well as improvement of information management, regular on-site technical assistance, monitoring and mentoring.

Although these interventions led to improved laboratory services, there was no standardized form of laboratory audit in place to accurately measure the level of improvement. The only form of assessment in place was Data Quality Assurance (DQA). However, the DQA simply checks for consistencies in results generated by the laboratory and its transcription into various forms and registers used for documentation. The main objective of DQA was to guide against transcriptional error and to ensure that the right result for the right patient is transmitted to the users and captured on the project’s electronic database, the District Health Information System (DHIS). Thus, there was limited focus on the processes to ensure accuracy and reliability of the laboratory test results.

Providing quality laboratory test results that are accurate, reliable, and reproducible will most effectively come through adopting laboratory standards that guide daily laboratory practices and institute a framework for quality management at the facility and network levels [5] .

There is the need for these laboratories to institute periodic and systematic assessments like internal audits [6] to improve important functions, work processes and their outcomes [7] , so as to ensure quality standards, generate reliable results and guarantee patient safety. Audits assess actual against expectation and provide feedback to staff and users thus creating room for continuous quality improvement for patient safety and laboratory efficiency.

Constraints faced by poor countries call for innovative approaches [8] [9] that would guarantee the minimum acceptable quality in laboratory processes while striving to meet the highest standards such as International Organization for Standardization (ISO)-15189 [10] for medical laboratories. In resource-limited setting, it may be justifiable to develop and use ‘fit for purpose’ local quality standards based on internationally recognized laboratory quality management frameworks or protocols [6] [11] [12] . In line with this, World Health Organization, Regional Office for Africa (WHO-AFRO) in collaboration with other stakeholders developed and launched a novel laboratory accreditation approach in Kigali, Rwanda in 2009 [13] . This effort led to the Strengthening Laboratory Management towards Accreditation (SLMTA) training program [8] , and Stepwise Laboratory (Quality) Improvement Process towards Accreditation (SLIPTA) tools [9] . Member states were encouraged to implement this novel approach to ensure quality improvement in laboratory services. Due to limited in-country technical capacity and resources to roll out the WHO-AFRO laboratory accreditation standards, it was piloted only in Infectious Disease Hospital (IDH) Kano. One of FHI360 supported health facilities in North Western Nigeria. Access to services with the highest standard possible remains the major focus in FHI360 ART program implementation.

Although assessment is an extremely rewarding exercise in health care quality improvement processes, it is always considered a very time-consuming and expensive process in developing world settings. Thus, FHI360 developed a simple internal audit checklist to mirror the WHO-AFRO standard which could be easily administered. As part of routine program implementation process, periodic internal audit was undertaken in the 25 ART laboratories in North West Nigeria to identify gaps with the view of recommending appropriate interventions and action plan to address the gaps and monitor improvements overtime. Here we present the findings and discuss on the challenges faced in the field.

2. Methodology

2.1. Project Design and Setting

This was a quantitative longitudinal audit of laboratory quality systems. The audit was done quarterly as routine process in ART sites supported by FHI360 in seven states in North West region of Nigeria (Table 1).

^*IDH Kano was involved in the pilot phase of WHO/AFRO laboratory accreditation process and not part of these audits. GH: General Hospital, FMC: Federal Medical Center.

2.2. Assessment Tool

On-site monitoring checklist was developed to mirror the WHO/AFRO SLIPTA check list [14]. The 12 Quality essentials (QEs) were compressed to seven composite quality management elements to suit the current practice and to make it simple and easy to administer. The quality management elements with various percentage weighting that were considered are: 1) Documents and records (10%); 2) Organization and personnel (10%); 3) Internal and external quality control (21%); 4) Inventory control system (10%); 5) Equipment (10%); 6) Signage/bench top references (17%); and 7) Facility and safety (22%). The modified assessment check list consists of a total of 46 (yes/no/ partial) questions. Full points are awarded if all requirement were fulfilled and partial when some requirements are not fulfilled and no point when all requirements not fulfilled.

2.3. Assessment

This quantitative audit was carried out quarterly between January to December 2012 for various laboratory quality management elements. Marks were awarded based on direct observation, review of records and documents, direct open-ended question techniques to staff or combination of all in some cases. The audit was carried out by FHI360 laboratory technical officers, facility heads of laboratory departments and State Implementation Team (SIT) laboratory focal persons. These focal persons are government staff from respective State Ministries of Health providing support to these laboratories as part of FHI360’s capacity building, sustainability and ownership strategy. The team was trained on the use of the audit tool however FHI360 laboratory technical officers participate in the audits, and also review audit reports conducted by SIT laboratory focal persons and seek clarification where applicable. Nonfulfillment of specific requirements per QE were outlined as non -conformities (NCs). The sum of a particular NC per quarter was derived by counting the number of facilities not meeting a particular requirement of the QE in each quarter. At the end of the assessment, summary of findings and observed NCs were reviewed with facility staff and a joint action plan to address such deficiencies was drafted. An example is shown in Table 2. Following an audit, a laboratory overall quality score of above 80% denotes good quality management system and such laboratories were encouraged to maintain continuous quality improvement. Laboratories with a score of 60% - 80%, were required to address all observed NCs within 30 days, while those that scored less than 60% required an urgent intervention to address observed NCs

2.4. Data Collection, Management and Analysis

We used the adapted checklist to capture data manually at sites. Data was entered into Microsoft Excel worksheets at FHI360 Kano office, cleaned, validated and sent to FHI360 country office for analysis. Data was imported into GraphPad Prism software version 5.0 (GraphPad Software, Inc., CA, USA) for analysis. Facilities were ranked on the basis of difference between total score for all quality essentials between quarter 4 and quarter 1. Quality essential scores were expressed as percentages to show the most improved quality essentials. Repeated measures analysis of variance (ANOVA) was used to compare quality essential scores by facility across all four quarters. A p-value of 0.05 was considered significant for all analysis.

3. Results

A total of 25 health facilities were assessed. Four (16%) facilities were located in each of Kano, Kaduna, Katsina, Kebbi, Sokoto and Zamfara states with one (4%) in Jigawa state. Twenty three (92%) were secondary health facilities while two (8%) were tertiary health facilities (Table 1).

In ranking facilities based on differences of total scores between quarter 4 (Q4) and quarter 1(Q1), Kachia General Hospital, Kaduna showed the highest increment with a 27 point increase. In all 18 (72%) facilities showed improved performance over four quarters. Seven (28%) facilities had a decrease at quarter 4 compared to quarter 1 (Table 3).

On analyzing the overall mean score for all quarters across facilities, only four (16%) facilities scored 80% and above while majority 19 (76%) scored between 60% - 79% and 2 (8%) of the facilities scored less than 60%. Kaura Namoda General Hospital Zamfara had highest overall average score of 83% (81.5 Q1: 81 Q2: 83.5 Q3: Q4 86.5%), while the least average scores was 59% Q1 46; Q2 46; Q3 79; Q4 65.5) (Table 3)

QE Percentage, Mean Score and Non-Conformities

The percentage score for each QE was determined through the four quarters, was calculated to see the trend in

HOD: Head of department, PEP: Post Exposure Prophylaxis, CAPA: Corrective Action and Preventive Action, LJ: Levey Jennings, SOPs: Standard Operating Procedures, EQC: External Quality Control, IQC: Internal Quality Control.

quality change per QE. Organization and personnel showed the highest improvements with a 32.2 percentage point change, while signage/bench top references had an 18.6 percentage point decline between Q1 and Q4 (Figure 1).

For the mean score of QEs across all facilities, statistically significant differences were found for the following QEs between quarters 1, 2, 3 and 4; documents and records, organization and personnel, internal and external quality control, inventory control system, and signage/bench top references (p < 0.0001) (Table 4). The difference in scores across all four quarters were not statistically significant for equipment (p = 0.70) and facility and safety (p = 0.10). However, a decreasing trend in scores was observed for signage/bench top reference.

Mean total score on all quality essentials was 63.34 ± 9.77 in quarter 1, 68.8 ± 10.91 in quarter 2, 72.59 ± 8.02

in quarter 3 and 72.72 ± 9.16 in quarter 4 (p < 0.0001) (Table 4).

The total NCs cited for QI to Q4 were 185, 100, 78 and 64 respectively. Details and a follow through of the NCs cited in at least 10 facilities at the first audit is shown in Table 5. The QE internal and external quality control had the highest (66) NCs in Quarter (Q1) while facility and safety was the least. For all the QEs, the NCs steadily reduced form Q1-Q4 with the exception of Signage/bench top reference.

4. Discussion

Through quarterly audits and monitoring visits, we have been able to evaluate and report on changes in the quality status of 25 laboratories in the North West region of Nigeria over a year period. Although the findings showed that the total mean score on all QEs was highest in Q4 (72.72 ± 9.16), only 4 (16%) facilities showed continuous improvement in quality performance over the one year period. However, quality management system implementation in most of these facilities is still suboptimal due to the fact that it is a new concept requiring more time and efforts to be entrenched in routine practice. Similarly, a 2013 report on laboratory assessment in Uganda showed that only 4.7% of 954 laboratories met or surpassed the lowest quality standard [15] . In addition, another 2013 report showed that 37 of 49 countries in sub-Saharan Africa had no laboratories meeting international quality standards [16] .

Resource constraints, limited technical knowhow and time factor in terms of available certified auditors for technical assistance, necessitated the design of a modified version of the WHO/AFRO quality assessment checklist to attain minimum acceptable local quality standards [6] [11] . A general qualitative appraisal of the laboratory situations served as guidance on where to place emphases in adapting the WHO-AFRO checklist. Other studies in Uganda [15] and Nigeria [17] have reported using modified version of this checklist.

It is worth noting that none of the four facilities that had 80% and above in the overall mean score were among the first five facilities ranked based on point increase between Q1 and Q4 denoting significant improvement in quality standard. This highlights the fact that the most improved facility may still be down the quality ladder. Ideally, we should strive at having both better ranking and improvement in mean score over time. Significant number of the facilities required urgent intervention to address gaps and specific identified challenges. None of the tertiary facilities was among the top four good performing facilities in terms of improved performance between quarters or overall mean score. This was at variance with documented report of a study in Nigeria on HIV Testing and Counselling facilities where the upper tier (secondary) facilities were better than the lower tier (primary) in quality standards [17] . The marginal increase in three and decrease in seven facilities (Table 3) in quality improvement between Q4-Q1 can partly be attributed to staff transfer and attrition as reported

GH: General Hospital, FMC: Federal Medical Center.

ported in the Lesotho laboratory accreditation implementation program [18] . This is more critical with the de parture of the head of departments like in Yauri GH, Daura GH and Dutsin Ma GH compounded with improper handing over or completed neglect of the process. Therefore new heads eventually takes over leadership roles without training or knowledge of quality system management implementation. Furthermore, supportive supervision was less frequent especially in Kebbi, Sokoto and Zamfara states where a few facilities (Illela GH, Yauri GH, Zuru, GH Shinkafi GH and KauraNamoda GH) are located in hard to reach areas. For effective resource management, integrated supportive supervisory site visit to these three states along the same axis were carried out with other program team members. On the other hand states such as Kano, Kaduna, Katsina and Jigawa benefited from more frequent and laboratory focused supportive supervisory visits and mentorship. However, as part of FHI360 effort to build capacity, ensure ownership and sustainability, the government of Nigeria (GON) staff who are State Implementation Team (SIT) focal persons for laboratory in Kebbi, Sokoto and Zamfara states mostly carried out the assessment and other laboratory related supervisions. Additionally, about half (48%) of audited facilities have only one laboratory scientist and one facility (Illela GH) had only laboratory technicians and coping with documentation and record keeping remain a challenge. Thus availability of qualified, teachable

Q1: Quarter 1, Q2: Quarter 2, Q3: Quarter 3, Q4: Quarter 4. p-value is significant when p < 0.05.

Q1: Quarter 1, Q2: Quarter 2, Q3: Quarter 3, Q4: Quarter 4, PEP: Post Exposure Prophylaxis, CAPA: Corrective Action and Preventive Action LJ: Levey Jennings, SOPs: Standard Operating Procedures.

and dedicated staff in some cases was a challenge.

Overall, there was an improvement in the quality standards as judged by QE mean score, QEs percentage and the trend of NCs over the quarters with the exception of signage/bench top references. The poor performance on signage/bench top references is partly due to policies in some facilities that prohibits wall posters in the laboratory. Significant improvement was seen in some of the QEs like document and records and inventory control system. This was, partly because even before the introduction of this structured internal audits, reports on completion of laboratory monthly summary forms was a key laboratory performance indicator monitored for all facilities. Information on commodity utilization and stock balance is mandatory for resupply of ART commodities, thus documents pertaining to the inventory control system like the combined report requisition and issued from (CRIFF) as well as Tally cards were mostly available. Furthermore, stock status is regularly verified on-site and any discrepancies promptly addressed to avoid stock outs, thus keeping lab staff conscious of this QE. The common NCs encountered in this study (Table 5) are not unique to these laboratories as similar NCs were reported during the assessments of medical laboratories in Lesotho [18] and Hong Kong [19] . In our study, NCs was more common in the area of internal and external quality control. This is mainly due to poor documentation of internal quality control practice and information on external quality control performance review was uncommon. A study by Chizizi et al. in Malawi reported the common absence of external quality assurance systems [20] in testing facilities assessed. The decrease in number of NCs over the period substantiated that audits and reviews contributed to quality improvement.

Some general challenges to smooth implementation of QM program include: Lack of required training and skills on quality management systems, insufficient number of laboratory staff available at sites, attrition, frequent transfer and more importantly poor staff attitude prohibits smooth and sustained implementation of QM programs. Some laboratory staff still do not realize the usefulness of quality management systems and consider these requirements as unnecessary extra burden. The complaint of extra work and no monetary incentive are the most common excuses in most facilities. Limited resources remain another major challenge. For example, in 2012 among all the expected trainings to cover laboratory quality management, only four trainings; (equipment operations, quality management system, good clinical laboratory practice and sample collection and sample referral) could be accommodated within the project budget for at most two laboratory staff per facility. Thus we resort to on-site step-down trainings by facility staff. The scope and quality of these steps down training could not be ascertained as putting the concept to practice has always been an uphill task. Furthermore, when financial resource is involved to redress quality gaps like servicing fire extinguishers and refrigerator, transfer of specimen, printing of documents, travelling for mentoring and supervision etc., there is much delay. This is mostly due to weak financial management systems within the facilities and the state ministries. However, FHI360 is working in collaboration with Deloitte Consulting, Inc. USA to provide capacity building in organizational and financial management to ensure better governance, ownership and sustainability of the program.

With this simple and easy to administer tool, we can quickly determine the minimum quality standard and evaluate output of technical support overtime. Furthermore, based on performance in these audits coupled with facility commitment and staff attitude, we have been able to make informed decision on facilities to present for the WHO/AFRO accreditation rollout. Namely, Hasiyya Bayero Pediatrics Hospital Kano, Federal Medical Center, Gusua-Zamfara, Sokoto Specialist Hospital Sokoto and Sir Mohammed Sanusis Specialist Hospital Kano. These results support the usefulness of a fit-for-purpose checklist and the benefit of the internal audit programs for assessing and monitoring clinical laboratories in our setting to ascertain if minimum level of quality standards is in place, while striving toward accreditation preparedness through SLIPTA WHO/AFRO and other international standards.

In as much as the overall improvement was minimal, the exercise was crucial for identifying weak points in laboratory activities and to allow for tailored interventions and targeted plans to match the limited resources and knowhow. Overall, more is still required to be done in quality management system implementation in our laboratories. The concept has not yet become a culture in these laboratories especially with the series of documentation and record keeping required coupled with human resources and attitudinal challenges. Similarly, some studies have reported that laboratory staff view accreditation process as being over bureaucratic [21] [22] and staff attitude [23] [24] possess a lot of challenge to implement quality management systems.

Some limitations of this study include; Non uniform assessment of performance in external quality assurance (EQA) in the facilities. Some facilities were at a disadvantage as they were not registered for other EQA other than CD4+, and registration of facilities for EQA was not at same time. Therefore, incomplete data specific to this QE was unavoidable within the audit period. Our adapted fit for purpose checklist did not allow for “Star 1-5” incremental level rating like in WHO-AFRO SLIPTA checklist [8] . Thus, it is difficult to match performance with other standards. However, with much caution, a score of over 80% in our assessment could likely translate to at least “Star 1” rating. Due to security challenges, quarter four audit assessment was conducted in about 80% of facilities by SIT laboratory focal person having limited experience in QMS and structured audits. Hence Q4 results need to be interpreted with more caution, even though it was reviewed by FHI360 laboratory technical staff. The ranking of sites based on Q4-Q1 difference (most improved site) does not directly relate to the highest/ best site in terms of quality standing as judged by average mean score. Thus a combination of both ranking and average mean performance is a better approach to assess overall quality improvement. However, this analysis to rank sites is very critical in our program as it allows us to rate the level of commitment and provides guidance in the choice of facilities to target in piloting special quality improvement initiatives. The frequency (quarterly) of the audits may not have allowed facility staff enough time to assimilate and understand the quality concept. This current audit focuses only on ART laboratory and does not reflect the quality of services offered in the facility main laboratory.

5. Recommendations and Conclusions

There is a need to institute a policy of periodic staff training, refresher trainings and competency assessment in facilities. Incorporating quality management system in laboratory training curriculum at universities and institutes of medical laboratory science where laboratory technicians and assistants are being trained is also needful. In-service training for laboratory professional on QMS in order to address knowledge gaps in quality management system should also be instituted. Respective State Ministries of Health and Health Management Boards should address training gaps by taking responsibility for training more laboratory personnel when implementing partner program budgets accommodates only a limited number of personnel, as part of sustainability and ownership strategy.

Temporal deployment of laboratory staff that are trained and skillful in quality management systems to other facilities to provide, peer mentorship and conduct on-site training for incoming heads of laboratories should be encouraged.

Advocacy to State Ministries of Health/ Hospital Management Board to minimize staff transfers should be carried out. In addition, work load analysis should be carried out for all facilities to determine workforce related issues and deploy laboratory staff to sites with human resource gaps to address excess work load syndrome.

Quality teams should be formed with clear term of reference and quality and safety managers different from the laboratory head should be appointed. Special motivational packages like periodic awards for excellent performance, travel awards to serve as peer mentors or conference support should be introduced. Quality improvement projects should be initiated and monitored across laboratories through quality indicators.

Proficiency testing (PT) should be extended to more facilities and cheaper PT options such as Medical Laboratory Sciences Council of Nigeria (MLSCN) PT scheme and PT alternatives like peer/inter laboratory comparison should be explored. The capacity of local biomedical engineers on equipment maintenance and repair should be strengthened and the culture of planned preventive maintenance enforced.

Audits and trainings should go beyond the ART laboratory applying an integrated approach [25] to minimize manpower shortages, knowledge gaps and encourage staff rotation. Customer satisfaction survey involving other technical collaborators with the laboratory team and patients should be instituted periodically.

We recorded improvement in most quality essentials substantiating the benefits of audits, review and follow up. This exercise has been used as a platform for informed decision on selection of facilities towards WHO/AFRO accreditation roll-out. More attention is required to improve on the QEs like internal and external quality control and signage /bench top references in our settings. Overall these laboratories still lack the quality standard to benchmark minimal quality requirements like “Star-1” rating of the WHO/AFRO accreditation scheme [8] . Continuous mentoring and monitoring of laboratories by knowledgeable and skillful laboratory personnel, commitment from facility technical team, management support at facility and state level is a key to improvement in laboratory quality standards. This approach can surely provide reference for other countries.

Authors Contributions

Conceived and designed of experiment: FEJ, TNY, HAM; Performed the experiment: FEJ, TNY, analyzed the data: OA, HAM, FEJ, ORN-M, KT; wrote the paper: FEJ, TNY, HAM; reviewed the manuscript: FEJ, TNY, HAM, ORN-M, OA, KT.

Ethical Approval

FHI360 Office of International Research Ethics (OIRE), North Carolina, USA, approved the study (IRB No. 537314-1).

Financial Disclosure Statement

Support for this study was provided by FHI360 Nigeria with funds from the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) through US Agency for International Development (USAID)

Competing Interest

The authors have declared no competing interests. The views expressed herein are those of the authors and do not necessarily reflect those of FHI360.

Acknowledgements

Our sincere appreciation also goes to all stakeholders in respective state ministry of health including laboratory staffs of all Hospitals in North West of Nigeria where this study was conducted.

References

NOTES

^*Corresponding author.