Introduction: Public-private partnership (PPP) has been adopted widely in infrastructure construction projects, and its application in medical and health service industry is increasing day by day. However, the performance of a PPP project could be affected by a number of factors and their interactions, due to its complexity in multiple stakeholders and stages. Previous researches on hospital PPP projects mainly focus on the factors which lead to success/failure, or the results of implementing PPP on hospital projects, or risk management, rather than its evaluation mechanism. So, to comprehensively promote and guarantee the effective implementation of the hospital PPP project, a scientif-ic quantitative evaluation method according to certain evaluation procedures and the performance evaluation system for hospital PPP projects are pro-posed from the perspective of the government.
Public-private partnership (PPP) is cooperation between the public and private sector that combines the abilities of partners involved to achieve better outcomes, and is seen as a revolution in construction industry, being welcomed by agencies and governments. What’s more, PPP projects lay more emphasize on hospital assets, with all focus on leveraging capacities and integrating resources of both public and private sectors, just to introduce funding and expertise into healthcare sector.
To achieve the common goals, both the public and private sectors need to sign an agreement and form a project company to help construct the project. In PPP hospital projects, the public sector is in charge of the operation of core health services, clinical services for instance. Meanwhile, private sector is concentrating on no-core services, such as safety secure and cleaning. However, there are still problems that occur in such projects, caused by complexity of multiple stakeholders and long periods of construction. So, research mainly focused on the risk management of hospital PPP projects, and quantities of studies put effort into how to save costs. However, in some projects, the government’s specific efficiency gains from part of the successful bidder’s proposal evaluation, and it may be possible to include performance indicators related to the availability of these efficiency gains when the project is operational. According to literature review and investigations, there are few examples of PPPs in the hospital sector in developing countries, and how to comprehensively evaluate the performance of a hospital PPP project from the standpoint of the public sector has not been proposed yet. In order to deal with such problem, the project company shall regularly monitor and analyze the construction and operation of the project according to the concession agreement, conduct performance evaluation with the relevant departments, and establish a mechanism for adjusting the price or financial subsidies according to the performance evaluation results and in accordance with the concession agreement. The quality and efficiency of the public services will then be guaranteed. Project companies should also consider public opinions as an important part of monitoring the implementation of the hospital PPP project and performance evaluation.
Based on the above research status, this paper sets output specifications and constructs the framework of evaluation indicators for evaluation, using analytic hierarchy process (AHP) method to assess the implementation of hospital PPP projects. Finally, the framework and its evaluation method are applied in a specific hospital PPP project as a case study. Therefore, the result of study can be provided to the public sector to regulate and guarantee the successful implementation of hospital PPP projects, and to improve the level of management of medical and health.
The study adopted a questionnaire survey approach using a self-administered questionnaire, released by mail. The mail survey approach was considered as feasible due to samples that are geographically dispersed. Respondents were enticed to responds with rewards. The survey was supported by personal face-to-face interview. The interviewers were guided by a set of similar questionnaire used in the mail survey. A total of 142 questionnaires were e-mailed and 69 questionnaires were collected in this survey, of which 58 were valid questionnaires. Among the effective responses, there are 13 experts, scholars in universities or academic institutions, 17 government officials, 28 design institutes, contractors and other business people. Most of the respondents have a rich PPP project (average 12 years, maximum duration of 38 years) experience. Additional 31 responses were obtained from the personal face-to-face interview. This technique will also help to reduce the rate of misinterpretation of questionnaires and to prevent respondent’s error.
The sample size of this questionnaire is relatively small, probably due to the following two reasons: Firstly, the application of PPP mode in China started late, there are not much experienced experts or scholars in this field as expected. Secondly, the questionnaire contains 22 primary indicators and 56 secondary indicators, which makes filling the questionnaire time-consuming, thus retard many respondents. However, the respondents of this questionnaire are targeted to the relevant personnel in the PPP fields within a certain scope, and the response rate is 40.8%. According to the Moser and Kalton research, when the questionnaire survey rate is less than 30% - 40%, the result of questionnaire survey is biased [
In the questionnaire, the output specifications and indicators through literature review were listed, and questions were written to inquire suggestions on the reduction/increase of indicators by respondents. While interviewing, pairwise comparison of indicators was asked and recorded as data to estimate weights of each indicator. After the evaluation system was created, an interviewee was asked to provide a score of each indicator to get the whole evaluation score for the output of the hospital PPP project which is actually put into operation.
The analytic hierarchy process (AHP) was officially proposed by the American operations researcher T. L. Saaty in the mid-1970s [
The basic idea of AHP is basically the same as the process of thinking and judging a complex decision-making problem. This method provides a basis for quantitative evaluation of indicators, selection of optimal solutions, and has been widely used.
The objectives of the decision, the factors considered (decision criteria) and the decision objects are divided into the highest layer, called Objective (Level 1).
The middle layer concludes factors to consider, guidelines for decision making, namely Criteria (Level 2). The lowest layer contains the alternatives when making decisions, called Attributes (Level 3). Hence, the hierarchical structure diagram is drawn [
The weight of each indicator is determined by means of pairwise comparisons of the activities to indicate the power with which one activity dominates another.
The pairwise comparison is established using a nine-point scale that converts the human preferences between available alternatives as equal importance, weak importance, strong importance, very strong importance and absolute importance, and each scale is empowered 1, 3, 5, 7, 9 relatively. As for Intermediate values between adjacent scale values, we use 2, 4, 6, 8 as transition scores. The comparison is based on experts’ judgment. Suppose n experts take part in the survey. Each expert’s opinion is obtained and analyzed individually to determine the weight vector ω i belonging to Indicator Group i (i = 1 − l). The following procedures provide a method to obtain the weight of indicators at Level 3 in
Then, the pairwise judgment matrix for the weight of indicators based on the kth expert’s thoughts (k = 1 − n) is an x-by-x nonzero reciprocal matrix P, as presented below (Pij means the pairwise value after comparing αi with αj):
P = [ 1 p 12 ⋯ p 1 x 1 / p 12 1 ⋯ p 2 x ⋮ ⋮ ⋮ ⋮ 1 / p 1 x 1 p 2 x ⋯ 1 ]
As an example, the eigenvector of indicator is calculated by
ω ′ 1 k = ( ω ′ 1 k 1 , ω ′ 1 k 2 , … , ω ′ 1 k n )
ω ′ 1 k i = ∏ j = 1 n p i j n
Then the eigenvector is normalized by
ω 1 k i = ω ′ 1 k i ∑ j = 1 n ω ′ 1 k j
Based on the scores given by Expert k, the weight vector ω 1 k of the indicators of Indicator Group 1 is: ω 1 k = ( ω 1 k 1 ω 1 k 2 … ω 1 k n ) .
At last, a way to prevent artificial error and a consistence check need to be conducted until the precise result is obtained. The maximum eigenvalue λ max is a measure of consistency of judgement.
λ max = 1 x ∑ i = 1 x P ω 1 k ω 1 k i
CR = CI RI = 1 RI ( λ max − 1 x − 1 )
CR represents the consistency ratio; CI represents the consistency index; Ri represents average consistency index, as is shown in
After all the results are calculated and the weight vector ω 1 k ( k = 1 − m ) are worked out, the weight vector for Indicator Group 1 is determined by:
ω 1 = 1 n ∑ i = 1 n ω 1 k = ( ∑ k = 1 n ω 1 k 1 n ∑ k = 1 n ω 1 k 2 n … ∑ k = 1 n ω 1 k x n )
The weight vectors ω 2 − ω 2 l for indicators at Level 3 of Indicator Group 2-l can be obtained by means of the above method. And the weight ω m for the indicator groups at Level 2 can also be calculated with the method.
According to the literature review and a questionnaire survey conducted previously, the output specifications and indicators can be synthesized and summarized into five categories, namely economic evaluation, operation & maintenance evaluation, medical technology evaluation, evaluation of hospital management level, public satisfaction evaluation [
As is shown in
The economic benefits of resource allocation shall be assessed by the use of resources in terms of personnel, beds and equipment. It mainly investigates whether hospitals can organize production factors such as personnel, beds and equipment according to medical needs during operation, maximize economic benefits, and avoid waste of resources caused by idleness.
x | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
---|---|---|---|---|---|---|---|---|---|
RI | 0.00 | 0.00 | 0.58 | 0.90 | 1.12 | 1.24 | 1.32 | 1.41 | 1.45 |
Evaluation Content | Primary indicator | Weights | Secondary indicator | Weights |
---|---|---|---|---|
Economic Evaluation | Economic benefits of resource allocation | 20% | Average effect of each staff | 40% |
Average effect of each bed | 30% | |||
Average effect of each equipment | 30% | |||
Financial indicator | 20% | Return on total assets | 30% | |
Net asset turnover | 20% | |||
Hospital operating benefit rate | 30% | |||
Operating cost growth rate | 20% | |||
The operation cost of hospital | 30% | Hospital operating direct costs | 50% | |
Hospital operating indirect costs | 50% | |||
Medical price management level | 30% | / | / |
Financial indicators are based on the internal perspective of the hospital to consider the financial situation of the hospital, the main indicators are: total return on assets, net assets turnover rate, hospital operating efficiency rate, operating cost growth rate.
The operation cost of hospital means to assess the economic status of hospital management and decision-making from the aspects of direct cost and indirect cost, classifying, recording, collecting, and analyzing reports on various expenses in the medical service process, analyzing relevant costs information.
The level of medical price management, that is, from the perspective of the degree of openness and transparency of the price, the procurement process, and the true and accurate price information, whether the hospital’s drug price and the medical treatment price are controlled within a reasonable level, and truly serve the people’s livelihood.
As is shown in
The evaluation of hospital operation and maintenance is divided into maintenance and management of construction equipment system and evaluation of logistics management work.
The maintenance and management of construction equipment system takes up 60% of “Operation & Maintenance Evaluation”, which makes it extremely important when a hospital PPP project is evaluated from this perspective. The hospital’s operation status is inspected from the perspective of hospital construction equipment hardware. The index is divided into construction equipment
Evaluation Content | Primary indicator | Weights | Secondary indicator | Weights |
---|---|---|---|---|
Operation & Maintenance Evaluation | The maintenance and Management of Construction Equipment System | 60% | Construction equipment maintenance and repair system | 30% |
Timeliness of repair | 20% | |||
The impact of the maintenance process on hospital operations | 30% | |||
The react of emergency | 20% | |||
Logistics management | 40% | Logistics management system | 20% | |
Parking lot, venue traffic organization is reasonable | 10% | |||
Canteen hygiene, diet safety | 20% | |||
Environmental sanitation | 20% | |||
Greening and landscape | 10% | |||
Sewage Disposal | 10% | |||
Solid Waste | 10% |
maintenance and repair system, maintenance technology advancement, maintenance timeliness, maintenance process impact on hospital operation, emergency Handling job evaluations.
The evaluation of logistics management is to investigate the operation of the hospital from the soft environment of daily affairs indirectly related to hospitals and medical treatment. The indicators are divided into logistics management system, parking lot, reasonable traffic organization, canteen hygiene, food safety, environmental sanitation, green landscape, Seven tertiary indicators for sewage discharge and solid waste disposal.
When evaluating the infrastructure of a hospital, special considerations should be given to the layout, function setting, medical procedures, safety requirements, sanitation requirements, environmental protection and supporting facilities of the hospital building; the climate and resources of the local area should be combined according to the principle of adapting to local conditions. So, the environmental, economic and cultural characteristics were evaluated in this paper.
Operational management evaluation is carried out during the operational management phase. The evaluation of the operation and management stage shall be carried out after the completion of the design scale and normal operation for more than one year (including one year), after the construction and equipment audit.
As is shown in
The evaluation of medical technology is divided into service level of outpatient and inpatient, surgical and anesthesia skill, and quality of nursing.
Evaluation Content | Primary indicator | Weights | Secondary indicator | Weights |
---|---|---|---|---|
Medical Technology Evaluation | Service level of outpatient and inpatient | 40% | Qualified rate of prescription | 25% |
Admitting and discharge diagnosis accuracy | 25% | |||
The success rate of critically ill patients rescue | 25% | |||
The average length of stay | 25% | |||
Surgical and anesthesia skill | 30% | The incidence of complications of the patients | 50% | |
The mortality of anesthesia | 50% | |||
Quality of nursing | 30% | / | / |
Service level of outpatient and inpatient is divided into four aspects, including the qualified rate of prescription, admitting and discharge diagnosis accuracy, the success rate of critically ill patients rescue, and the average length of stay.
Surgical and anesthesia skill is divided into the incidence of complications of the patients and the mortality of anesthesia, that is, accidents caused by improper operation of anesthesia during surgery.
Quality of nursing is evaluated from the perspectives of medical care ratio, guard bed ratio, nurse-to-patient ratio, implementation of job responsibilities, and number of patient care accidents. Different from the above grading, quality of nursing is assessed comprehensively, so secondary indicator doesn’t show up in
As is shown in
The evaluation of hospital management is divided into volume of medical service, level of medical service, evaluation of medical environment, level of medical information management, mandated task by the government, doctor-patient relationship and patient safety.
Volume of medical service is divided into three aspects: patients at emergency department, hospitalized patients discharged, and number of people who did surgery.
Level of medical service is divided into the reserved medical service level, waiting time at the service window, basic medical security service management level, clinical pathway management and electronic medical record establishment.
Level of medical information management is assessed from the following perspectives, namely surveillance system and intelligent management information system.
Mandated task by the government is evaluated from counterpart support, health education and public health emergency rescue.
Evaluation Content | Primary indicator | Weights | Secondary indicator | Weights |
---|---|---|---|---|
Evaluation of Hospital Management Level | Volume of medical service | 10% | Patients at Emergency department | 40% |
Hospitalized patients discharged | 30% | |||
Number of people who did surgery | 30% | |||
Level of medical service | 15% | Reserved medical service level | 20% | |
Waiting time at the service window | 20% | |||
Basic medical security service management level | 20% | |||
Clinical pathway management | 20% | |||
Electronic medical record establishment | 20% | |||
Evaluation of medical environment | 20% | / | / | |
Level of medical information management | 15% | Surveillance system | 50% | |
Intelligent management information system | 50% | |||
Mandated task by the government | 10% | Counterpart support | 30% | |
Health education | 30% | |||
Public health emergency rescue | 40% | |||
Doctor-patient relationship | 10% | / | / | |
Patient safety | 20% | Medical safety | 30% | |
Nursery safety | 30% | |||
Medication safety | 40% |
Patient safety is evaluated from three aspects, medical safety, nursery safety and medication safety respectively.
Evaluation of medical environment and doctor-patient relationship are evaluated according to the average score of different aspects.
In this section, the hospital PPP project is evaluated from the perspective of Public satisfaction rating, indicators and weights to assess were described in
Public satisfaction can measure the extent to which medical development of medical PPP programs provides social infrastructure and transportation services to meet user expectations. The evaluation results of public satisfaction can provide important decision support for the improvement of medical management level of PPP project operators and improvement of medical service level.
Evaluation Content | Primary indicator | Weights | Secondary indicator | Weights |
---|---|---|---|---|
Public satisfaction rating | Medical environment satisfaction | 20% | Hospital building facade and interior decoration | 15% |
Convenient traffic | 10% | |||
Convenient traffic flow in the hospital | 25% | |||
The degree of perfection of the medical environment | 20% | |||
Indicate the degree of perfection of the logo | 10% | |||
Overall environmental cleanliness | 10% | |||
Hospital attitude satisfaction | 20% | Doctor nurse attitude satisfaction | 40% | |
Care worker support satisfaction | 60% | |||
Hospital charge level satisfaction | 20% | Treatment cost satisfaction | 40% | |
Medical cost satisfaction | 30% | |||
Transparent satisfaction of fees | 30% | |||
Doctor technical level satisfaction | 20% | / | / | |
Medical information confidentiality satisfaction | 5% | / | / | |
Hospital maintenance and repair satisfaction | 15% | Maintenance and repair efficiency | 30% | |
Maintenance and repair effect | 40% | |||
The impact of maintenance and repair on hospital operations | 30% | |||
Social supervision assessment | Description: Telephone hotline, news media exposure, morning inspection, night check, and social reflections can be. Once it appears, 5 points/time of the impact is verified, and 10 points/time is deducted for major problems and adverse consequences. |
Public participation is a necessary procedure for the evaluation of PPP projects. Public participation means that the legal interests of the public, groups, and units, in which the project is located are fully guaranteed in order to make the argumentation of the PPP construction project more scientific and reasonable.
The purpose of public participation is to consider and compensate the interests and opinions of the groups directly or indirectly affected by the project through effective consultation with the public. Accepting social supervision and encouraging the public to participate in the supervision of the project is to effectively improve the implementation level and efficiency of the project. It is also an effective way to reduce the problems that may arise from the project construction and improve the social and environmental benefits of the construction project.
The government shall set up special funds for investigations for the public satisfaction rate survey, and the PPP project operators can actively cooperate and conduct public satisfaction surveys in accordance with the requirements of the Ministry of Health and the Ministry of Environmental Protection and survey specifications.
The evaluation method uses a third-party regulatory agency to issue a public opinion questionnaire or a household visit to the affected groups of the PPP project, conduct statistical analysis on the opinions of the respondents, and perform verification scores based on the questionnaire survey.
The third-party regulatory agencies of the PPP project are required to collect public opinions and, after verification, provide feedback to the government and PPP project operators.
The PPP project operation unit announces the performance evaluation status of PPP medical projects to the public every year; and can actively cooperate with the establishment of local PPP project supervision, complaints, and reporting hotlines, and there are special people to answer; the complaint handling rate is 100%.
Based on the above research and results, the final performance evaluation of hospital PPP project can be summarized in
Using the data of a certain PPP hospital provided by an interviewee charged of operation, it is estimated that the total score for the hospital PPP project which has been put into operation for 2 years in the UK equals 86.78 (100 is full mark).
Background information provided by the interviewee was as follows:
The project was constructed to improve the network of urban clinics and provide publicly funded health care services with hospitals. The PPP hospital project includes the construction of a new 489-bed national referral hospital, a portal clinic near the hospital, and refurbishment and re-equipment clinics. The second purpose of PPP is to engage the private sector in new ways to ensure that these facilities operate effectively and to provide more effective, higher quality care and a wider range of services. The English PPP hospital project is managed by government employees whose operations are governed by laws, regulations and traditions that govern public sector procurement, human resource management, finance and administration. Partnership mechanisms, including a comprehensive
The output of hospital PPP project | Indicators of Evaluation | Weights |
---|---|---|
Performance evaluation of hospital PPP project | Economic Evaluation | 15% |
Operation & Maintenance Evaluation | 35% | |
Medical Technology Evaluation | 20% | |
Evaluation of Hospital Management Level | 20% | |
Public Satisfaction Rating | 10% |
network of contracting construction and operations, and the use of output-based payment mechanisms, aim to increase accountability for service delivery and quality of care by reducing government accountability for day-to-day operations.
Through tendering, the alliance of social capital companies signed contract, agreed to build, operate, manage and provide clinical and non-clinical services through the integrated network over 20 years. The project has a total value of about $120 million. Capital expenditures are co-financed through public (42%) and private funds 58%).
Based on the above information, a whole new questionnaire was designed to get an overall rating of the PPP hospital project. Firstly, when the investigating the stakeholders of this PPP project, scores on each dimension listed in the above tables were collected, and then we use SPSS to confirm the validity and consistency of the data and remove the deviated number from database. 48 sets of data were proved valid. Then, after calculating the average number of each category, the weight provided by previous study was used to be multiplied by the average score of each corresponding indicator. Finally, the overall rating of the hospital PPP project was calculated (87.5), which means there’s still much room for improvement.
The study elucidates into the output specifications of hospital PPP projects and works out a way to quantify the indicators of performance evaluation system. At last, a case study was conducted to further confirm the validity and feasibility of the evaluation model. However, the weight of each indicator might not suit every hospital PPP project, and the indicators should be updated as project changes. What’s more, more scientific methods should be put forward to further eliminate subjective bias when determining weights of indicators, for example, fuzzy analytic hierarchy process (FAHP) approach [
The author declares no conflicts of interest regarding the publication of this paper.
Pei, Y. (2019) A Framework of Output Specifications and Evaluation Method for Hospital PPP Projects. Open Journal of Business and Management, 7, 167-179. https://doi.org/10.4236/ojbm.2019.71012