Abstract

The article analyses several key issues which restrict the effectiveness of fund project peer review work. It analyses the evaluating theory and matter-element theory to access the expert anti-evaluation model, and also studies the expert anti-evaluation index system to support the anti-evaluation method. The practical basis is the true score data of the experts which is collected from the actual anti-evaluation in Liaoning province science and technology fund project peer review system. With the practical experience of the actual project, we prove that the expert index system anti-evaluation model and expert anti-evaluation method can improve the fund project peer review work and play a positive role for the peer review work and also make the review work more scientific and more rational.

1. Introduction

Science fund is the main method to support the basic research in China, which has a characteristic of “Government funding, macro guide, free application, democracy review” [1]. It has a promoting effect in the funded projects, person's training, stability of basic research team and raising the research level. Therefore the impartiality of the fund project review is very important, which most applicants pay close attention to. Nowadays, Peer communication review is commonly used in the review work, which means experts of the same field review the project back to back.

Peer review can take full advantage of scientists who are in the scientific community to realize the reasonable allocation of scientific and technological resources. It also contributes to the democratization of the scientific decision-making, and putting an end to the purely administrative decision-making. But at present, in the scientific and technological project evaluation works of China, there are still many problems in peer review system. Such as, the difference of expert judgments and evaluating standard; the difference of academic field and academic level; the contradictory between different schools of thought; different self-cultivation between scientists and so on. All these factors will bring unfair factors to the project evaluation. It requires us to evaluate the effectiveness of fund project peer review work. This accreditation system for the evaluation is called antievaluation. Anti-evaluation aims at evaluating the anti-come problems, improving the fund project evaluation system of peer review, so as to make it work better, more scientific and more rational.

There are two most important factors in the fund project anti-evaluation system of peer review, expert index system and effectiveness evaluation method. Therefore, the two factors will be a study of an anti-evaluation model in the peer review system. The article analyses the evaluating theory and matter-element theory to assess anti-evaluation model, and then analyses the mathematical meaning and physical meaning of each parameter which can make us know the advantages and disadvantages. The article also studies the expert index system in order to support the anti-evaluation method. The practical basis is the true score data of the experts which is collected from the actual anti-evaluation in Liaoning province science and technology fund project peer review system.

With the practical experience of the actual project, we prove that the expert index system anti-evaluation model and expert anti-evaluation method can improve the peer review system and play a positive role for the work of the peer review and also make the review work more scientific and more rational.

2. Matter-Element Analysis

In the 1980s, the Chinese mathematician called CaiWen made the matter-element analysis theory. Matter-element theory is usually used to solve the complex incompatible problems. The comprehensive evaluation matter-element model defines the grade areas for individual index firstly, and then gets the single index state by calculating the correlation function of a single index, and then gets the comprehensive level of multi-index by integrating models, which can make the results more objective and detailed. The model can quantitatively show the evaluation result and reflect the comprehensive level completely, which is easy for the computer programming [2−4].

2.1. Basic Model

According to the matter-element analysis theory [5−7], giving things the name N and its characteristics c the value v, we describe the ordered triple R= (N, c, v) as the basic element of things, which we call matter-element for short. If things N has multiple characteristics, namely c₁, c₂, …, c_n , and their corresponding values v₁, v₂, …, v_n, we can define the matter-element as (1). In (1), R is n-dimensional matter-element, abbreviated as

(1)

2.2. Classical Domain and Section Domain

Classical domain matter-element [8−11] is defined as

(2)

where N_j(j=1,2,…,m) is the j-grade status of the review; c_i(i=1,2,…,n) is the corresponding characteristic for a grade status of the review; interval X_ji=ji, b_ji> is the range of the value c_i for N_j, namely, the data range of the corresponding index of the each grade of the review status—classical domain.

Section domain matter-element is defined as

(3)

where N_p is the whole evaluation grade; c_i(i=1,2,…,n) is a characteristic of the whole grade; interval X_pi=pi, b_pi> is the range of the value c_i for N_p, namely, the data range of the index of the whole review grade----section domain. Obviously, there is X_jX_p.

2.3. Correlation Function

Correlation function describes the degree of the required level of the matter-element, giving the matter-element value x mapping to the real axis in the matter-element analysis evaluation. The distance between node x_i and limited reality interval X= is defined as

(4)

where the distance between node x_i and interval X can be negative value, the difference of the negative value determines correspondingly the difference of the position x_i in the interval X.

We can get the distance between the node x_i and the interval X_ji of classical domain, as well as the distance between the node xi and the interval X_pi of section domain as

(5)

(6)

Then the computation formula of the correlation function K(x) is defined as

(7)

Correlation function describes the degree of the required level of the matter-element, giving matter-element the value x₀ mapping to the real axis in the matter-element analysis evaluation. Actually, it describes the degree of ownership of the each index of the project to be reviewed about each review grade j, which makes mathematics “belong” and “not belong” qualitative description the extended to quantitative description.

3. Anti-Evaluation Model

3.1. Expert Index System

Since the research purpose is different, the evaluation index system is different in different research fields. This article mainly studies the expert anti-evaluation in the fund project peer-review work. According to the expert information relevant to the fund project to be reviewed, we sort the expert anti-evaluation index into two comprehensive indexes which contain six single indexes. The comprehensive indexes describe the experts’ comprehensive academic level and their review ability; meanwhile the single indexes describe the specific sub-set of the comprehensive index, which are used to realize the comprehensive index evaluation. The expert anti-evaluation index system is defined as Table 1.

The comprehensive index of experts includes academic level index(b₁) and review ability index(b₂) , on behalf of the expert’s academic research level and ability of project review. The academic index includes foundation level index(c₁), literature level index(c₂) and research level index(c₃). Among them, the foundation level index refers to the academic research level embodied in the expert’s nature information, such as expert’s unit, age, professional title etc. The literature level index refers to the literature contribution ability about the published academic papers, published monographs, cited cases etc. The research level index refers to the expert’s development and application level embodied in the state of his own fund projects, application of the projects, rewards and patents etc. The review ability index includes field familiarity index(c₄), review participate index(c₅) and review accuracy index(c₆). Among them, field familiarity index refers to expert's familiar situation of the review project, reflecting the authority of the expert in this field. Review participation index refers to the number of project review that expert has involved in, reflecting project familiarity and experts experience rich or not from the review work. Review accuracy index refers to the accuracy of the funded project that expert suggests, reflecting the expert's confidence levels in project review.

According to the indexes above, we can master every aspect of experts' quality and ability, which is helpful for the reasonable selection of experts and ensuring fairness and justice of the project review.

3.2. Expert Matter-Element Anti-Evaluation Model

According to the expert anti-evaluation index system, we can establish the expert matter-element anti-evaluation model.

If there are n single indexes in the comprehensive index b_k, namely, ci (i=1,2,...,n), and the corresponding single index matter-element to be evaluated is R_c, we can define the expert single index matter-element as

(8)

where N_o is the expert to be anti-evaluated; C is the whole characteristics of the comprehensive index b_k(k=1,2,...,m) of No, namely, all the single indexes c_i(i=1,2,…,n) of the comprehensive index b_k of N_o; v_ci is the value of the single index c_i.

If there are m comprehensive indexes for experts in the fun project peer-review, namely, b_k (k=1,2,...,m), the comprehensive index matter-element Rb to be antievaluated is defined as

(9)

where N_o is the expert to be anti-evaluated; B is the comprehensive indexes of N_o; v_bk is the value of the comprehensive index b_k (k=1,2,…,m).

If the comprehensive index b_j is composed of n single indexes, namely, c_i (i=1,2,…,n), the algorithm to get the value of comprehensive index b_k, namely, v_bk is defined as

(10)

where v_ci is the value of the single index c_i.

3.3. Characteristic Index Weight

Because of the different influence degree of each characteristic index towards matter-element to be anti-evaluated, we need to give different weights according to their relative importance.

Determining the weights of each characteristic index is based on the principles and policies formulated by the fund management in the fund project review system. For the anti-evaluation key of the experts is the review ability, we pay more attention to the expert field-familiarity of the projects, review accuracy and peer-review participation ability, all of which take larger weight of the corresponding index.

3.4. Correlation Calculation of Comprehensive Index Matter-Element

According to the Formula (2), (3), define the classical domain and section domain of the comprehensive index matter-element. Then according to the Formula (4), (5), (6), (7), calculate the correlation of each comprehensive index for j grade of the review status, namely, K_j(b_k).

After introducing the weight of comprehensive index, we can define the correlation of N_o for j grade of the evaluation status as

(11)

where n is the number of the comprehensive indexes, b_k(k=1,2,...,m) is the kth comprehensive index, is the weight of the comprehensive index b_k.

If max(K_j(N_o))=K_jo, we can conclude that the matter-element to be reviewed belongs to j_o grade.

3.5. Expert Comprehensive Evaluation Grade

If K_jo(N_o)=max(K_j(N_o)), j_o∈(1,2,…,m), N_o belongs to the grade j_o. The value of the correlation in the real axis is corresponding to the membership degree of the standard grade of the matter-element to be reviewed. The bigger its value is, the higher its membership degree is.

If K_jo(N_o)<-1, it means that the matter-element to be evaluated does not meet the requirements of the standard grades, and does not have the conditions to be converted into the standard grades. The smaller its value is, the farther the standard level is away from.

If -1jo(N_o)<0, it means that the matter-element to be evaluated does not meet the requirements of the standard grades, but has the conditions to be converted into the standard grades. The bigger its value is, the more easily it is to be converted into the standard grades.

If 0jo(N_o)<1, it means that the matter-element to be evaluated meets the requirements of the standard grades.

If K_jo(N_o)>1, it means that the matter-element to be evaluated exceeds the upper limit of the standard grade. The bigger its value is, the greater the potential for development is.

4. Anti-Evaluation Instance

4.1. Classical Domain and Section Domain

Each single index is divided into 5 standard grades which are excellent, good, medium, general and poor grade. We can define the classical domain matter-element R_A、R_B、R_C、R_D、R_E and section domain R_p according to (2) (3) as

,

,

,

4.2. Expert Matter-Element to Be Anti-Evaluated

Take the evaluation result of one expert for example, shown as Tables 2, 3, and 4.

According to the scoring conditions of the experts, we can define the corresponding comprehensive index matter-element R_bA, R_bB, R_bC for the experts by (8), (9), (10) as

, ,.

4.3. Correlation Calculation of Comprehensive Index Matter-Element

According to (7), calculate the correlation of each comprehensive index of the expert in each evaluation grade, the results are shown as Table 5 below.

Determine the weight of the comprehensive index {b₁,b₂} of the expert, ={0.4,0.6}, calculate the comprehensive correlation of the expert, the results are shown as Table 6 below.

By the actual data of the comprehensive correlation, we can see that expert A belongs to good grade, expert B belongs to medium grade, expert C belongs to excellent grade. If we want to choose two experts, we will select expert A and C.

5. Conclusions

In the paper, one main contribution is the establishment of the expert anti-evaluation index system in the fund project peer review system. By assessing the main evaluation index (comprehensive index), namely, the expert academic research level and the expert project evaluation experience and ability, and by assessing the reference index (single index), namely, the expert foundation level, literature level, research level, field familiarity, review participate and review accuracy, we can describe the expert characteristics of each aspect in the peer review work.

In addition, this paper presents an expert anti-evaluation method based on the theory of matter-element model. By assessing the value of the comprehensive index and the single index, we use the method to invalidate the effectiveness of each expert in the peer review work.

With the practical experience of the actual project, we prove that the implementation of the expert anti-evaluation index system and expert anti-evaluation method can improve the peer review system and play a positive role for the work of the peer review. The expert anti-evaluation model makes the review work more scientific and more rational.

6. References

[1] X. Xuan and Y. Jiang, “The thinking of improving local science fund operation mode,” Science Fund, Vol. 11, pp. 66–68, 1997.

[2] W. Cai, “Extension theory and its application,” Chinese Science Bulletin, Vol. 44, No. 17, pp. 1538–1548, 1999.

[3] W. Qiu, “Study on properties of a kind of matter-element extension sets,” Mathematics in Practice and Theory, Vol. 36, No. 1, pp. 227–231, 2006.

[4] B. He and W. Cai, “Matter-element propositions and affair-element propositions,” Journal of Guangdong University of Technology, Vol. 18, No.1, pp. 88–93, 2001.

[5] H. Hao, Y. Feng, and J. Tan, “Customer dynamic evaluation based on extended matter-element model,” Journal of Zhejiang University (Engineering Science), Vol. 43, No. 1, pp. 57–60, 2009.

[6] B. He and R. Wang, “Matter-element deductive inference”, The System Engineering Theory and Practice, Vol. 1, No. 1, pp. 85–92, 1998.

[7] C. Han, L. Liu, and Z. Wang, “Evaluate infrastructure system sustainability based on matter-element analysis method,” China Population Resources and Environment, Vol. 19, No. 2, pp. 117–119, 2009.

[8] X. Zhou, “Research on method of vague matter-element decision making based on entropy weight,” Journal of Systems & Management, Vol. 18, No. 4, pp. 454–457, 2009.

[9] T. Zhan, X. Wang, and R. Liu, “Research on matter element model of investment decision for highway,” Journal of Wuhan University of Technology (Transportation Science & Engineering), Vol. 33, No. 4, pp. 722–725, 2009.

[10] J. Wang and Z. Fan, “A model framework for enterprise knowledge management system based on web technology,” Journal of Northeastern University (Natural Science), Vol. 24, No. 2, pp. 182–185, 2003.

[11] J. Lu and Z. Jia, “The evaluation on competitiveness of property management enterprise based on matter-element model,” Technology Economics, Vol. 28, No. 1, pp. 113–116, 2009.

NOTES

Identify applicable sponsor/s here. (sponsors)