Hierarchical aggregation approach to building the integrated criterion of R&D project efficiency

Размещено на http://www.allbest.ru/

Abstract

HIERARCHICAL AGGREGATION APPROACH TO BUILDING THE INTEGRATED CRITERION OF R&D PROJECT EFFICIENCY This work is partially supported by the Russian Academy of Sciences, Research Programs "Intellectual Information Technologies, Mathematical Modeling, Systems Analysis, and Automation", "Information Technologies and Methods for Complex Systems Analysis", the Russian Foundation for Basic Research (projects 08-01-00247, 08-07-13532, 09-07-00009).

Alexey B. Petrovsky¹, Gregory V. Royzenson¹, Igor' P. Tikhonov²

¹ Institute for Systems Analysis, Russian Academy of Sciences, Prospect 60 Let Octyabrya, 9, Moscow 117312, Russia,

² Russian Foundation for Basic Research, Moscow 119991, Russia, pab@isa.ru, rgv@isa.ru/

The paper describes a new approach to the step by step aggregation of attributes and the construction of integrated criterion. The approach is based on the interactive method for multiple criteria classification consistent with reducing the attribute space dimension, which has been applied for the evaluation of R&D project efficiency.

Keywords: verbal decision analysis, ordinal classification, multi-attribute alternatives, attribute aggregation, reduction of attribute space dimension, integrated criterion, project efficiency.

Introduction

There are a lot of various practical problems connected with building the integrated criterion of efficiency, which combines manifold criteria and attributes. We consider here the construction of such integrated criterion as an ordinal classification problem with hierarchical aggregation of attributes based on the decision maker (DM) preferences. Immediate classification of alternatives, especially described with qualitative (verbal) attributes, is an onerous procedure, which requires considerable time of DM. Often, when a number of objects is rather small, and a number of attributes is enough huge, the analyzed objects may be formally incomparable by their properties. These reasons significantly complicate the application of popular decision making methods (Roy and Bouyssou, 1993; Larichev and Olson, 2001; Doumpos and Zopounidis, 2002). Thus, we need to develop special techniques of processing data for the multiple criteria decision problems in the attribute space of large dimension (Petrovsky and Royzenson, 2008).

This paper presents a new interactive method for step by step reducing the attribute space dimension and its application to evaluate the efficiency of R&D projects, which are subsidized by the Russian Foundation for Basic Research (RFBR). The space reduction is examined as the classification problem, where multi-attribute objects are combinations of project estimates given by RFBR experts upon many verbal criteria. Grades of the integrated efficiency criterion represent the decision classes. The hierarchical aggregation of multiple criteria estimates allows us to reduce the multi-attribute space dimension and diminish considerably difficulties of the problem solution. The suggested approach to building the integrated criterion facilitates an analysis and explanation of R&D project efficiency.

1. Construction of complex criterion scale

The problem of reducing the multi-attribute space dimension has the following statement:

X₁ґ…ґX_m ® Y₁ґ…ґY_n, n<m,

where X₁,…,X_m are the sets of initial attributes, Y₁,…,Y_n are the sets of new attributes, m and n are dimensions of initial and new attribute spaces. Grades of attribute scales:

X_i={x_i¹,…,x_i^gi}, i=1,…,m, and

Y_j={y_j¹,…,y_j^hj}, j=1,…,n

are ordered.

Consider the above problem as a problem of ordinal classification of multicriteria alternatives. It means that different combinations of initial attributes (or corteges of criteria estimates in the space X₁ґ…ґX_m) are alternatives, which are aggregated into a smaller sets of classes (categories) Y₁,…,Y_n with ordered scales. The verbal grades of every new attribute Y_j have a concrete content for DM. This integrated attribute is called as a complex (composite) criterion. By step by step combining attributes into a small number of complex criteria, DM may construct a hierarchical system of criteria up to a single top criterion, which grades are a solution of the applied problem.

The suggested procedure of building the complex criterion scale consists of several unified blocks of classification. These blocks are selected by DM depending on the problem specifics and executed consistently step by step. Each classification block of i-th hierarchical level includes any attribute set and a single complex criterion. Corteges of attribute estimates represent classified objects. Decision classes of i-th level provide scale grades of the complex criterion.

In a classification block of the next hierarchical level, complex criteria of the i-th level are considered to be new attributes. Corteges of their scale grades represent new classified objects in the reduced attribute space, whereas decision classes of the (i+1)-th level will be now scale grades of new complex criterion. A scale of single complex criterion of the top hierarchical level provides the required ordered classes D₁,…,D_q. The found boundaries of classes allow us to sort easily the real alternatives (variants) V₁,…,V_p estimated upon many criteria K₁,…,K_m.

To build scales of complex criteria, DM may apply different procedures. The simplest way for composing the complex criterion scale is the tuples stratification technique that uses cutting the multi-attribute space with parallel hyper-planes. Each layer (stratum) consists of combinations of the unified initial estimates with fixed sum of numbers and represents any generalized grade on the scale of complex criterion. DM determines a number of layers (or scale grades). DM may combine the initial estimates into the generalized grades, for example, as follows: the best initial estimates form the best generalized grade, average initial estimates form the average generalized grade, and the worst initial estimates form the worst generalized grade. The maximal number of layers is equal to

L=1m+_ig_i.

The number of classes is equal to qL.

A more complicated composition of the complex scale involves the ZAPROS and ORCLASS methods of Verbal Decision Analysis (Larichev, 2006). The ZAPROS method allows DM to construct a joint ordinal scale of complex criterion from separate initial estimates. The ORCLASS method builds a complete and consistent classification of all the corteges of initial estimates where classes form an ordinal scale of complex criterion In these cases, all the possible combinations of initial estimates in the attributes space are multi-attribute alternatives, which number is equal to:

p=_ig_i.

2. Interactive method of reducing the multi-attribute space dimension

The ISKRA method (the abbreviation of Russian words: Hierarchical Structuring CRiteria and Attributes) provides a solution of the multiple criteria problem with reducing the multi-attribute space dimension (Petrovsky and Royzenzon, 2008). This technique is based on the DM preferences and includes the following steps.

Step 1. Choosing the type of multiple criteria problem T that is T₁ - to select the best alternatives; T₂ - to order alternatives; T₃ - to divide alternatives into ordered groups (classes).

Step 2. Forming the set of variants V, |V|2.

Step 3. Forming the sets of basic indicators (initial attributes or criteria) K₁,…,K_m, m2. These characteristics can either be specified in advance or generated in the course of problem analysis with the help of analyst or expert depending on the problem specifics.

Step 4. Forming an ordinal scale:

X_i={x_i¹,…,x_i^gi}

of every basic indicator K_i, i=1,…,m. The set of tuples, which form Cartesian product X₁ґ…ґX_m of the initial attribute grades, are considered as the set of all the possible variants V.

Step 5. Forming the set of complex criteria or integrated attributes Y₁,…,Y_n (n<m), which define the property of variants selected by DM and aggregated basic characteristics. integrated criteria project efficiency

Step 6. Forming an ordinal scale:

Y_j={y_j¹,…,y_j^hj},

j=1,…,n of every complex criterion. Each grade of the complex criterion scale is a combination of basic indicators.

Step 7. Choosing the option W for building the complex criterion scale (combining attributes) as follows: W₁ - tuples stratification, W₂ - ranking tuples, W₃ - classification of tuples.

Step 8. If DM accept a solution of the problem T (alternatives are classified, ordered or the best alternatives are selected using constructed complex criteria), then the algorithm completes the job. Otherwise go to step 9.

Step 9. If the result obtained in step 8 is unsatisfactory, then it is possible to either change the option of building the complex criterion scale W (go to step 7), or change grades of the complex criterion scale Y_j (go to step 6), or form a new set of complex criteria (go to step 5).

The attributes are being aggregated consistently step by step. DM defines a number, structure, content of criteria, and rating scales for the every hierarchy level. DM establishes also, which basis indicators are to be considered as independent criteria and which ones are combined within the complex criteria. The obtained groups of criteria may be combined, in turn, into new groups (the following level of hierarchy) and so on. DM may build the scales of complex criteria on the different stages of aggregation procedure using various options mentioned in Section 2.

3. Multicriteria evaluation of research efficiency

The evaluation of research efficiency is one of practical problems, which require the development of integrated indicator. The Russian Foundation for Basic Research (RFBR) is the federal agency that possesses an extensive experience in organizing and conducting basic research, and examination of their practical application. In RFBR, there is the special expertise for multi-expert and multicriteria assessment of grant applications and completed projects.

For instance, each completed goal-oriented project, that is performed in the interest of federal agencies and departments of Russia, and the obtained results are evaluated by several experts upon eight criteria (basic indicators) such as:

K₁ "Degree of the problem solution",

K₂ "Scientific level of results",

K₃ "Appropriateness of patenting results",

K₄ "Prospective application of results",

K₅ "Result correspondence to the project goal",

K₆ "Achievement of the project goal",

K₇ "Difficulties of the project performance",

K₈ "Interaction with potential users of results".

Each criterion has an ordered scale of verbal estimates. For example, the scale of the criterion "Degree of the problem solution" looks as follows: x₁¹ - the problem is solved completely, x₁² - the problem is solved partially, x₁³ - the problem is not solved. The "Achievement of the project goal" is estimated as x₆¹ - really, x₆² - non-really.

To evaluate the feasibility of effective practical use of obtained results the notion of "Project efficiency" has been formalized. Construction of the integrated indicator of project efficiency was examined as the multiple criteria classification problem with reducing the multi-attribute space dimension. Classified objects were the combinations of multicriteria estimations of completed projects in the attribute space X₁ґ…ґX₈. The ordered classes were the rates of project efficiency, which correspond to grades on the scale of the top level complex criterion D "Project efficiency" as d¹ - superior, d² - high, d³ - average, d⁴ - low, d⁵ - unsatisfactory. DM has been able to form the integrated indicator of project efficiency in different ways and compare the constructed indicators.

The model database for verification of the algorithms and construction of the integrated indicators includes expert assessments of the goal-oriented projects, that have been completed in 2007 in the following areas: 01. Mathematics, mechanics and computer science (total 48 projects), 03. Chemistry (total 54 projects), 07. Information and telecommunication resources (total 21 projects).

The integrated indicators of project efficiency have been identified for each area of expertise and estimates of two experts using the following options for building the complex criterion scale:

(i) the ORCLASS method on all levels of the criteria hierarchy (OC);

(ii) the stratification of tuples on all levels of the criteria hierarchy (ST),

(iii) the stratification of tuples on the lower level of the criteria hierarchy, and the ORCLASS method on the upper level of the criteria hierarchy (ST+OC); (iv) the ORCLASS method on the lower level of the criteria hierarchy, and the stratification of tuples on the upper level of the criteria hierarchy (OC+ST). Grades of the integrated indicators of project efficiency (Fig. 1) coincide in 74 % and 48 % cases (the area 01); in 72 % and 24 % cases (the area 03); in 76 % and 62 % cases (the area 07). The first number is related to projects estimated by the first expert, the second number - by the second expert.

Fig. 1. An example of the integrated indicators of project efficiency

It was also considered two ways for aggregating the complex criterion of top level D "Project efficiency". There are a construction of the criteria hierarchy, which combines the initial criteria K₁-K₄ and K₁-K₄ into two intermediate sub-criteria (option A), and the initial criteria K₁-K₃, K₅-K₇, and K₄, K₈ into three intermediate sub-criteria (option B). So, in the area 03, 16 and 6 projects have the superior rate of efficiency, 75 and 40 projects - the high rate of efficiency, 13 and 59 projects - the average rate of efficiency, 2 and 1 projects - the low rate of efficiency, 2 and 2 projects - the unsatisfactory rate of efficiency. The first number is related to the option A, the second - to the option B. In general, the integrated indicators of project efficiency coincide in 41 cases out of 108. In other cases, the integrated indicators differ by no more than one rate.

These data testify the high stability of the suggested approach to building the complex criteria on all levels of the hierarchy and constructing the integrated indicators of project efficiency.

Conclusion

The new approach to constructing the integrated indicators of activity estimated by many criteria is suggested. The ISKRA method for reducing the multi-attribute space dimension may be applied together with other methods of decision making and information processing. Firstly, a hierarchical system of complex criteria is constructed in the reduced attribute space. Then, using the performed system of criteria, the considered problem is solved. This approach provides to operate with numerical, symbolic, and verbal data.

Using the ISKRA method in the practice DM has ability to determine the most preferable integrated indicators, select the method or combination of methods for building the complex criteria, and compare the obtained results for different sets of criteria. A number of addresses to DM required for constructing a complete collection of criteria may be served as an effectiveness value of the chosen way for the problem solution.

Sequential division of all the criteria into several groups provides an opportunity to solve the problem in parallel ways, investigate systematically the available information, analyze and explain the final decision. These allow users to simplify solving hard practical multiple criteria problems and reduce significantly the time that DM spends in order to achieve his goals.

References

1. Doumpos, M. and C. Zopounidis (2002); Multicriteria Decision Aid Classification Methods; Kluwer Academic Publishers, Dordrecht.

2. Larichev, O.I. and D.L. Olson (2001); Multiple Criteria Analysis in Strategic Siting Problems; Kluwer Academic Publishers, Boston.

3. Larichev, O.I. (2006); Verbal Decision Analysis; Nauka, Moscow (in Russian).

4. Petrovsky, A.B. and G.V. Royzenson (2008); Sorting multi-attribute objects with a reduction of space dimension; Advances in Decision Technology and Intelligent Information Systems - Vol.IX (ed. by K.J. Engemann, G.E. Lasker); The International Institute for Advanced Studies in Systems Research and Cybernetics, Tecumseh, Canada (pp.46-50).

5. Roy, B. and D. Bouyssou (1993); Aide Multicritиre а la Dйcision: Mйthodes et Cas; Economica, Paris.

Размещено на Allbest.ru