Most of the known methods for group ordering multi-attribute objects like group Analytic Hierarchy Process (Saaty, 1990), TOPSIS method (Hwang, Yoon, 1981), and others are focused on the well-defined goals where predominantly quantitative information is used. So, such methods are not applicable to qualitative features intrinsic for the ill-defined goals, for instance, estimation of efficiency organizations that carry out fundamental research. The principal obstacles of group ranking and sorting multi-attributed objects are associated with processing a large amount both verbal and numerical data without any additional transformations like averaging, mixing and/or weighing, which may result in ungrounded or irreversible distortions of the original data. Furthermore, it is also very difficult to ground the destination of the criteria weights, especially when several experts are engaged. In the quantitative methods, a convolution of a wide range of the criteria using weighing factors does not allow to recover the initial data on the aggregated criteria, and, thus, no interpretation of the obtained results is actually possible.

To evaluate effectiveness of scientific organization we elaborated a system of qualitative criteria with verbal scales, gradations of which specify various aspects of research and financial-economic activity of the scientific organization. Using these criteria, several experts evaluated activity of the considered scientific organizations. To operate with verbal data and determine the complex index of organization effectiveness we applied the original methodology of verbal decision analysis, which has been developed at the Institute for Systems Analysis of Russian Academy of Science for several years and successfully applied to solve various practical problems.

Verbal decision analysis (Larichev, Olson, 2001), (Larichev, 2006) deals with ill-structured problems described on a professional language. In a verbal decision analysis, fairly complex object properties and decision rules can be described even with a small amount of rating gradations on criteria scales. Numerical coefficients for the importance of criteria and the value of versions are not calculated or applied/ Verbal data are not transformed into a numerical form. Thus, when using only quality measurements, through a set of tuples of multicriteria assessments it is possible to define superiority and equivalence relations of the decision alternatives for their ordinal classification, partial ordering or the best option selection. A decision maker participates actively in specification, analysis and solution of the problem considered. Personal decision maker's judgements are verified and corrected in the course of the problem solution. So, the obtained results can be explained to a decision maker.

Group verbal decision analysis enlarges verbal methodology to group decisions [Petrovsky, 2008]. In group choice problems, preferences of several decision makers can be discordant, whereas objects are described with repeating quantitative and/or qualitative attributes and can be presented as multisets. Methods of group verbal decision analysis do not exclude discordant information and provide a reasonable decision. In general, verbal methods are more “transparent”, slightly subject to measurement errors and less time-consuming for human beings. From the methodological point of view, it is just the verbal decision analysis that is the most suitable for the efficiency assessment of scientific organizations performing basic research.

We consider the expert evaluation of scientific organization efficiency as group sorting multi-attribute objects on a complex criterion. For this purpose, we use the multi-stage technique PAKS (the abbreviation of Russian words: Consistent Aggregation of Classified Situations) that has been modified in accordance with specifics of the criteria of organization activity. At the first stage, the complex criterion is constructed with a reduction of the attribute space dimension that is based on decision maker's preferences. A construction of the criterion scale is considered as the classification problem, where the classified alternatives are combinations of object attributes, and the decision classes are verbal grades of the complex criterion (Petrovsky, Royzenzon, 2008). At the second stage, the criterion grades are composed step by step by using various verbal decision tools such as the tuples stratification technique, ZAPROS and ORCLASS methods. Thus, each object is assigned into some classes correspondent to the grades of complex criterion, which are obtained with different methods (Petrovsky, et al, 20090. At the third stage, all objects, which have been classified by several experts with different techniques, are sorted by ARAMIS method for group ordering multi-attribute objects presented as multisets (Petrovsky, 2010). In ARAMIS (Aggregation and Ranking Alternatives nearby the Multi-attribute Ideal Situations) method, the multi-attribute objects are arranged with respect to closeness to the hypothetically best object A⁺ or the worst object A ^{- (}i. e. the ideal or anti-ideal situations) in the multiset metric space. All objects are ordered by the indices of relative proximity to the best object A⁺.

The proposed methodology was applied to elaborate decision rules for analysis and evaluation of scientific organizations engaged in basic research, which had been estimated by several experts upon many qualitative criteria.

3. Multiple criteria expertise of scientific organizations

For expert assessment of the efficiency of scientific organizations performing basic research we elaborated the system of 11 criteria, which were combined in two groups related to research activity and financial-economic performance. Each criterion has an ordinal or nominal rating scale with the developed verbal formulations of quality gradations.

The group “Research activity of the organization” includes the criteria: Q₁₁. “Rate of research results in comparison with the level of world's achievements”; Q₁₂. “Received awards and prizes for research results”; Q₁₃. “Qualification of academic staff”; Q₁₄. “Average age of researchers”; Q₁₅. “Average age of scientific equipment”. For instance, the scale of the criterion Q₁₄. “Average age of researchers" is as follows: q₁₄¹ - the average age of researchers is less than 35 years; q₁₄² - the average age of researchers is 35-45 years.; q₁₄³ - the average age of researchers is over 45 years.

The group “Financial and business performance of the organization” includes the criteria: Q₂₁. “Reliability of the organization balance sheet relating to the property complex and the compliance with its constitutional and founding documents”; Q₂₂. “Tendency to change of the general balance indices ”; Q₂₃. ”Solvency and financial stability ratios”; Q₂₄. “Efficiency indices for current capital (business activity), profitability and financial performance (profitability) ”; Q₂₅. ”Performance indicators for non-current capital and investment activity”; Q₂₆. ”Quality of the organization development (business) plan”. For example, the criterion Q₂₃. ”Solvency and financial stability ratios” has the following scale: q₂₃¹ - the company is solvent and financially stable; q₂₃² - the company is insolvent and financially unstable. The scale of the criterion Q₂₆. ”Quality of the organization development (business) plan” is as follows: q₂₆¹ - the plan includes all of the verified documents substantiating any progress in financial conditions of the organization; q₂₆² - the plan is not confirmed by the documents justifying the improvement in financial conditions of the organization; q₂₆³ - the plan has not been elaborated.

Thus, using a multiset mathematical tools, each object (scientific organization) A_i, i=1,…,n is represented as the following set of repeating attributes:

A_i={k_Ai (q₁¹) ?q₁¹,…,k_Ai (q₁^h1) ?q₁^h1; …; k_Ai (q_m¹) ?q_m¹,…,k_Ai (q_m^hm) ?q_m^hm}.

Here k_A: XZ₊={0,1,2,…} is a multiplicity function of multiset, a set X=Q₁. Q_m consisted of m attribute (criteria) scales Q_s={q_s^es}, s=1,…,m. The value k_Ai (q_s^es) is a number of the attribute q_s^es, which is equal to a number of experts evaluated the object A_i with the criterion estimate q_s^esQ_s. The sign ? denotes that there are k_Ai (q_s^es) copies of attribute q_s^es within the description of object A_i.

Applying the developed methodology, experts of the Expert-Analytical Center carried out a monitoring of some federal state organizations engaged in basic research. After processing the questionnaire data, estimates of organizations given by 3 experts on 11 criteria were represented as the following multisets:

A₁ = {2,1,0; 3,0,0; 3,0,0; 1,1,1; 1,2,0; 2,1,0; 3,0,0; 2,1; 3,0; 3,0; 3,0,0};

A₂ = {3,0,0; 2,1,0; 2,1,0; 1,2,0; 2,1,0; 3,0,0; 2,1,0; 3,0; 2,1; 2,1; 3,0,0};

A₃ = {2,1,0; 2,1,0; 2,1,0; 2,1,0; 3,0,0; 2,1,0; 2,0,1; 3,0; 3,0; 3,0; 2,1,0};

A₄ = {2,1,0; 1,2,0; 2,1,0; 1,2,0; 2,1,0; 2,1,0; 2,1,0; 1,2; 3,0; 3,0; 2,1,0};

A₅ = {2,1,0; 2,1,0; 3,0,0; 2,1,0; 2,0,1; 2,0,1; 2,1,0; 3,0; 2,1; 3,0; 3,0,0};

A₆ = {2,1,0; 1,2,0; 2,1,0; 0,3,0; 1,2,0; 3,0,0; 2,1,0; 3,0; 3,0; 2,1; 2,1,0};

A₇ = {2,1,0; 2,1,0; 2,1,0; 2,1,0; 2,1,0; 1,1,1; 2,1,0; 1,2; 2,1; 1,2; 2,0,1};

A₈ = {3,0,0; 3,0,0; 2,1,0; 1,1,1; 1,0,2; 3,0,0; 3,0,0; 1,2; 3,0; 3,0; 2,1,0};

A₉ = {2,1,0; 2,1,0; 1,1,1; 0,2,1; 0,2,1; 2,0,1; 1,1,1; 1,2; 2,1; 2,1; 1,1,1};

A₁₀= {1,2,0; 1,1,1; 2,0,1; 2,0,1; 2,1,0; 2,0,1; 2,0,1; 2,1; 2,1; 2,1; 2,0,1}.

The hypothetically best and the worst organizations are represented as the following multisets:

A⁺ = {3,0,0; 3,0,0; 3,0,0; 3,0,0; 3,0,0; 3,0,0; 3,0,0; 3,0; 3,0; 3,0; 3,0,0};

A ^- = {0,0,3; 0,0,3; 0,0,3; 0,0,3; 0,0,3; 0,0,3; 0,0,3; 0,3; 0,3; 0,3; 0,0,3}.

Every series of numbers above corresponds to the values of multiplicity function of multiset elements, which are the gradations of the criteria scales Q₁₁ - Q₁₅; Q₂₁ - Q₂₆.

In the assumption that all criteria Q_s have the equal relative importance, the calculated distances between single organization A_i and the ideal A⁺ and the anti-ideal A ^- ones in the multiset metric space, and the indices of relative proximity l (A_i) =d⁺ (A_i) / [d⁺ (A_i) +d^- (A_i)] of organization A_i to the best organization A⁺ are as follows:

A₁ A₂ A₃ A₄ A₅ A₆ A₇ A₈ A₉ A₁₀

d⁺ (A_i) 15 17 14 26 14 25 29 16 37 24;

d^- (A_i) 61 60 63 59 60 61 50 55 46 50;

l (A_i) 0, 1970,2210,1820,3060,1890,2910,3670,2250,4460,324.

The final ordering the organizations in accordance with their relative proximity is represented as follows:

A₃A₅A₁A₂A₈A₆A₄A₁₀A₇A₉.

It should be noted that application of various methods for constructing the aggregated complex index of organization efficiency and ranking organizations with various metrics in ARAMIS method can result in some different final orderings. The results obtained in the approbation on a model data base confirmed the effectiveness of the suggested methodology. Thus, we identified organizations with different efficiency rates, which provides better decisions on the point of giving credits or some other supports to the organizations.

4. Conclusion