Russian mutual funds: skill versus luck

Pic. 7 Real daily alpha distribution

All three models are presented on the graph. CAPM is with dotted line and has the highest peak. Specification with Brent oil in rubles has the middle pick and is the one with dashed line. The solid line goes for model with Brent oil and USD/RUB with the lowest peak. The distributions are practically the same for all model specifications. The difference is in the middle peak and there is a slight spread among the lines all over the graph to compensate this peak. According to the statistics above most funds have negative alphas and the mean value for the distributions are also negative, but the peak is to the right of zero. It means that the mode is positive. The difference in mode and mean (and median) are due to a sharp slope in the left part of the graph. There are more high negative alphas than high positive. But it does not necessarily mean that there are more funds with bad skill. It is because many funds are industry oriented or have strict policy to invest in second-tier stocks or blue chips. And due to bad luck that industries or part of the market face problems and show negative dynamic. The mentioned funds will follow them and thus have negative alphas. At this point it is hard to conclude which model is underestimating or overestimating alphas. And since the differences in results are minor the final results for bootstrap may not differ much. As an efficiency criteria can be used r-squared to compare the models. Also, it is important to test the residuals of regressions to move on. Since real market data usually do not have normal distribution of returns and errors of regression, the result will probably be the same for this research. The so goes for autocorrelation of residuals. It also is one of the points to use bootstrap approach. As it was mentioned in core papers bootstrap is a good method to deal with data that has autocorrelation and not normal distribution of residuals.

Pic. 8 R-squared of three models

The lines are the same as on Picture 7: CAPM - dotted, Brent oil in rubles - dashed, Brent oil and USD/RUB - solid line. From Picture 8 it can be clearly seen that model specification with separate factors Brent oil and USD/RUB is better than the others in terms of r-squared. But the results for all models are poor: for most funds r-squared is close to zero. That is because there is quite a number of funds that invest in foreign assets: shares, ETFs, commodities and etc. These funds have low correlation with Russian market and thus low beta. Without any appropriate factors for foreign assets the only part of regressions that has influence for such funds is constant value or alpha. In this case alpha will be close to mean value (of dependable variable) and fitting errors and their standard deviation will be high. Thus r-squared is low. Its mean for each model is about 0,2. Most likely plenty of funds invest in assets that have almost none correlation with the benchmark and other factors in the model. Also, if the models are based on another benchmark - Micex, then r-squared is almost the same for that period. The statistics for three models with MSCI Russia benchmark and one model with MICEX are presented in Table 5.

Table 5

Statistics for R-squared

Judging from r-squared the models are close enough to each other. The results for different benchmarks are practically even. Higher median for model based on MICEX means that among low r-squared funds have higher r-squared than for model with MSCI Russia, while lower mean suggests that there are more funds with high r-squared for MSCI Russia than for MICEX. Since median is lower and mean is higher (with higher max values) than standard deviation should be higher too. Model with MICEX seems to have higher r-squared in the second quarter and it's the only part where this model prevail. All these can be seen from the table above. Since the minimum, first quarter and mean values are higher for MSCI Russia model it is slightly better to use this benchmark instead of MICEX. Among the models the best one seems to be specification with separate factors of Brent oil and USD/RUB. It is probably because the currency pair has its own unique influence on the market and funds, while crude oil do not suffer dramatic changes from switching the currency.

Before moving on to bootstrap simulations the errors of the regressions should be tested for autocorrelation and normality. Along with errors funds` returns are also tested. As normality test is used Jarque-Bera test. Most funds have more than 1500 observations and so do residuals. The null hypothesis for this test is that the data has normal distribution, which means skewness and excess kurtosis are zero. The p-value for all funds` returns and regression residuals are close to zero. Only one fund (VTB fund of power industry) has p-value around 0,01 for regression with Brent oil and USD/RUB, but since the significance level is 5% the null hypothesis is rejected for all funds` returns and residuals. So, no fund has normal distribution of residuals (and returns too). To test sample for autocorrelation in residuals was chosen Durbin-Watson test. It only tests first order autocorrelation and the null hypothesis is that the residuals are serially uncorrelated. The alternative claims that there is first order autocorrelation. Null hypothesis was rejected for from 134 to 139 funds depending on model specification. But the dw statistics was always between 1 and 3, meaning the spotted serial autocorrelation is not dramatic. As the core papers suggest bootstrap approach can deal with autocorrelation in residuals since the residuals and changed in random order to create portfolios. The results for residuals tests are presented in Appendix 4.

The first method to measure the skill of a funds is suggested by M. Jensen as it was mentioned in theoretical background: the higher alpha the fund has the higher level of skill the manager has. According to this theory the funds that have top alphas can be described as skillfully managed. They should have picking skills. It should be better to track the dynamic of alphas and if a fund has alpha among top for many periods then the fund is truly managed skillfully. But since the overall period of time should be long enough there are no such funds. The funds in the sample have been functioning for different period of time and thus have different number of observations. It makes them uneven. A certain fund can maintain high alpha for a short period of time but with time alpha will drop. That is why alpha as it is may be not the most accurate measure of managers` skills. But it still can be useful to look through it. Even though it may not be the appropriate method, but it is assumed that skilled funds are in the tales of alpha distribution. At this point the market is assumed to have 2,5% funds with positive skill and the same amount of negative skill funds. The proper way should be to choose the interval of alpha`s values, one for positive skill and one for negative. But this intervals may be assumed to be equal those from previous method. Then for significance level of 5% the best and worst 2,5% of funds have positive and negative skills respectively. Thus for 175 funds there are 4 funds with positive skill and 4 funds with negative skills. Table 6 shows the top alphas according to three models. Table 7 - the worst alphas.

Table 6

Top 4 alphas from each model.

Table 7

Lowest 4 alphas from each model.

The results for different models are similar. The first and last three are the same and the 4^th and 5^th are swapped in one model. What is more the funds with top alphas have almost the same places in terms of mean returns. The top five alphas have the top five mean returns. Since all funds from with lowest alphas are oriented on investments in power industry, the last has probably have harsh times during the analyzing period. As it was mentioned above the results are sensitive to the number of observations or the period of time for which alpha is estimated. And that can be seen through the tables above. The funds that have top alphas have low number of observations (3-4 times lower than the whole sample). Thus the results are inappropriate. If, for example, the total number of observations in the models are shorten to 500 observations, which means 2 years, there could be different results. The statistics of this experiment are presented in Table 8.

Table 8

Statistics of alphas for 500 observation models.

The statistics represents poorer performance for the funds with shorter period. It means that the last two years of the initial period are worse than the sample at average and they even impair the nine years of performance. It can be seen through the lower values of all statistics except for standard deviation, which is higher, but that also is worse. The mean and median values for previous setup are almost zero, but for 500 observation models they are negative and ten times further from zero. In general new alphas are lower by 20-30%. The purpose of the experiment was to compare top alphas of these sample with the initial sample. Top and worst alphas are presented in Table 9 and Table 10. There are still funds that have less than 500 observations, but there are very few of them. Also, among top and worst there is one fund that has a little less than 500 observations. But the difference is insignificant.

Table 9

Top 4 alphas from 500 observation models.

Table 10

Lowest 4 alphas from 500 observation models.

The mean returns in the tables are for the whole sample (9 years), alpha and number of observations are for new sample. It is made so that funds alphas can be compared with previous results (since mean returns are close enough to alpha in the previous regressions). As it can be seen from the tables that the top and worst funds in terms of alpha have changed. But the same as for initial sample energy sector has been underperforming the market during the last two years of the initial period. Among the funds with top alphas only “Aleksandr Nevskij” is still presented. This can only mean that previous top funds with a bit more than 500 observations have huge and positive returns during the period of time which is out of the current sample. Previously the worst fund “Merkuri globalnaya elektroenergetika” is no longer among such probably because most significant negative returns it has had before the current sample. “Uralsib energeticheskaya perspektiva” is far away from worst in nine-year period, “sberbank potrebitelskij sector” and “alfakapital potrebitelskij sector” are not among the top in initial sample too. These funds outlines the problem of having funds with different life time when estimating skill with alphas: while a fund with smaller number of observations can have good results it may be would not have such for the whole sample period, moreover, if funds with lower results but full history will be shorten to the same amount of observations it may outperform that fund.

The problems mentioned above can be solved using approaches that create the distribution of luck for each fund. As long as they bring out critical values for each fund that characterize skill even funds with the highest alphas will not be considered skillfully managed if they do not pass the border. One of the method to obtain this distribution of luck is bootstrap approach.

The process of bootstrap approach is described in short in chapter “Theoretical background”. More detailed information can be obtained from the core papers, the authors of which are the founders of this methodology of evaluating funds' skills. The approach is used to produce a huge number of random portfolios that are created without skill. This zero skill portfolios make distribution of luck, which should represent all possible results in general. To be managed skillfully a fund should be among top in this distribution of possible results. Bootstrap is the process of creating a huge number of portfolios that have same characteristics - coefficients in regression. It is an artificial sample for each fund. By changing the alpha coefficient to zero for new portfolios during this process the distribution of luck for initial fund is created.

As it was already mentioned, the first step is to store beta. Also, analyzing beta can provide useful information about the funds` performance and models used. The statistics can be seen in table 11.

Table 11

Statistics for beta coefficients for three base models.

The statistics of beta can explain low r-squared for some funds. For a certain fund with a certain model specifications with beta coefficients close to zero r-squared will be low. It is because fitting errors will be very large with high standard deviation and alpha of such fund will be close to mean value of dependent variable. Risk market premium or the benchmark has the highest betas. It means that many funds have high correlation with the market. But the funds that are uncorrelated with benchmark have low r-squared. Judging from the table the model with most influential coefficients is specification with Brent oil and US dollar prices in rubles. The higher standard deviation for risk market premium in this model in comparison with CAPM specification can be explained by correlation of explanators, but the difference is small, so it is not critical. Most funds have positive beta and correlation with the market (RMP). No fund can show returns systematically opposite to the market, as there is no beta with high negative values for RMP. Also, rising dollar can harm most of the funds as they have negative beta with US dollar ruble prices. Crude oil in US dollars have small betas, most of them are positive, but most values are close to zero. Since the data represents returns and Brent oil do not have higher volatility than the market, small beta can mean low influence on funds` returns above risk-free rate. Brent in rubles have different influence and its direction on different funds, because median and mean are close to zero. It is logical since USD has negative influence and Brent - slightly positive (while currency changes have smaller volatility, small positive Brent influence can match negative from dollar).

The next step to get distribution of luck is connected with residuals. Fitting errors are the only thing that differentiate random portfolios for one fund. Unique features of fund`s performance are stored in residuals, which are used to create a new portfolio: the only limitation of new series of residuals is that it should have the same length, meaning that not all observations can be used and the order is random. Also, these residuals are the part of research were high risk or small number of observations is properly dealt with. If a certain fund has small number of observations and because of that has high returns and is outperforming most other funds with longer series it will have huge fitting errors. In the process of resampling (with possibility to replace) of residuals new series of pseudo residuals will also have large fitting errors and some of them may have even greater. Because of that the distribution of luck for this particular fund can be shifted to the right and the real fund`s alpha can be smaller than the critical value and thus it will not be considered as skilled. The process of resampling is shown on Picture 9.

Picture 9 The process of creating pseudo errors

Some residuals are duplicated and some excluded. But the problems appear when funds have missing values. Here different assumptions may take place. Residuals can be mixed alongside missing values and then sorted. In such case the initial length for funds with missing values can be changed a bit. The less observations a fund initially has the more new missing values will appear. For funds with a few missing values it is vice versa, the sample will increase. More logical will be to make all the new residuals with full length. The initial residuals will be reorder and replaced with 2209 observations (maximum length). Another possibility is to replace only existing values, so that the length of pseudo returns will match number of observations of the initial sample. Picture 10 illustrates simple examples for all three options.

Picture 10 Pseudo residuals types

This research is based on option 3, only existing values are shuffled and replaced. But one model is also tested with option 2. The results are presented after the final step in this approach. As a result of this step a matrix with 175 columns and 2209 rows is created. Each fund has its own column with one set of pseudo residuals in it.

When pseudo residuals are ready new “unskilled” returns can be estimated. Using the same regression with the same beta but new residuals and zero alpha will bring out a set of returns above risk-free rate with zero skill. Since the regression from which initial residuals were taken has returns above risk-free rate, the new pseudo returns also include this factor. These returns forms a new portfolio that is operated solely by luck.

The last step in the loop is to estimate alpha for pseudo portfolio. This time all the coefficients belongs only to it. The estimated alpha is one of many that forms a distribution of luck in terms of picking stocks. This is an important note, because alpha is responsible for picking skills.

One loop consists of 3 parts. The first on is to estimate one series of pseudo residuals for each fund. The second - to restore pseudo returns (again one series of returns for each fund) using zero alpha, beta (that was stored earlier), pseudo residuals in the regression. And the third - to estimate alpha of new returns using one of the model specifications. Repeating the loop by many times provides many alphas. This alphas form distribution of luck. In this research number of bootstrap iterations is 1000. Accuracy of results is rising with number of iterations. But in order to include all the assumptions the code in R is made complicated and it takes about 30 hours to estimate the results on average home computer.

Table 12

Statistics of simulated alphas and real alphas.

The distributions of all simulated alphas in comparison with real alphas for Brent in rubles and CAPM model can be found in Appendix 5. Each one consists of 175000 alphas and represent the whole market`s possible results according to one of the models. The statistics of real and simulated alphas are compared in Table 12.

Previously it was mentioned that there are two proper options to deal with pseudo residuals. All the cases presented in the table above mix and replace only existing observations. One of them also fill up the missing values by duplicating existing residuals in the same way they are mixed replaced. Doing so all the pseudo returns have maximum observations and so have the new pseudo portfolios. There are both pros and cons from economic point of view. Fitting errors for small existing period will be retranslated on period with worse economic situation on the market and thus the pseudo portfolio will be outperforming the benchmark and vice versa for better economic situation. At one point it makes funds with high alphas and short time period have a more suitable simulated distribution with higher alphas. But if the situation on the market was better in the past periods and residuals from worse times are retranslated than some funds can be overestimated. Same goes for funds with bad skills that will be underestimated in the first case and overestimated in the second. This model seems to have lower standard deviation, so the alphas are not scattered far away from each other. This grouping will lead to more funds with good and bad skills. More about the model can be found next to overall results.

A proper comparison should involve distributions of both real and simulated alphas. The research involves 3 models, model with separate factors Brent oil and USD/RUB is slightly better than others. That is why its results are presented below on Picture 11 and distributions for other two are presented in Appendix 5.

Solid line on the graph represents distribution of all simulated alphas (for each fund) for model specification with Brent oil and USD/RUB, dashed line is real alpha distribution for the same model. As expected the mean value of simulated distribution is close to zero. The distribution is almost symmetric to zero line except for the peak, which is slightly to the left. This peak is compensated with a higher left part. Picture 11 illustrates that there more funds that are underperforming, since there are more funds with high negative alpha and less funds with high positive. The statistics tells the same: 1^st and 3^rd quarter values of real alphas are smaller than simulated. Higher long tails of real distribution suggests that there some funds that may have skill (both positive and negative). But in general according to this graph there is no evidence of superior performance for Russian funds.

Picture 11 Distributions of real alphas compared to bootstrap alphas

And what comes for single funds? The bootstrap approach allows to estimate skill of each fund separately, because there is special distribution of luck for each fund. The process of measuring skill was already mentioned in other chapter. Picture 12 illustrates it on an example for one fund.

The graph presents the distribution of simulated alpha for fund "Alfa capital aktsii" estimated with model containing Brent oil and USD/RUB factors. The dashed lines are critical values. They are estimated as percentiles of series with simulated alpha. The solid vertical line is value for real alpha. If the solid line is between two dashed, then there is no evidence of skill. If the real alpha is in the right part of the graph and pass right dashed line then the fund is managed with positive skill, to the left - the skill is negative and managers are destroying fund`s value. But here there is a dilemma: what percentiles should be chosen. If the zero hypothesis is “random returns for a fund” then there are two sides of nonrandom returns for alternative hypothesis: skill to provide positive returns and outperform benchmark and negative skill to destroy value and underperform. If this zero hypothesis is being tested, then for 5% significance level the percentiles should be 2,5% and 97,5% (so that 95% is in the center). But then the alternative hypothesis is two sided and it can be split into two: one, fund`s positive performance is nonrandom, the other one: fund`s negative performance is nonrandom. Then zero hypothesis can also be divided into two parts: funds positive results are random and funds negative results are random. In this situation for 5% significance level the percentiles will be 5% and 95%. This way there will be more funds with positive and negative skill. This case is equivalent to 10% significance level for the previous hypothesis. Picture 12 illustrates only an example. The results for all funds (only for model with Brent oil and USD/RUB) are presented on the Picture 13.

Picture 12 Measuring skill

There are two graphs below. The upper one represents real alphas of funds and their critical values. The lower graph shows which fund has positive or negative skill. The critical value on top is 97,5% percentile and on bottom - 2,5%. As it is clear from the graph no fund has positive skill. Even the highest alphas have even bigger critical values, because their high fitting errors form a few series of high pseudo returns. But there are some funds on the border and can be assumed skilled if the significance level is raised. Another case occurs with negative skill. Some of them are also on the border but from the other side. There are total of 6 funds that have nonrandom negative returns according to this approach. A few funds that are close to the border can have their results changed if number of simulations is changed or by chance (since simulations are random then if number of bootstrap iterations is low there can be different results for the same setup).

Picture 13 Funds results for Brent and USD/RUB model, 5% significance level for initial H0

Since there are no funds with positive skill with 5% significance level, the last should be increased. As it was already mentioned if the initial hypothesis is divided in two, then the fund`s alpha has to beat 95% of simulated alphas instead of 97,5% to claim skill. Thus the two new hypothesis should be tested. The real and simulated alphas remain unchanged, only the critical values will be a bit closer to zero. The graph with new borders for skill is presented on Picture 14.

The graph shows that with changes in percentiles 3 more funds are suggested to have negative skill and also 3 funds with positive appears. In this situation inside the borders there are 90% of simulated alphas. Among the 5% top alphas there are three funds, among the bottom 5% 9 funds. These funds will be mentioned later. There is no point to further increase the significance level and press the borders even further to zero. Other funds that are not outside the borders can be ranked by the difference between their alphas and critical values. The closer they are to the top border - the more chances that they are managed skillfully. This goes only for funds inside the borders. The ones that are outside should be ranked how much further from the border they are.

Picture 14 Funds results for Brent and USD/RUB model, 10% significance level for initial H0, 5% for two new H0

Practically the same situation is with the two other models. The results can be found in Appendix 6. For initial H0 with 5% significance level CAPM claims that there are 6 funds with bad skill and model with Brent oil in rubbles suggests 7. For both cases no fund has positive skill. If borders are expanded to 10% significance level (or 5% for two new hypothesis) then CAPM has 2 funds with alphas above critical value and 9 below the bottom border, while Brent oil in rubles suggest that 3 funds have positive skills and 10 - negative. Both cases with initial H0 and two new hypothesis shows that the market of equity funds in Russia in general is managed without good skills as most funds have no evidence of nonrandom performance and negative skills prevail.

A little different situation is when all pseudo residuals are filled with full length observations (2209 in this research). Since the distribution of simulated alphas is more compressed (its minimum,1^st and 3^rd quarter, maximum are closer to zero in comparison with other simulated alphas distribution) there are more funds with good and bad skills. Moreover, there is no need to increase significance level as the number of skilled funds is comparable with 10% significance level for other models. The results are presented on Picture 15.

Picture 15 Funds results for Brent and USD/RUB model with 2209 obs residuals, 5% significance level for initial H0

Once again funds with negative skill prevail. The graph shows that 2 funds have positive skill and 13 funds - negative. If the significance level is increase the number of funds with skill rise to 7 for positive and 18 for negative. As it can be seen from the graph the border lines have lower peaks. That is why the funds that are not considered skillful even for 10% significance level for other models can have skills according to this one. This model is arguable but the results still may be taken into consideration (with adjustment using other models) in the process of decision making.

The funds that are considered to have nonrandom positive returns for all models are presented in Table 13. The funds with negative skill can be found in Table 14. The tables show funds` names, their alphas, critical value and distance from border (real alpha minus critical value - a way to rank funds).

Table 13

Funds with positive nonrandom returns.

Table 14

Funds with negative nonrandom returns.

The most valuable information in these tables is in the column Difference. It represents how much higher the real alpha of a fund is above the border. For table with positive skill is used the top border, for negative skill - bottom. The higher the value Difference is - the better for all funds. The data in the table is sorted by the difference between real alpha and critical value for 5% significance level of initial H0 that claims random returns for a fund. As it can be seen from the tables the funds with positive and negative skills are mostly the same for three different models. But their ranking differs. The funds that have lower rank with 5% significance level probably have high standard deviation of residuals, meaning that there is much difference between percentiles. The negative skill results for model with 2209 observations of pseudo residuals for all funds have includes all negative skill funds from other models with 10% significance level and add one more. It can mean that this method of dealing with pseudo residuals can be applied and its results can be taken into consideration.

According to the tables above the best funds are “sberbank globalnyj internet” and “interfin telekom”. The funds that destroy their value the most: “merkuri globalnaya elektroenergetik”, “interfin elektroenergetika”, “alfakapital elektroenergetika”, “pallada energetika”, “bks aktualnye idei”. More results with detailed information and economic point of view can be found in the next chapter “Results”.

Synthetic portfolio approach has similar to bootstrap idea: to create a huge number of random portfolios without skil...