Размещено на http://allbest.ru

National Research University -Higher School of Economics

Predicting stock market returns using text mining of online news headlines

Eskerhan Dzhantamirov,

Vladimir Pyrlik

Saint-Petersburg

Contents

Introduction

1. Literature review

2. Methodology

2.1 Data

2.1.1 Textual data

2.1.2 Market data

2.1.3 Feature selection

3. Modeling strategy

3. Modeling strategy

3.1 Description of the models

Bibliography

Annexes

Abstract

Introduction

Data for the research is collected from different sources as there are two types of data needed for this research:

a) Textual data (headlines);

b) Financial data.

Since there are significant differences between these two datasets so different approaches should be implemented for its retrieval, processing and interpretation.

2.1.1 Textual data

For our research textual data, or rather, headlines were generated from website of an informational agency RBC (RosBusinessConsulting) http://www.rbc.ru - the first and only business channel in Russia. There are several significant reasons in favor of this particular informational agency:

1) Since this is a business channel there is no need in too scrupulous news filtration (we can be certain that there is no general news, for instance, in financially related section of news);

2) Rather big archive of the news (RBC has almost ten times more headlines than other news source LENTA.ru that we used in the previous research) and small interval between the releases of the news.

The data were obtained by parsing and only related to economics or finance section news were selected, also intentional data sorting and filtering have been made to prevent the emergence of "noise" in the final stages of the study. Data retrieval was executed with help of commercially available informational agency Integrum http://www.integrum.ru (agency specializes on the analysis of mass and social media) that provides access to the news archive of different informational portals including RBC. Thus, totally 502000 headlines were obtained for the period from 16.11.2009 to 06.10.2015, the total number of unique words in them - 50046.

For further description and data processing, a process of preliminary data processing was held. Time and date of occurrence of the news has been separated from headlines for easy syncing them with market data on the next stages of research.

At the next process step with help of Morphological Analyzer pymorphy2 https://pymorphy2.readthedocs.org/en/latest/ for Python programming language extra characters and stop words (the list of stop words was gathered manually from different sources such as: list of common stop words in various languages in Python https://pypi.python.org/pypi/stop-words/2014.5.26, list of stop words from website of Ranks NL http://www.ranks.nl/stopwords/russian and from the “tm” package for R) were removed, words were normalized (for instance, verbs were converted to indefinite form, nouns were given in the nominative case and so on for all parts of speech), headers were divided into individual words and converted to lowercase, whereby the data was in form of structured table as follows:

Table 1 - headlines after primary processing

At the next step after the initial processing of information certain headlines were deleted because they were editor's notes or did not concluded much information but were too recent and could bias our results. For instance, in such cases as with the last three lines in the following table 2:

Table 2 - example headlines of with lack important information

After delete of these headlines the total amount of them decreased from 502000 to 470000 and visual evaluation of the data was possible: the minimum number of words in the title - 1, the maximum - 25, the average length of the header about 9 words.

Figure 2 - histogram of probability densities

The latter histogram shows a distribution of headers according to the count of words in them. As can be seen from the histogram distribution of headers is close to normal, however, it has some asymmetry to the left side.

Figure 3 - wordcloud (words with the frequency >5000)

Figure 3 shows the most common words in headlines, the larger the word and oilier secretions, the more frequent word among the headlines. It should be noted that this picture shows the most popular words without pre-treatment of the text, since even stop words and numbers are not removed from the data under consideration. For example, the most popular are the word “РФ” which stands for Russian Federation is found in 16.4% of all titles, the second occurrence frequency titles is word “Рубль” which stands for Russian Ruble- 23.5% of cases.

However it should be noted that these words are not final terms, to determine them we should execute in a subsequent part of the study certain algorithm for text clusterization. Finally at the next picture we can see the most common 50 words along with their frequencies.

Figure 4- 50 the most popular words with their frequencies

2.1.2 Market data

As the numerical data historical data archival values of the MICEX index from the investment-analytical resource Finam.ru http://www.finam.ru were collected. A total of 1483 daily values of the MICEX index for the period from 16 November 2009 to 16 October 2015 were extracted.

The primary analysis showed that the minimum value of the MICEX index at the close of trading - 1197, maximum -1860, an average of 1495.

Figure 5 - Distribution of the daily index values

Also for the further research goals we introduce a new variable log-return: r (for daily data) and r2 (for hourly data), which is the log difference of the MICEX index.

Figure 6 - Daily dynamics of the MICEX index

Figure 7 - Daily dynamics of the log-returns

Figure 8 - The empirical distribution of daily log-returns

Figure 5 shows us the distribution of index values at the time of closing, as can be seen from its distribution is approximately normal, as in the case of text data. From the graph we can see that it has bimodal shape and explanation for that is that log-returns are non-stationary time-series and that during this time period time- series had several different regions.

Figure 7 illustrates us daily dynamics of the log-returns of the MICEX index. According to the figure 7 we can say that it is a stationary time series because there is a long-term steady-state level. However we have conducted a test for stationarity, to have convincing evidence:

1) According to the KPSS test for level stationarity the time series is stationary and this assumption could not be rejected at the significance level no less than 0.1;

2) According to Augmented Dickey-Fuller test the time series is stationary and unit root is rejected at significance level 0.01.

The next figure 8shows the empirical distribution of daily log-returns. Distribution graph has slightly more weighted left tail and it happened due to sharp and vivid falls that are typical for the market during the period of reduction.

Figure 9 - Distribution of hourly values of the MICEX index

Since Moscow Exchange works only on weekdays, respectively observing for weekends and holidays was no possible to get, which resulted in a de-synchronization of dates market and text data. To resolve this problem those observations that came out on the weekends or on the public holidays have been removed from the data, as there is no data to synchronize with them.

Then hourly archival values of the MICEX index were obtained from the same investment-analytical resource. A total number of observations were 15777 for the period the same period of time.

The primary analysis showed that the minimum value of the MICEX index at the close of trades - 1193, maximum - 1863, the average value is 1488.

Figure 10 - The empirical distribution of hourly log-returns

Figure 9 shows us the distribution of values of the index at the close of trading, as can be seen from it; data has approximately normal distribution.

In hourly data the problem desynchronization has occurred not only with the dates but also with the hour's data of the MICEX index. Since the Moscow Exchange works only for 9 hours a day from 11.00 to 19.00 hours and also on weekdays only, all news, published on weekends and holidays, were discarded, and the news, released after 19 hours, were attributed to the values of the index of 11:00 hours of the next day .

2.1.3 Feature selection

At this stage we describe pre-processing of the text during which we conduct specific operations to turn textual information into a numeric representation. The first part of pre-processing is feature extraction, during which we have eliminated those words that were too general or have not contained any valuable information (e.g. stop words). The next step is feature selection which main goal is to lower dimensionality of our feature set. For this purpose we have implemented a certain methods of filtration: 1)we have created a corpus that contained only those words that appeared in the whole corpus more than 5 times; 2) from the new corpus we have chosen the most frequent 200 words that in total amount of fords appeared 1713858 times which is almost 40% of total appearances.

The next phase of the study required the extraction of topics. By topic is meant a more general word that includes in its meaning some other words. Figure 3 shows the most common words in headlines, the larger the word, and oily secretions, the more common word. Among those 200 words there were certain “words” that had no meaning or were lead to incorrect form during text normalization, so they were deleted (for example, such words as “i” or “к”). To the 180 words that were left we have assigned certain topics (in total 12 topics) each of which represents a vector of words related to similar content. The example of topics can be seen in Annex 1. Even though we have assigned topics manually, there are certain methods for automatization of this process. For example, LDA topic modelling algorithms for discovering the abstract "topics" that occur in a collection of documents. This technology has been successfully used in similar to ours works and showed good results (Jin et al., 2013) however the main disadvantage of this method is requirement of computing power. We have tried to implement this technique to our goal, in our case the task was specific because of the extremely short texts (sometimes shorter than twitter), and even in spite of that we were able to receive reliable results on short period of time using “twitter lda” (topic modeling for twits). For the long period it was not so easy and the only way to receive topics that were reliable was iterative repetition (one iteration took from 8 to 10 hours on the computer with 4 Gb of RAM and CPU 2.3 GHz), until the predictive models did not show improvement, but due to lack of computing power it seems impossible.

3. Modeling strategy

Thus, we have obtained a new dataset (see example in table 3) via merging two different datasets (textual and market data). The new dataset has information about features that appeared in textual data for every hour of trading on the market. Our main hypothesis is that inclusion and utilization of textual data helps to improve predictive power of the model in comparison with the models that do not utilize textual data. To test this hypothesis our corpus is divided into three parts in a ratio of 60% to 20% to 20%. First 60% of our corpus is a training sample on which we train our model, next 20% is validation sample on which different models are compared with each other to reveal the best and finally the last 20% on corpus are used to determine the predicting power of the best model.

Table 3 - sample of the new data set with textual topics features

3.1 Description of the models

There are three types of models that are utilized in this work: linear regression model, SVM (Support Vector Machine regression) and random forest. Moreover models are also divided by two types of factors for the model (10 lags, and text features that were determined manually).

1) general linear model with stepwise model selection by AIC (Akaike information criterion) - used as baseline model and predicts log-returns of MICEX. This model is a semi-automated process of building a model by successively adding and removing variables based solely on the AIC criterion and the chosen as a baseline model because of its generality and simplicity of comparison with other models. The first model includes only lags and predicts log-returns of MICEX, whereas second linear model also includes lags and additionally text features;

2) support Vector Machines regression model can be applied not only to classification problems but also to the case of regression and one of the reasons in favor of this model is relatively small time consumption for training large datasets. Moreover one of the advantages of Support Vector Machine is that it can be used to avoid difficulties of using linear functions in the high dimensional feature space and optimization problem is transformed into dual convex quadratic programs. In regression case the loss function is used to penalize errors that are greater than certain threshold - е. Such loss functions usually lead to the sparse representation of the decision rule, giving significant algorithmic and representational advantages. The same way as with general linear model there two types of SVM: the first model includes only lags and text features are added to the second model;

3) random Forest - is a bit more sophisticated and versatile machine learning method capable of performing both regression and classification tasks. A distinct advantage of random forest is the same as SVM's - power to handle large datasets and what is more important random forest has inbuilt regularization so we cannot over fit the model. However ability to discover more complex dependencies costs more time for fitting and computational complexity. In the same fashion as for the two models above we provide two different types of features to the random forest model.

We use mean absolute percentage prediction error (MAPPE) for the training of the models and for the forecasting as a measure of prediction accuracy of a forecasting method. Utilization of MAPPE metrics is partly explained by the fact that we compare the results not only of different models with each other in a certain period but also one model in a different time periods and one of the few ways to compare such models is based on percentage error. Moreover Spyros Makridakis in his article (Jin et al., 2013) claims the MAPPE to be the standard metrics with best characteristics among the various accuracy criteria In addition, it can be used for both evaluating large-scale empirical studies and for presenting specific results.

Finally MAPE is calculated as:

,

where A_t is the actual value and F_t is the forecast value.

The difference between A_t and F_t is divided by the actual value A_t again. The absolute value in this calculation is summed for every forecasted point in time and divided by the number of fitted points n. Multiplying by 100 makes it a percentage error. The smaller the value of MAPPE, the closer are the predicted time series values to the actual values, that is to say, a smaller value suggests a better predictor.

4. Results

Results have shown that inclusion of textual data as one of the factors for forecasting not improved the results of general linear model; improvement of outcome is observed only for certain specifications of the SVM model.

More specifically: as it can be seen from the table 4 (highlighted cells) decrease of the MAPE is observed only for the second test sample and only with inclusion of lags from 7^th to 10^th with the decrease of MAPE 0,0024; 0,0023; 0,0015 and 0,0017 percent respectfully.

Full evaluation results for different regressions are gathered and shown in ANNEX A2.

Table 4 - results of SVM evaluation for the model with 7:10 lags.

text headline financial regression

Conclusion

The main aim of this work has been to find evidences in favor of hypothesis stated by Khadjeh Nassirtoussi - that headlines themselves includes a lot of valuable information and inclusion of them in different forecasting models will help to improve the predicting power of the models. Results have confirmed this hypothesis to some extent. Even though a small decrease of the MAPE was observed only for certain specifications of the SVM model we can say that there are several possible ways to improve the results and this research itself:

1) Weakest part of the research is feature selection. All the features were extracted manually on the rule based basis. There different automated and semi-automated techniques, for instance, like LDA topic modeling, that extract features in accordance with certain mathematical basis and shows good results in numerous researches. It will be fruitful to overcame current barriers for their implementation and utilize them on the data for the Russian market;

2) Another possible way to enhance the results is choice of additional models with possibility of scrupulous tuning of their parameters;

3) For the purpose of reliability of the research it would be more useful to use several different metrics of model performance. Thus we would guarantee the accuracy of the results;

In overall research has partially reached its goals and has shown that future work in this area has a potential with a certain adjustments to the proposed technique.

Bibliography

1. Bollen, J., Mao, H., & Zeng, X. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2(1), 1-8. http://doi.org/10.1016/j.jocs.2010.12.007

2. Geva, T., & Zahavi, J. (2014). Empirical evaluation of an automated intraday stock recommendation system incorporating both market data and textual news. Decision Support Systems, 57, 212-223. http://doi.org/10.1016/j.dss.2013.09.013

3. Harris, L. E. (1999). The Information-Content of the Limit Order Book: Evidence from NYSE Specialist Actions The Information-Content of the Limit Order Book: Evidence from NYSE Specialist Actions Abstract.

4. Jin, F., Self, N., Saraf, P., Butler, P., Wang, W., & Ramakrishnan, N. (2013). Forex-Foreteller: Currency Trend Modeling using News Articles. KDD Demo, 1470-1473. http://doi.org/10.1145/2487575.2487710

5. Junquй de Fortuny, E., De Smedt, T., Martens, D., & Daelemans, W. (2012). Media coverage in times of political crisis: A text mining approach. Expert Systems with Applications, 39(14), 11616-11622. http://doi.org/10.1016/j.eswa.2012.04.013

6. Junquй de Fortuny, E., De Smedt, T., Martens, D., & Daelemans, W. (2014). Evaluating and understanding text-based stock price prediction models. Information Processing & Management, 50(2), 426-441. http://doi.org/10.1016/j.ipm.2013.12.002

7. Khadjeh Nassirtoussi, A., Aghabozorgi, S., Ying Wah, T., & Ngo, D. C. L. (2015). Text mining of news-headlines for FOREX market prediction: A Multi-layer Dimension Reduction Algorithm with semantics and sentiment. Expert Systems with Applications, 42(1), 306-324. http://doi.org/10.1016/j.eswa.2014.08.004

8. Lampenius, N., & Zickar, M. (2005). Development and validation of a model and measure of financial risk-taking. The Journal of Behavioral Finance, 6(3), 129-143. http://doi.org/10.1207/s15427579jpfm0603

9. Li, Q., Wang, T., Gong, Q., Chen, Y., Lin, Z., & Song, S. (2014). Media-aware quantitative trading based on public Web information. Decision Support Systems, 61, 93-105. http://doi.org/10.1016/j.dss.2014.01.013

10. Wang, B., Huang, H., & Wang, X. (2013). A support vector machine based MSM model for financial short-term volatility forecasting. Neural Computing and Applications, 22(1), 21-28. http://doi.org/10.1007/s00521-011-0742-z

Annex A1

Annex A2

Table A2 - shows a MAPE for different specifications of the models

Abstract

With the rapid spread of the Internet around the world large amounts of free unstructured data emerged and consequently emerged a new field of the research - data science. There are many fields for utilization of this unstructured data from politics and medicine to finances. The main aim of this particular research is to predict intraday movements of the MICEX index based on the text of financially related news-headlines from the RBC new company.

Data was collected, processed and at the next stage features on 12 topics were extracted from this data. Using these features as one of the factors in several regression models we have tried to predict the log-return of the MICEX index. Simple model linear model failed to find any improvement, however for the model that is slightly complicated (SVM) there found specifications when inclusion of the textual data would yield better results.

The results of the proposed study will be hopefully useful directly for investors that are going to make investments in different types of stocks and for the other economic agents that are involved in Russian financial market.

Keywords: machine learning, financial market prediction, topic modeling, intraday market data, autoregressive model, support vector regression, RBC, MICEX.

Размещено на Allbest.ru