Processing, analysis and analytics of big data for the innovative management

Размещено на http://www.allbest.ru/

Kyiv National Economic University named after Vadym Hetman

Department of Information Systems in Economics

Processing, analysis and analytics of big data for the innovative management

Maxim Krasnyuk Ph.D. in Economics, Associate Professor

Svitlana Nevmerzhytska PhD in Technics, Associate Professor

Tetiana Tsalko PhD in Economics, Associate Professor

Ukraine

Summary

The changes that have taken place in the economy of Ukraine in recent years as a result of the consistent impact of global macroeconomic, epidemiological and military factors of the current deep crisis have actualized for organizations, enterprises and corporations the urgent need to maximize their effectiveness, the key component of which is the innovative management of their complex projects.

The primary scientific and practical results presented in the article regarding the specifics of the methodology and technologies of deep intellectual analysis and analytics of Big structured and semi-structured Data in innovative project management must be taken into account during crisis reengineering of the management systems of domestic private and public organizations, enterprises and corporations. intellectual innovative crisis reengineering

Moreover, the obtained results are relevant and applicable for not only local organizations, enterprises and corporations, but for international on emerging markets in the context of future global and regional macroeconomic and possible epidemical crisis phenomena.

Keywords: Big Data, innovative management, project management. data analysis and analytics

Introduction

The term "Big Data" has been around for some time, however, there is still a lot of confusion surrounding its meaning and interpretation. The fact is that this concept is constantly developing and improving, as it remains the driving force behind many waves of digital transformation, including in the field of innovative management.

Despite such a recent appearance of the term, big data existed before, but it did not have much practical and applied value in everyday life and, even management, because studying information requires significant computing power, a large amount of time and high financial costs.

The concept of "big data" itself arose in the days of mainframes and supercomputers and related scientific computing, since scientific computing has always been characterized by complexity and is usually inextricably linked to the need to process large volumes of information.

Nevertheless, it should be noted that in most cases it is appropriate to use a simpler definition that corresponds to established and simpler definitions that fully correspond to the opinion of scientists and practitioners: big data is a set of technologies designed to perform three operations:

storage of hundreds of terabytes or petabytes of data, which conventional relational databases cannot effectively use;

organization and management of rapidly arriving data in very large volumes, in particular unstructured data consisting of texts, images, videos and other types of data;

analysis and analytics of Big structured and unstructured data in scripted and parallel hybrid modes.

Big Data is often characterized by:

a large amount of data in many environments;

a large variety of data types that are often stored in big data systems;

the speed with which most of the data is generated, collected and processed.

Today, the term Big Data is usually used to refer not only to the accumulated huge arrays of structured, semi-structured and unstructured data, but also to the corresponding technologies and tools for their processing, in-depth analysis and analytics with the aim of extracting all possible additional advantages with the aim of total improvement of management efficiency [1 ].

The main advantages of using technology include:

obtaining qualitatively new knowledge through comprehensive analysis of all information in a single analytical repository;

expanding the functionality of the existing business support information systems;

increasing the efficiency of using server hardware resources;

ensuring the minimum cost of using all types of information due to the possibility of using open source software and cloud technologies.

However, before launching a Big Data project, more key questions often need to be asked: What are the desired business goals we are trying to achieve and what challenges will we have to overcome in order to achieve this? What data is currently available? Are they structured or unstructured? What types of data should be collected? How to best use this data? What technologies will help us use this data to support and expand the enterprise? What protocols can be put in place to ensure data security and quality standards are met?

The modern concept of Big Data Analysis & Analytics is constantly being developed and revised, as it remains one of the driving forces of many waves of digital innovative transformation of management in conditions of dynamic and stochastic crisis factors [2-5].

After all, all individuals and legal entities have been generating, registering and storing (in DBs, repositories and data showcases) huge amounts of heterogeneous structured and semi-structured information (quantitative, qualitative, text, hypertext, transactional, geo-information, multimedia, meta-information, etc.) for years regarding all aspects of their business, technological and managerial activity. In addition, the rapid development and spread of Big Data, Web 4.0 and Web 5.0, IOT, FinTech, blockchain in recent years caused an additional avalanche-like increase in stored data [6].

In the modern conditions of the development of the global economy, and in connection with the emergence of new branches of economic activity in the field of digitalization, the application of innovative technologies for the preparation, processing, analysis and analytics of extremely large arrays of heterogeneous data leads to the obtaining of additional competitive advantages by users at the state, regional, branch and corporate levels of management, which is especially relevant in the conditions of dynamic crisis phenomena [7-8].

It is in the process of their activity that ALL industrial and service enterprises, corporations, state and local authorities, public and non-profit organizations generate and accumulate large volumes of structured and semi-structured data in both batch and streaming modes. This big data contains a significant potential for finding and formalizing hidden new regularities (knowledge, patterns), which are the basis for making optimal and effective managerial and technological decisions, including and in the field of innovative project management.

However, it is hoped that these large volumes of structured and semistructured information will significantly and directly help users, enterprises and government structures in making operational and high-quality managerial and technological decisions (including in the field of innovative project management) without effective and justified application of relevant innovative intellectual technologies, algorithms and means of in-depth intellectual analysis of these big data are in vain [9].

After all, currently a small part of the stored structured data is thoroughly processed by experts using classical statistical methods of analysis in batch mode. This determines the additional relevance of the effective application of big data mining or knowledge discovery in big databases, which involves in-depth intellectual analysis of accumulated volumes of heterogeneous big data in batch and streaming modes (including using scenario and ensemble technologies) in order to identify new, non-trivial, hidden, practically useful and interpreted regularities [11 -13].

From these regularities and patterns (after their verification and interpretation), the corporate knowledge base is mainly formed, including and for expert systems and classical artificial intelligence systems [14-19].

The goal of solving an important current research problem - the study of the features of intellectual analysis of big data in the field of management - requires a systematic study of thorough theoretical provisions, relevant and proven practical skills and heuristics regarding the organization, design, deployment and adjustment of technology, methods, algorithms and scenarios, low code software tools for detecting knowledge (patterns) in structured and semi-structured packet and stream data (including Text-Mining, Social Network Analysis, Web-Mining, Multimediamining, metadata mining).

As a result, the above-mentioned comprehensive goal of the research will provide scientific and practical results, which will have increased relevance for not only local organizations, enterprises and corporations in the current conditions [20], but for other emergency markets in conditions of multisystem crises [21 ].

Problem statement and relevance of the research

Thus, taking into account the above, the following components of the comprehensive concept developed by the authors of the effective use of big data analysis and analytics as a component of innovative anti-crisis management become particularly relevant:

the study of best practices in the use of in-depth intellectual analysis of large volumes of structured and semi-structured information by private enterprises and corporations, state authorities and state enterprises for the preparation of operational and high-quality management and technological solutions;

knowledge of the concept, mathematical basis and features of using the entire range of basic algorithms/methods of intellectual data analysis;

the ability to correctly configure ETL or ELT or custom pre-processing of input data in accordance with the requirements of the selected type of machine learning and a specific method of data mining;

mastering the specifics of optimal selection of hyper parameters and optimization of local settings of data mining algorithms/methods [22];

familiarization with the technologies of organizing the effective application of data mining algorithms/methods in batch and stream modes, the possibility of their sequential and parallel hybridization and assembly [23-26] in the conditions of structured and semi-structured batch and stream data, metadata.

The main part

Regardless of the specifics of the industry, every company has two areas of application of technologies based on Big Data analysis, i.e. internal and external interaction.

As part of the study of external interaction, the accumulated customer experience is of interest, namely the understanding of customers through the analysis of social networks, their social status, age, preferences, etc., information about regions, market segments, satisfaction with the product or service, methods of promotion, as well as methods of contact, etc. External interaction can also include everything related to the business model and structure of the business and its interaction with the outside world, such as suppliers, partners and sales channels.

The study of internal interaction is aimed at studying and optimizing operational processes in the company, the purpose of which is to increase the productivity of not only equipment, but also employees, as well as the rational use of resources [27]. It is worth noting that enterprises will be able to gain the main competitive advantage not so much at the expense of data collection, but at the expense of the ability to quickly obtain useful information from the overall huge volume of generated Big Data.

Let's consider the advantages of using Big Data in enterprise management:

First, it helps to increase the efficiency of decision-making. The Big Data platform has the function of collecting real-time data resources and can obtain key information based on the rapid processing and analysis of massive data, which can meet the urgent needs of enterprises.

Second, promote more and more diverse decision-making tools.

Thirdly, it increases the persuasiveness and quality of the decisions made, as they are based on a large statistical base of source information, which significantly strengthens confidence in decision-making schemes.

Fourth, big data technology is also a guide for the company's operational strategy.

Evaluating the directions of influence of Big Data technologies on the process of management decision-making, the following should be highlighted:

Impact on the management decision-making environment.

Impact on the participants of the management decision.

Influence on the process of making management decisions by the organization.

Influence on management decision-making technologies.

As a rule, until recently, structured Big Data of relatively small dimensions and medium volumes were used in the field of management with the expectation of a constant of the following principle: the more information we have, the more accurate a forecast can be made. Also, the possibility of multidimensional comparison of certain data and relationships between them made it possible to find simple hidden regularities that were hidden from statistical analysis in the mode of hypothesis testing. All this provided a certain understanding of the problems and, ultimately, allowed finding suboptimal management solutions.

Typical features of big data analysis and analytics in comparison with typical classic Data Mining are listed in the table. 1.

Table 1 Comparison of traditional Data Mining and Big Data analysis and analytics

However, currently, Big Data in the field of innovative management of complex projects is not only very large in volume and growth dynamics, but also multi-criteria, semi-structured, mainly in NoSQL format and often requires the simultaneous use of several target variables to solve one management problem. That is why the traditional technologies of their collection, cleaning, transformation and preparation for use by machine learning algorithms are irrelevant. For example, large data with insufficient number of records (rows) can potentially provide unsatisfactory statistical power, while data with higher complexity (more attributes or columns) can lead to the "curse of dimensionality" or higher false pattern rates in case of a poorly performed preparatory reduction operation their dimensions.

In the case of using supervised ML or semi-supervised ML on big data, especially in the field of management of complex projects, special attention should be paid to the quality/truthfulness/correctness of labeling of big data. That is, additional attention should be paid to auditing/checking/controlling not only with regard to the target variable set of already marked records, but it is also necessary to carry out 24/7/365 monitoring (even at the ETL or ELT stage) of all features selected for use in ML.

Difficulties are also possible in the allocation/preparation of data sets for training and the subsequent quality control of ML results within the framework - especially if there is a need for express prototyping of previous results of Big data analysis and analytics.

Most often, the process of processing and further analysis of large volumes of data in the field of management includes data processing, configuration of methods/algorithms, and running simulations in an iterative cascade mode - because there is a constant need to change not only algorithm/method parameters, but also machine hyperparameters training, and sometimes the repeated process of preparation and processing of input data. And in all stages of this sequential iterative cascade process, there is an urgent need for feedback - analysis of the impact of decisions made on the results of analysis and analytics. That is why the AutoML mode for Big data analysis and analytics is gaining special relevance in the field of innovative management of complex and complex projects.

Conclusions

It is worth noting that Big Data Analysis and Analytics is a multidisciplinary science that involves sequential and parallel hybridization of various mathematical tools and a wide range of both classical and innovative methods and algorithms. In Big Data Mining technology, both deterministic and well-formalized methods of data mining and complexly formalized stochastic methods of analysis and analytics should be harmoniously combined (in particular, to optimize the hyper parameters of algorithms and take into account uncertainty in data and knowledge).

It can be unequivocally and reasonably stated that in the modern conditions of the development of the global information economy, in connection with the emergence of new branches of economic activity in the field of informatization and in the conditions of Big Data - the use of Big Data Analysis and Analytics - leads to obtaining additional competitive advantages by users, corporations and government structures at the regional and interstate levels, and therefore increasing their efficiency, capitalization, which is especially relevant in conditions of all types of uncertainty and multifactorial crisis phenomena.

Therefore, as a result of prospective research on the problem posed above in the text, the scientific and practical results obtained in the course of further author's research will relate not only to the features of algorithms and methods of deep intellectual analysis of big data, but also to such problems as:

organization of effective recording of big data and its optimal ETL\ ELT;

principles of optimal configuration and setting of their hyper parameters and operational parameters;

fundamentals of organizing their application in batch and stream modes;

the urgent need for systematic and total implementation and use 24/7/365 of the mode of searching for anomalies in big data and further detection of threatening signs and trends among them [28];

possibilities of their sequential and parallel hybridization and machine learning assembly.

In addition, the promising results of the authors' further research will be effective recommendations for all stages of the life cycle of CRISP DM projects on intellectual analysis of big data and knowledge within the framework of innovative project management: starting from setting the task of data analysis, through the stage of design and configuration, and up to implementation and support.

Even the obtained current intermediate results are relevant and deserve to be implemented not only in the framework of innovative management of Ukrainian organizations, companies and corporations, but also for implementation in the practice of management of foreign and international projects in the conditions of current and predicted multifactorial crisis phenomena.

References

1. [1 ] Maxim Krasnyuk, Illia Krasnuik (2024) Big data analysis and analytics for marketing and retail. Збірник тез Міжнародної наукової конференції «Штучний інтелект у науці та освіті» (AISE) 1 -2 березня 2024 року, Київ, Україна.

2. В. В. Тимофєєнко, С. М. Невмержицька // Управління розвитком підприємств в умовах динамічної ринкової кон'юнктури: матеріали Міжнародної науково-практичної конференції, м. Київ, 27 грудня 2019 року. - Київ : ТОВ "ВІПО", 2019. - С. 257-260.

3. M. Krasnyuk, Yu. Kulynych, I. Hrashchenko, S. Krasniuk, S. Goncharenko, T. Chernysh (2023) Innovative management information system in post-crisis economic conditions on emerging markets (on the example of the oil and gas industry) // Modern: aspekty vedy - Modern aspects of science : svazek XXXVII mezinarodni kolektivni monografie. - Ceska republika : Mezinarodni Ekonomicky Institut s.r.o., 2023. - Р. 185-203.

4. Maxim Krasnyuk, Iryna Hrashchenko, Svitlana Goncharenko, Svitlana Krasniuk, Yurii Kulynych (2023) Intelligent management of an innovative oil and gas producing company under conditions of the modern system crisis. ACCESS Journal: Access Press Publishing House, Volume 4, Issue 3: 352-374.

5. Микитенко, В. В., & Грищенко, І. С. (2008). Адаптивна система управління інноваційними процесами на підприємствах. Проблеми науки, (4), 32-37.

6. Krasnyuk, M., Kulynych, Y., Tuhaienko, V., & Krasniuk, S. (2022) E-business and ecommerce technologies as an important factor for economic efficiency and stability in the modern conditions of the digital economy (on the example of oil and gas company). Grail of Science. - 2022. - №. 17. -69-81.

7. Невмержицька С. М. Формування стратегії інноваційного розвитку підприємств в умовах невизначеності [Текст] / С.М. Невмержицька, Я.В. Левчук // Науковий вісник Херсонського державного ун-ту. Серія «Економічні науки». - Випуск 32/2018. - С. 99103.

8. Дербенцев, В. Д., В. М. Соловйов, and О. В. Сердюк Передвісники критичних явищ в складних економічних системах // Моделирование нелинейной динамики экономических систем. - Донецк: ДонНУ 1 (2005): 5-13.

9. Krasnyuk M.T., Hrashchenko I.S., Kustarovskiy O.D., Krasniuk S.O. (2018) Methodology of effective application of Big Data and Data Mining technologies as an important anti-crisis component of the complex policy of logistic business optimization. Economies' Horizons. No. 3(6). pp. 121-136.

10. Krasnyuk, M., Krasniuk, S. (2021) Association rules in finance management. ЛОГОС , 2021. 9-10 https://doi.org/10.36074/logos-26.02.2021 .v1.01

11. Krasnyuk, M., & Krasniuk, S. (2020). Comparative characteristics of machine learning for predicative financial modelling. Збірник наукових праць ЛОГОС, 55-57.

12. Krasnyuk, M., & Krasniuk, S. (2021). Modern practice of machine learning in the aviation transport industry. Збірник наукових праць ЛОГОІ.

13. Krasnyuk, M., Tkalenko, A., & Krasniuk, S. (2021). Results of analysis of machine learning practice for training effective model of bankruptcy forecasting in emerging markets. Збірник наукових праць ЛОГОІ.

14. Ситник В.Ф., Краснюк М.Т. (2002) Політика управліня знаннями нафтогазової компанії як ключовий фактор підвищення її ефективності. Проблеми формування ринкової економіки: Міжвідомчий науковий збірник. Заснов. у 1992 р. Вип. 10 / Відп. ред. О.О. Беляев. - К.: КНЕУ, 2002. - 326 с.

15. V. Tuhaienko, S. Krasniuk Effective application of knowledge management in current crisis conditions // International scientific journal Grail of Science, № 16 June, 2022. - P. 348-358.

16. Krasnyuk, M. T. (2006), “Problems of applying corporate knowledge management systems and their taxonomy” // Modeliuvannia ta informatsiini systemy v ekonomitsi, vol. 73, p. 256

17. Y. Kulynych, M. Krasnyuk and S. Krasniuk. (2022) Knowledge discovery and data mining of structured and unstructured business data: problems and prospects of implementation and adaptation in crisis conditions. Grail of Science, (12-13), 63-70.

18. Tuhaienko, V., & Krasniuk, S. (2022) Effective application of knowledge management in current crisis conditions. Grail of Science, (16), 348-358

19. Krasnyuk, M. T. (2006) Problems of applying corporate knowledge management systems and their taxonomy. Modeliuvannia ta informatsiini systemy v ekonomitsi, vol. 73, p. 256

20. Hrashchenko I. and Krasniuk S. (2015) Problems of regional development of Ukraine under globaliation process. Visnyk Mizhnarodnoho humanitarnoho universytetu. Seriia: Ekonomika i menedzhment, 2015. - №11. - p. 26-32.

21. Hrashchenko Iryna, Krasnyuk Maxim, Krasniuk Svitlana (2020) Iterative methodology of bankruptcy forecast of logistic companies in emerging markets, taking into account global/regional crisis. Збірник наукових праць «Проблеми системного підходу в економіці», Випуск 1 (75) / 2020. - C. 138-147.

22. Kulynych Y., Krasnyuk M., Krasniuk S. (2022) Efficiency of evolutionary algorithms in solving optimization problems on the example of the fintech industry. Grail of Science, №14-15, May 2022. 63-70.

23. Краснюк, М. Т. (2014) Гібридизація інтелектуальних методів аналізу бізнесових даних (режим виявлення аномалій) як складовий інструмент корпоративного аудиту // Стан і перспективи розвитку обліково-інформаційної системи в Україні: матеріали ІІІ Міжнар. наук.-практ. конф. [м. Тернопіль, 10-11 жовт. 2014 р.] / редкол. : З. В. Задорожний, В. А. Дерій, М. Р. Лучко [та ін.] ; гол. ред. З. В. Задорожний. - Тернопіль : ТНЕУ, 2014. - С. 211 -212.

24. Krasnyuk, M., Hrashchenko, I., Goncharenko, S., Krasniuk, S. (2022) Hybrid application of decision trees, fuzzy logic and production rules for supporting investment decision making (on the example of an oil and gas producing company). Access to science, business, innovation in digital economy, ACCESS Press, 3(3): 278-291.

25. Гращенко І.С., Краснюк М.Т., Краснюк С.О. (2019) Гібридно-сценарне застосування інтелектуальних, орієнтованих на знання технологій, як важливий антикризовий інструмент логістичних компаній в Україні. Вчені записки Таврійського Національного Університету імені В. І. Вернадського. Серія: Економіка і управління. Том 30 (69). С. 121 - 129.

26. M. Krasnyuk, S. Goncharenko, S. (2022) Krasniuk M. Intelligent technologies in hybrid corporate DSS (on the example of Ukraine oil&gas production company) // Інноваційно- інвестиційний механізм забезпечення конкурентоспроможності країни: колективна монографія І за заг. ред. О. Л. Гальцової. - Львів-Торунь : Ліга-Прес, 2022. - С. 194211.

27. Krasnyuk M., Kulynych Yu., Tkalenko A., Krasniuk S. (2021). Methodology of Effective Application of Economic-Mathematical Modeling as the Key Component of the MultiCrisis Adaptive Management. Modern Economics, 29(2021), 100-106.

28. Краснюк, М. Т., & Краснюк, С. О. (2020). Fraud detection in the business data as an important corporate anti-crisis method of audit. Сучасні виклики і актуальні проблеми науки, освіти та виробництва: міжгалузеві диспути [зб. наук. пр.]: матеріали III міжнародної науково-практичної інтернет-конференції (Київ, 15 квітня 2020 р.). Київ, 2020. - С. 14-16.

Размещено на Allbest.ru