Big data analysis influence on public administration processes

UK

Statistics, economy

“The Îffice îf Nātiînāl Stātistics is interested in the pîtentiāl îf Big Dātā tî cîmplement, āugment îr replāce cînventiînāl dātā sîurces tî imprîve the quālity îf îfficiāl stātistics. Fîr instānce, ÎNS hās used Gîîgle trends ānālysis tî prîvide eārly wārning îf migrātiîn trends (e.g. Time trends in the vîlume îf Gîîgle seārches fîr Pîlish terms cîrrelātes well with trends in the number îf Pîlish migrānts tî the UK āfter the EU8 cîuntries jîined Eurîpe in 2006-08)” (Europe, 2015).

UK

Open data,

public services

“The UK gîvernment is ā wîrld leāder in îpen dātā ānd since 2010 hās releāsed îver 10,000 dātāsets thrîugh its dātā.gîv.uk site. The gîvernment is ālsî creāting ā Nātiînāl Infîrmātiîn Infrāstructure, which will cîntāin the dātā held by gîvernment which is likely tî hāve the brîādest ānd mîst significānt ecînîmic ānd sîciāl impāct if māde āvāilāble ānd āccessible îutside îf gîvernment” (HM Government. Department for Business, 2013).

UK

Healthcare

“McLāren Electrînic Systems āre using their reāl-time dātā system expertise, develîped ās pārt îf their Fîrmulā Îne wîrk, tî help Birminghām Children's Hîspitāl imprîve the mînitîring îf children in intensive cāre. The prîject, which stārted in 2010, enābles the hîspitāl tî benefit frîm the ādvānced telemetry cîmmînly used in Fîrmulā Îne tî āssess ā cār's perfîrmānce whilst în the trāck. In 2011, the reāl-time dātā system, which is used by every teām in Fîrmulā Îne, wās instālled întî the hîspitāl's cîmputer netwîrk. This āllîws physiîlîgicāl dātā tî be streāmed live frîm bedside mînitîrs in intensive cāre ānd frîm the speciālist child trānspîrt āmbulānce, which cān then be prîcessed in reāl-time tî understānd the shifting pātterns thāt cān tell medicāl stāff when ā pātient's cînditiîn is chānging” (HM Government. Department for Business, 2013).

UK

Labor market, public services

“The Nuffield Fîundātiîn, the Ecînîmic ānd Sîciāl Reseārch Cîuncil (ESRC) ānd the Higher Educātiîn Funding Cîuncil fîr Englānd (HEFCE) hāve lāunched Q-Step, ā Ŗ19.5 milliîn initiātive designed tî prîmîte ā step-chānge in quāntitātive sîciāl science trāining. Îver ā five yeār periîd frîm 2013, 15 universities ācrîss the UK āre delivering speciālist undergrāduāte prîgrāms, including new cîurses, wîrk plācements ānd pāthwāys tî pîstgrāduāte study. Expertise ānd resîurces will be shāred ācrîss the higher educātiîn sectîr thrîugh ān āccîmpānying suppîrt prîgrām, which will ālsî fîrge links with schîîls ānd emplîyers. Q-Step wās develîped in respînse tî the shîrtāge îf quāntitātively-skilled sîciāl science grāduātes, which hād led tî emplîyers ācrîss āll sectîrs being unāble tî recruit peîple with the skills tî āpply quāntitātive methîds when evāluāting evidence ānd ānālysing dātā” (GOVERNMENT, 2014).

Germany

Labor market,

Public services

“Bundesāgentur für Ārbeit (Germān Federāl Lābîur Āgency) ānālysed its huge āmîunt îf histîricāl dātā în its custîmers, including histîries îf unemplîyed wîrkers, the interventiîns thāt it tîîk, ānd îutcîmes including dātā în hîw lîng it tîîk peîple tî find ā jîb. The ideā wās tî develîp ā segmentātiîn bāsed în this ānālysis sî thāt the āgency cîuld tāilîr its interventiîns fîr unemplîyed wîrkers. This prîcess, ālîng with îther initiātives āpplied îver three yeārs, āllîwed the āgency tî reduce its spending by ˆ10 billiîn ānnuālly āt the sāme time ās cutting the āmîunt îf time thāt unemplîyed wîrkers tîîk tî find emplîyment, ānd increāsing the sātisfāctiîn āmîng users îf its services” (GOVERNMENT, 2014).

China

Statistic

“Nātiînāl Bureāu îf Stātistics îf Chinā lāunched the prîject Big Dātā Enterprise Stātisticāl Indicātîr Ten-dāy Repîrt. The prîject is māinly fîr reseārch purpîse, îffering references fîr reseārches in ecînîmic situātiîn ānd stātisticāl prîductiîn” (Europe, 2015).

Singapore

Banking,

economy

“Ās ā vehicle fîr GDP grîwth, Singāpîre hās sîught in the pāst tî estāblish itself ās the Bānking hub, medicāl hub ānd ā “hub” îf Sîutheāst Āsiā in severāl îther āreās. Ālîng thāt line îf thîught, the Infîcîmm Develîpment Āuthîrity hās ā wîrking grîup tāsked with estāblishing Singāpîre ās ān Internātiînāl Dātā & Ānālytics Hub” (Europe, 2015).

Singapore

Healthcare

“In 2004, tî āddress nātiînāl security, infectiîus diseāses, ānd îther nātiînāl cîncerns, the Singāpîre gîvernment lāunched the Risk Āssessment ānd Hîrizîn Scānning (RĀHS) prîgrām within the Nātiînāl Security Cîîrdinātiîn Centre. Cîllecting ānd ānālyzing lārge-scāle dātāsets, it prîāctively mānāges nātiînāl threāts, including terrîrist āttācks, infectiîus diseāses, ānd finānciāl crises. The RĀHS Experimentātiîn Center (REC), which îpened in 2007, fîcuses în new technîlîgicāl tîîls tî suppîrt pîlicy māking fîr RĀHS ānd enhānce ānd māintāin RĀHS thrîugh systemātic upgrādes îf the big-dātā infrāstructure. Ā nîtāble REC āpplicātiîn is explîrātiîn îf pîssible scenāriîs invîlving impîrtātiîn îf āviān influenzā intî Singāpîre ānd āssessment îf the threāt îf îutbreāks îccurring thrîughîut sîutheāst Āsiā” (Gang-Hoon Kim, 2014).

Singapore

Open data,

public services

“Āiming tî creāte vālue thrîugh big dātā reseārch, ānālysis, ānd āpplicātiîns, the Singāpîre gîvernment ālsî lāunched the pîrtāl site http://dātā.gîv.sg/ tî prîvide āccess tî publicly āvāilāble gîvernment dātā gāthered frîm mîre thān 5,000 dātāsets frîm 50 ministries ānd āgencies” (Gang-Hoon Kim, 2014).

South Korea

Strategy, public services

“The Big Dātā Initiātive, lāunched in 2011 by the President's Cîuncil în Nātiînāl ICT Strātegies (the highest-level cîîrdināting bîdy fîr gîvernment ICT pîlicy), 16 āims tî cînverge knîwledge ānd ādministrātive ānālytics thrîugh big dātā. Its Big Dātā Tāsk Fîrce wās creāted tî plāy the leād rîle in building the necessāry infrāstructure. The Big Dātā Initiātive āims tî estāblish pān-gîvernment big dātā-netwîrk-ānd-ānālysis systems; prîmîte dātā cînvergence between the gîvernment ānd the privāte sectîrs; build ā public dātā-diāgnîsis system; prîduce ānd trāin tālented prîfessiînāls; guārāntee privācy ānd security îf persînāl infîrmātiîn ānd imprîve relevānt lāws; develîp big dātā infrāstructure technîlîgies; ānd develîp big-dātā mānāgement ānd ānālyticāl technîlîgies” (Gang-Hoon Kim, 2014).

South Korea

Healthcare

“The Ministry îf Heālth ānd Welfāre initiāted the Sîciāl Welfāre Integrāted Mānāgement Netwîrk tî ānālyze 385 different types îf public dātā frîm 35 āgencies, cîmprehensively mānāging welfāre benefits ānd services prîvided by the centrāl gîvernment, ās well ās by lîcāl gîvernments, tî deserving recipients” (Gang-Hoon Kim, 2014).

South Korea

Healthcare

“The Ministry îf Fîîd, Āgriculture, Fîrestry, ānd Fisheries ānd the Ministry îf Public Ādministrātiîn ānd Security, îr MÎPĀS, plān tî lāunch the Preventing Fîît ānd Mîuth Diseāse Syndrîme system, hārnessing big dātā relāted tî ānimāl diseāse îverseās, custîms/immigrātiîn recîrds, breeding-fārm surveys, livestîck migrātiîn, ānd wîrkers in the livestîck industry. Ānîther system MÎPĀS is plānning is the Preventing Disāsters System tî fîrecāst disāsters bāsed în pāst dāmāge recîrds ānd āutîmātic ānd reāl-time fîrecāsts îf weāther ānd/îr seismic cînditiîns. Mîreîver, the Kîreān Biîinfîrmātiîn Center plāns tî develîp ānd îperāte the Nātiînāl DNĀ Mānāgement System tî integrāte māssive DNĀ ānd medicāl pātient infîrmātiîn tî prîvide custîmized diāgnîsis ānd medicāl treātment tî individuāls” (Gang-Hoon Kim, 2014).

South Korea

Smart city,

Public services

“The gîvernment îf the Sîuth Kîreān cāpitāl hās been using big dātā bāsed în cāll vîlume tî îptimize lāte night bus rîutes tî help thîusānds îf cîmmuters get hîme. Similār smārt-city initiātives ācrîss Āsiā āre imprîving the lives îf citizens. (ĀFP) Mînitîring Jākārtā's multibilliîn-dîllār flîîd prîblem with the help îf Twitter. Îptimizing lāte night bus rîutes in Seîul bāsed în cāll vîlumes tî help îver 7,000 cîmmuters get hîme sāfely. Emplîying ānālytics tî māximize rîād use ānd the ālternātive rîutes tāken during peāk hîurs in Hîng Kîng” (Chinadailyasia, 2015).

Mexico

Statistic,

Tweet analyse

“Since tweets āre publicly āvāilāble, since ā number îf ācādemic prîjects āre lîîking intî hîw tî use them, ānd since ā cîntinuîus 1% geîreferenced sāmple cān be îbtāined fîr free, we decided tî use this dātā sîurce in îrder tî get first-hānd experience regārding technîlîgicāl ānd methîdîlîgicāl requirements, including humān ānd māteriāl resîurces, ānd their cāpābilities, fîr tāckling Big Dātā prîjects fîr the prîductiîn îf îfficiāl stātistics. Āll îf the ābîve while āssessing tweet's usefulness in three specific āpplicātiîn āreās: subjective well-being, tîurism ānd bîrder mîbility. Fîr the first we pārtnered with ā university tî get students tî clāssify cîntent. Fîr the îther twî, cîunting chānge îf lîcātiîn rāther thān cîntent is the issue” (Europe, 2015).

Chapter Summary

This Chapter illustrates how restrictions on cross-border data flows affect governments and businesses in a wide variety of industries and regions of the world. This Chapter provides a series of case studies to highlight companies or public agencies that are taking advantage of cross-border data flows to streamline their operations, improve their products and services, and provide consumer benefits.

Reviewing big data projects and big data collaborations, I identify that commercials and government partnerships have the biggest part while NGO and international partnerships are not so significant. I can make a conclusion that it is time for the international community to recognize the importance of cross-border data flows. Only by creating special frameworks on these issues, can countries encourage economic development in both information and traditional industries, as well as hold each other accountable in the future.

Chapter 3

Big Data Processing Maturity Model

This Chapter describes The Big Data Processing Maturity Model. First part of the Chapter shows the overview of big data maturity level of business sector in Russian Federation and factors, which influence on it. Second part of the Chapter is devoted for The Big Data Processing Maturity Model creation and verification:

The Big Data Processing Maturity Model will be designed based on results were received during my two years research in this field. In addition, for the Big Data Processing Maturity Model design I use contributions and conclusions received in previous chapters of my Master Thesis.

The Big Data Processing Maturity Model will be verified on the next countries:

Russian Federation

USA

Germany

Singapore

Nigeria

Big data maturity level of business sector overview in Russian federation

To understand the conception of government development it could be usefull to analyze the business sector in context of big data experience. I make the focus on Russian Federation here and try to show on Russian Federation example that maturity level of businecc sector and public sector in big data field are quite correlate.

38% of respondents said that their organization already using big data. The study was conducted among the interested companies - real and potential customers more data, so the resulting figure is not so great (Smirnov, 2016).

“At the end of 2015, Computerworld Russia, with the support of IBM conducted a study "Business in the era of Big Data" in order to assess the level of maturity of the domestic companies, to analyze the prospects for further penetration of big data technologies in the Russian business, as well as to identify the main obstacles encountered along the way. The survey polled 226 people attended, mostly CIOs and IT managers, representatives of research and business communities. Methodical support of the project was carried out experts OSP Data, Analysis Group publishing house "Open Systems"” (Smirnov, 2016).

The greatest number of organizations in which big data is already used, working in the sphere of science and education, finance and insurance, telecommunications, oil and gas industry and the energy industry, as well as in trade and transport. The penetration rate of big data technologies in other industries as a whole is low.

Although the principal obstacles to the development of big data technologies not, most organizations still do not hurry up with their implementation. More than half of those who have not yet used, but suggests to use them in the future, planning to start projects of big data is not earlier than in two years. The reason, it seems, lies in the fact that these technologies are not everywhere come to the attention of top management.

Decision on implementation of projects in the area of ??big data for the most part remains a zone of responsibility of the CIO. Nevertheless, in some cases, the decision is made jointly. For example, the initiators of the projects, along with the head of the IT department, may make heads of marketing services, finance department, or sales.

“What is sad, in the formulation of strategic decisions in the area of ??Big Data top executives are involved only in 23% of organizations. This means that more data will not become part of the strategy, is not considered a priority in a large number of companies” (Smirnov, 2016).

As it turned out, the practical application of big data trends are quite diverse. The most common area were marketing and sales. The difficult economic situation is not only the most obvious in terms of the scope of the impact, but also one of the hottest topics for all sectors of activity. The effectiveness of tools big data in this important business area is the least of doubt. Most often, companies from the group analyzed data on sales, engaged in analytics data CRM and ERP systems and studying customer loyalty. Slightly less it is an analyst client requests and preferences. These social networking analyzes a third of respondents from this group, which is a fairly high rate, given the wide scope of industry research (Smirnov, 2016).

Almost half of the users said that big data used in the management of the organization as a whole. This is a very encouraging indicator from the point of view of the prospects for the development of technologies in these organizations as tools that have proven effective in the management level, with high probability will be used at other levels.

Companies reporting the use of big data in logistics, some less. They help solve problems turnover control and risk management in the supply chain, including analysis of the impact of weather conditions.

In the current difficult situation, the majority of Russian companies coming to the projects for big data analytics is extremely pragmatic. Aimless accumulation of data is not interesting, everyone wants to earn profits from their information. This yield criterion recoupment is now crucial. Even if the project brought an interesting, unexpected results (the so-called wow-effect), but did not prove their economic viability, it will be closed.

The Big data Processing maturity Model - Design and Verification

Gîvernments thāt cān mānāge big dātā chānge effectively āre generālly mîre successful thān thîse thāt cānnît. Māny gîvernments knîw thāt they need tî imprîve their big dātā-relāted develîpment prîcesses in îrder tî successfully mānāge chānge, but dî nît knîw hîw.

The Big Dātā Prîcessing Māturity Mîdel āddresses this prîblem by prîviding ān effective methîd fîr ā gîvernment tî grāduālly gāin cîntrîl îver ānd imprîve its big dātā -relāted develîpment prîcesses. My mîdel prîvides the fîllîwing benefits:

It describes the prāctices thāt āny gîvernment must take into consideration in îrder tî imprîve its level îf big dātā processing level

It prîvides ā scāle îf meāsurement

The Big Dātā Prîcessing Māturity Mîdel wās designed bāsed în the fîllîwing guiding principles:

Tî hāve strîng ācādemic bāckgrîund

Tî use insights frîm ICT frāmewîrks ānd best prāctices

Bāsed în reliāble dātā

Bāsed în cîmprehensive ānālysis

Tî use vālid dimensiîns āpplying îppîrtunity îf retrîspective ānālysis

The Big Data Processing Maturity Model is designed as profiles of a big data implementation level in countries that governments would recognize as descriptions of possible current and future states. The Big Data Processing Maturity Model has six dimension:

Vision and strategy - this dimension shows how government integrates big data into IT strategy and answers on the question: does country have clear and strong vision about big data issue?

Open Data initiatives - Open Data is an important indicator of big data processing readiness for a country. Open data act as one of the main resources for big data processing in social context.

R&D institutions and initiatives - to get value form big data processing government has to invest in specialized education and programs to grow up professional competences in this field. Type and quality of educational programs show us ambitions and interest of a country in big data implementation.

Big data maturity level in business sector - business sector can act as an indicator of public sector loyalty in big data. Therefore, if business sector is not ready for big data it could mean that public sector has no foundation for big data implementation.

Data Governance - data governance is a foundation approach to build future big data landscape. Data governance is an indicator of how works internal IT processes in public agencies. What is more important how are they cooperate? This point include geographical and architectural factors (centralized or decentralized architecture landscape in public agencies and approach for collecting and processing distribute data streams).

Big data projects experience in public sector - current state of big data projects implementation is a valuable index of big data initiatives in a country.

Each dimension can be measured in four categories:

Aware

Exploring

Optimizing

Transforming

Four categories are the measurement system to define dimensions quality level. Table 5 describes this approach.

Table 5 - Big Data Processing Maturity Model in government and public agencies

	Aware	Exploring	Optimizing	Transforming
Vision and strategy	Big data is discussed but not reflected in government strategy. Government strategy use of data extends simply to financial and regulatory reporting or simply data analytics in.	Government has IT (ICT) strategy in whole. Big data vision and strategy is not clearly defined or not defined at all. Big data application is largely experimental and is not clearly defined in strategies.	Existence of short-term and middle term IT strategies and existence of insights from big data application in public administrations. Big data vision and strategy is not clearly defined or defined partly.	Existence of clear vision for a long-term period of using big data. Publication of regularly reports by governmental institutions encourages the use of insight from big data processing within government processes. Vision and strategy are developing based on the experience and lessons learned already available.
Open Data initiatives	There are no government Open Data initiatives in a country.	There are some Open Data portals but they do not have significant values. Quality and credibility of data are under concern. Data updates are not regularly or data are not relevant. Municipal Open Data portals are not presented or presented in small quantities.	Variety of Open Data portals. Municipal Open Data portals are essential. However, they are not cover all public agencies or not aggregated into one integrally hub. There is no single access point for federal and municipal (regional) data. There are gaps in data consistency and homogeneity.	Variety of Open Data portals. Municipal Open Data portals are essential. Share research results with the public. There is single access point for federal and municipal (regional) data. Open Data portals cover all sectors of life. Information is used as a strategic asset.
R&D institutions and initiatives	Science background is not developed. There are no big data educational programs and courses in public or private universities and schools. There are no research and development initiatives in public institutes or research centers.	Science background is developed only slightly. There are several commercial big data educational programs and courses in private universities or schools. There are no research and development initiatives in public institutes or research centers.	Science background is developed. There are various big data educational programs and courses in public and private universities or schools. There are several research and development initiatives in public institutes or research centers.	Science background is developed. There are various big data educational programs and courses in public and private universities or schools. There is big variety of research and development initiatives in public institutes or research centers. Government acts as sponsorships of educational programs, conferences, competitions, grants.
Big data maturity level in business sector	Majority of local companies form Fortune 100 (local rating) have no big data capabilities.	Just several local big players-companies have big data capabilities. Business understands the overall big picture from all available data. Big data providers are mostly overseas companies.	High level of big data implementations in the following fields: banks, retail, oil and gas, telecommunications. Approximately half of local big players-companies have big data capabilities.	High level of big data implementations in all fields. Majority of local big players-companies have big data capabilities. Business uses big data to predict outcomes or to adjust processes accordingly.
Data Governance	There is no clear data ownership assigned. No standards tools nor documentation is available for use data across the whole public sector. Data governance is largely manual.	A government data governance model does not exist or is immature; data owners are commissioned for short-term projects and initiatives. Understanding of data and its ownership are defined and managed in a piecemeal fashion. Limited collaboration.	Data governance model is implemented for the major data entities. Collaboration between stakeholders is in place. Governance process regularly reviews this model and its applications; updating and improving as needed. Government begins to realize benefits of enterprise-wide consistency of data.	A government-wide data governance model extends active stewardship to the majority of data assets; effective data governance processes are employed by stakeholders and stewards; well-defined standards are adopted.
Big data projects experience in public sector	There is no big data projects implementations in public agencies.	There are only several big data projects implementations in public agencies.	There are variety of big data projects implementation but in specific fields: tax or financial.	There are variety of big data projects are implemented in different fields.

To define the maturity level of a country we need to make mapping of scores for each category. Score system is described below:

Category	Score
Aware	1
Exploring	2
Optimizing	3
Transforming	4

Table 6 provides resulting information about maturity level of a country. Total value helps range countries level of big data maturity in four categories: Aware, Exploring, Optimizing and Transforming. The paper provides maturity level identification for four countries:

Russian Federation

USA

Germany

Singapore

Nigeria

Below you find the Big Data Maturity Score Map for each countries from list above in Table 7.

Table 8 consists results and conclusions.

Table 6 - The Big Data Maturity Level Map

Total value	Maturity level
0-6	Aware
7-12	Exploring
13-18	Optimizing
19-24	Transforming

Table 7 - The Big Data Maturity Score Map for Russian Federation, USA, Germany, Singapore, Nigeria

Dimension \ Country	Russian Federation	USA	Germany	Singapore	Nigeria
Vision and strategy	2	4	3	4	1
Open Data initiatives	3	3	3	4	2
R&D institutions and initiatives	3	4	4	4	1
Big Data maturity level in business sector	3	4	4	4	2
Data Governance	1	4	3	4	2
Big data projects experience in public sector	2	4	4	4	1
Total score	14	23	21	24	9



Table 8 - Results and conclusions

#	Country	Total Score	Maturity Level	Conclusion
1	Russian Federation	14	Optimizing	Russian Federation is at the early stage of big data development. Open Data initiatives, R&D institutions and initiatives and Big data maturity level in business sector have the highest scores in the model. While Vision and Strategy and Data Governance have the lowest score. There are few reasons for such a kind of scores desperation. First Russian Federation is a huge country. Cities are distributed. This fact influences on current network readiness landscape. It is noteworthy that governmental IT budget is not so big. Moreover, country is in crisis stage of development. That is why IT is not prior strategic direction for the government.
2	USA	23	Transforming	USA is a leader in big data development and implementation. Almost all categories have maximum score. USA have a strong leader positions in R&D in big data. Moreover, White House publish variety of White Papers regularly about big data, social values, big data challenges and practical cases or success stories about big data projects implementations in public agencies. USA invest huge money in IT budgets of the country.
3	Germany	21	Transforming	Germany act as European leader in big data field. There are significant numbers of big data implementations. Moreover, big data projects collaboration with other European countries is high developed. There are variety number of European big data collaboration projects: from refugees program to FUI financial system. R&D institutions and initiatives, Big data maturity level in business sector and Big data projects experience in public sector have the highest scores in the model.
4	Singapore	24	Transforming	Singapore is a visionary in big data development and implementation. All categories have maximum score. Singapore have a strong leader positions in R&D in big data and ultimate clear and applied Vision and Strategy in big data development of country. Moreover, Singapore government publish variety of reports about big data. Singapore also invest a lot of money in IT budgets of country. Success also can be described in next points: Singapore is a young and very ambitious country with very concentrate (centralize) geographical location. In addition, Singapore acts as business and professional hub in the world. This points give more clarify in reasons of Singapore success.
5	Nigeria	9	Exploring	Nigeria is on a starting position for big data processing and implementation. Almost all categories have minimum score. Vision and strategy, R&D institutions and initiatives and Big data projects experience in public sector have the lowest scores in the model. First steps had already been made: big data is discussed on government level and there is Open Data governmental portal.

Chapter Summary

Based on literature review: governments official ICT strategies, White Papers, Open Data portals, top advisory companies reports, articles and papers of public institutes and research centers, including here previous results were getting from my study I designed The Big Data Processing Maturity Model for governments and public sector in this Chapter. Using the Model I can compare in the correct unified way different countries using different dimensions what give me objective and comprehensive analysis. This Chapter provides next results:

The Big Data Processing Maturity Model was designed based on results of the study took place before in Master Thesis.

The Big Data Processing Maturity Model was verified on the next countries:

Russian Federation

USA

Germany

Singapore

Nigeria

For verification reason were selected orthogonal countries with different economy and social development level. Results were made using The Big Data Processing Maturity Model correlate with the economy and social level of the country.

Big data processing maturity levels of countries mentioned above correlate with economical level of development of the country.



Chapter 4

Chapter 4 is placed to illuminate big data challenges on a way of big data penetration in public sector. High level challenges can be classified on 3 groups:

Economical challenges

Social challenges

Technical challenges

However, this challenges classification is not detailed and not answer on the question about challenges and gaps in public sector. Previous results of my study give me enough insights and conclusions about more specified reasons of big data obstacles and challenges, which take place in public sector in different countries. Goal of this Chapter is to provide deep analysis of roots and reasons of challenges appeared on the way of big data penetration in public sector in different countries.

Big Data Challenges

Today only analyzed 0.5% accumulated digital data, despite the fact that there are objectively industry-wide problem, which could be solved by means of analytical solutions class big data. Development of IT markets already have the results of which we can estimate the expectations associated with the accumulation and processing of big data (Gang-Hoon Kim, 2014).

One major factor that hinders the implementation of big data projects, in addition to the high cost, it is considered the problem of selecting data to be processed that is, determining what data needs to be retrieve, store, and analyze, and what - is not taken into account.

Many business executives say that the complexity of the introduction of big data projects related to the lack of specialists - marketers and analysts. From the quality of the staff involved in deep and predicative analytics directly affects the speed of return on investment in big data. The marketers because of outdated business processes or internal policies cannot effectively use the enormous potential existing in the organization of the data often. So often big data projects are perceived as complex business not only in the implementation but also in evaluation: the value of the data collected. The specifics of working with data requires marketers and analysts switching focus from technology and create reports to address specific business challenges.

Other words almost every single big data project faces with some kind of challenges. Let is make the review of the most common or typical challenges are:

Difference in data formats and quality.

Multiple external and internal sources create data in different formats. Identifying authentic data is a big challenge in the world of big data.

Expensive infrastructure.

Investment in new tools, systems and bandwidth is a part of embracing big data. Additional storage is needed for these massive amounts of data.

Redundancy and performance.

Processing large amounts of data in real-time slows down existing systems. Many new processes have to be introduced and this is likely to lead to redundancy.

Integration with existing data.

Integration with existing data for meaningful analysis, some kind of techniques like Hadoop and NoSQL are incompatible with existing relational databases.

Public agencies and departments all types and sizes are looking to big data to help them make faster and more intelligent all inside and outside business processes. Many efforts involve the collection, storage, and deep analysis of new data sets. However, most IT landscapes have shortcomings when it comes to big data. IT landscapes of most public agencies are not suited to handle the volume of data. In addition, most of them cannot efficiently support the variable and data mining workloads (Report, 2014).

In the life of public sector, the speed of decision-making is critical. Government employee need to quickly analyze and evaluate the results of many datasets processing to make any decision.

Like big data efforts in other sphere, public sector activities in common relies on automated, multistep analysis workflows. Such workflows place high-performance demands on storage system, and network elements.

What is required is an infrastructure that can support diverse workflows. Most important, the infrastructure must be able to handle large sorts, which are common in big data workflows. Sometime the files could be larger than system memory and therefore cannot be retained in local cache. As a result, large sorting workloads require a file system and storage solution that can provide high throughput. Another requirement is low latency access to file system metadata form the storage system (CNEWS, 2014).

Any big data project needs to increase storage capacity to accommodate the datasets. However, just adding raw capacity without changing technical landscape issues into consideration can lead to complexity and inefficient use of resources.

Current trend of IT infrastructure is Interdependencies. Big data processing assumes interdependency of infrastructures elements also. Public agencies and departments are turning to virtualization trend to reduce operating cost, consolidate servers and make easier the deployment and management of application.

According to network approach, virtualization trend and big data processing can change the dynamics of traffic flow within a data center network. Networks links can become stronger and impacting network performance. It becomes obvious that it is needed the network with high-performance scalability (Charette, Spectrum.ieee.org. September 2, 2005.).

Another feature that has become actual in frame of big data processing is Unpredictable workloads. Efforts to derive decision-making information from big datasets typically use a number of analysis tools applied at different stages of a computational workflow. We need to place here the additional requirement for infrastructure could support diverse workflows -high-sustained throughputs. Also the infrastructure must be able to handle large sorts, which are quite common in big data workflows (Charette, Spectrum.ieee.org. September 2, 2005.).

Other important approach is Data management. External and internal processes in public agencies and departments has become more multidisciplinary and more collaborative. This complicates habitual data management and makes computational workflows more complex.

An additional implication of the collaborative and multitask behavior is that data increasingly must be shared. From a storage perspective, older data must be moved to lower-cost storage after its initial analysis. The suitable solution should have a robust file system that is capable of managing the explosive datasets as a single volume. Moreover, an intelligent solution will help agencies and departments manage, share, and protect data (Charette, Spectrum.ieee.org. September 2, 2005.).

Move further from technology landscape aspects to information environment we have to get deeper understanding what changes do we have in the level of business processes. To understand workflow management for big data processing, we have to know what a process is and how it relates to the workflow in high load environments. Processes tend to be designed as high level useful for decision making and stabilization how things are done in a public agency or department.

The best practice for understanding workflow management for big data processing is to do the following steps (J., 2012):

Identify the big data sources.

Make a mapping: big data types to workflow data types.

Ensure that you have the processing speed and storage access to support your workflow.

Select the data storage best suited to the data types.

Modify the existing workflow to accommodate big data or create new big data workflow.

Designated steps above helps executives of public agency or department to keep correct direction on the way of big data processing implementation.

To make the results of analysis of workflow management for big data processing more complete I would like to provide some real business cases and feedbacks from traditional big data users in Russian Federation - banks and financial institutions.

According to CNews Analytics, at the beginning of 2014 17 of the 30 largest in terms of net assets of Russian banks (including Sberbank, Gazprombank, VTB 24, Raiffeisenbank) used or were planning to implement the technology big data (CNEWS, 2014).

Big data allows you to analyze vast amounts of information, and, if necessary, and all the data available in real time. In addition, big data technology makes it possible to work with unstructured data - logs with a completely different systems, web-content, photos, audio and video information. Traditional solutions based on relational database, designed to handle tabular data and are poorly suited to modern needs.

The first key feature of big data is the ability to handle all the data accumulated in the information systems of the bank.

With Hadoop, Visa has reduced processing time 73 billion transactions from one month to 13 minutes. Of course, such a change in processing speed leads to significant business change (Clark, 2012).

The second feature - the ability to determine the relationship between events. Big data can detect which event (that), and when there is in the system. Bank employees may not know the cause of the event - a causal link can be reconstructed from the results of the analysis carried out with the help of technology big data. This means that the answer to the question "When the event occurs?" It is possible to understand its causes. For example, new fraud schemes can be identified on the basis of atypical behavior of customers of the banking system.

The Russian market of big data is gaining momentum, but foreign companies have already appreciated the opportunity. For example, the credit agency Experian offers a product Income Insight, predicts that the level of income a person based on their credit history. This approach allows us to avoid asking for information on the income of the client, with the cost analysis is about ten times lower than the traditional method.

This example still is a single case of application of big data, and it is unlikely that the domestic market in the coming years will be something like that. But a behavioral analysis of the actions of customers to personalize products, marketing campaigns, improve the quality of service in the near future will be widely distributed. For example, it is already very popular with the calculation of the bank uses the cache to replenish ATMs using technology big data. Based on the analysis of demand for cash among consumers is calculated for each ATM replenishment time and the amount of cash that can reduce the number of collection and downtime of money from an ATM, and ultimately makes it possible to increase the number of customers and increase their satisfaction with the services of the bank (Maggiore, 2015).

Observing the rapid development of big data technologies in the Europe and U.S., I can make a guess that the big data will soon dramatically change processes in banking and public sector as well.

To summarize the Chapter 4 in Table 9 placed the list of requirements and them description that accompanied changes in IT environment for big data processing in public agency or department.

Table 9 - Features of IT environment changes for big data processing

FEATURE	DESCRIPTION
Technical layer
Interdependence of infrastructure elements	Changes focused on integration way of servers, storage and network elements
Unpredictable workloads	IT infrastructure should be able to support diverse workflows, offering high-sustained throughputs
Data management	Older massive of data must be moved to lower-cost storage after its initial analysis. However, when new data analysis is required, the data must be easily located and automatically migrated to higher-performance storage
Distribution computing and remote access	More recently, distributed computing is used to refer to any large collaboration in which many individual personal computer owners allow some of their computer's processing time to be put at the service of a large problem. This requirement is become solicited in big data processing framework.
Information and process layer
Advanced SLA enforcement policies	According to big data processing requirements every single key business process should have an appropriate SLA policy. This requirement makes easier changing of workflow management in current architecture.
Massive scalability	Due to extensional growth of data missives, information architecture should to have some capacity for a further scalability for new workflows
High throughput	Under this requirement should be understood transaction speed, packet loss rate, fail-safe feature and others
Human layer
Employee qualification	Big data processing forces to advance training for employees. Human factor is very important in terms of big data implementation. This process demands employees to be prepared to either a new workflow management system or technical component.
Service oriented	Employees should be focused on service oriented contour of workflow management.
Loyalty	This requirement leads to changes should be unexpected. New technical and information environment in any way influence on habitual order of things in company or public agency. And should be placed proactive steps to the path of employee loyalty

Chapter Summary

Based on literature review: governments official ICT strategies, White Papers, top advisory companies reports, articles and papers of public institutes, including previous results were getting from my study, I conclude that there is the set of challenges and problems public agencies and companies are faced with on the way of big data projects implementation. Obviously, these problems and challenges are common for every innovation implementation.

Besides main obstacle - ICT government budget, that slow down spreading big data penetration in public sector, there are number of common challenges for all countries:

High cost of big data implementation.

Lack of executive understanding of the core value of processing data.

Risk management.

Weak methodological background.

Complexity of data integration and data sharing.

Data security.

Existing workflows and business processes.



Research results

First Chapter of the Master Thesis made the overview of big data projects implementations in public sector of different countries. In this Chapter were made the accent on big data current situation in Russian Federation.

Second Chapter is devoted for comparative analysis of international collaborations of the big data projects partnerships.

Third Chapter describes The Big Data Processing Maturity Model design and verification.

Fourth Chapter gives the big data challenges analysis. The Chapter provides information about what kind of problems face big data penetration process with.

The public sector is emerging as the largest consumer of big data and will benefit greatly from the smart innovations big data delivered by running R&D. Public agencies make their steps to big data processing. Next years all public agencies will be grappling with the deal of how to consolidate and integrate their distribution data, design and enhance cross-border data flows, build analytical capacities, and move toward a “data-driven” concept of decision-making processes.

I expect, the key players in international collaborations in big data research will become research institutions and global organizations such as United Nations, European Union, World Trade Organization, the World Bank and International Monetary Fund.

Public agencies and departments have been traditionally separated by internal rivalries and financial competition developing into an embedded silo mentality and culture. Business sector, public agencies and international institutions should think about big data not as an IT solution to solve reporting and analytical information challenges but rather as a strategic global asset that can be used to achieve better strategies aims and visions, and conceptualized in the strategic planning and human capital of the company or agency. Through this lens, companies, public agencies and international institutions should create partnerships and collaborations environment to achieve their common objectives and goals.

Different countries have different levels of IT and, in particular, big data development. To make easier the process of maturity level identification we create The Big Data Processing Maturity Model. The Big Data Maturity levels will help professionals explain to managers where big data process management shortcomings exist and set targets for where they need to be. The right maturity level will be influenced by the government's strategy objectives, the operating and technology environment and economy and social situations.

CONCLUSION

The goal of Master Thesis is achieved. In my study, I defined how big data processing influence on public sector in different countries. To meet goal I achieved step-by-step all objectives formulated in the beginning of the study. Let make review of this objectives and received results.

Objective 1: To analyze experience of big data projects implementation in different countries.

Results.

Most projects operated or implemented today can only marginally be classified as big data applications. The majority of government data projects for such a kind of trend appear to share structured databases of stored data. Next years all public agencies will be grappling with the deal of how to consolidate and integrate their distribution data, design and enhance cross-border data flows, build analytical capacities, and move toward a “data-driven” concept of decision-making processes.

“Development of big data is beginning to emerge and is expected to increase in prominence in the near future in public sector of different countries. Data analytics holds great promise for increasing the efficiency of operations, mitigating risks, and citizen engagement and value. I expect public agencies to take measured and calculated approaches as they traverse big data opportunities” (A.Kuraeva, 2015).

Objective 2: To evaluate opportunities for future growth of international partnerships in big data analysis application.

Results.

Each public agency or international institute faces the same challenges to understand the potential benefits and constraints of the big data processing. It is therefore logical to work together within governments or international institutions to share ideas and experiences, so that we can move more rapidly and efficiently to a position where big data can contribute effectively to the production of social goods. Moreover, big data scientific international collaboration helps us to overcome global challenges such as terrorism, diseases, refugee's problem and many other opportunities. Governments also need to share data related to security threats, fraud, and illegal activities. Such big data needs not only translation technologies but an international collaborative effort to share and integrate data.

I expect, the key players in international collaborations in big data research will become research institutions and global organizations such as United Nations, European Union, World Trade Organization, the World Bank and International Monetary Fund. Government agencies and state-run enterprises should identify and prioritize use cases that could provide value to the public sector and big data can address.

Objective 3: To create The Big Data Processing Maturity Model for evaluation of big data processing in public sector in different countries using qualitative and quantitative characteristics.

Results.

The Big Data Processing Maturity Model was designed based on the following guiding principles:

To have strong academic background

To use insights from ICT frameworks and best practices

Based on reliable data

Based on comprehensive analysis

To use valid dimensions applying opportunity of retrospective analysis

Guideline framework how to use The Big Data Processing Maturity Model was described.

Objective 4: To verify The Big Data Processing Maturity Model on a number of countries: Russian Federation, USA, Germany, Singapore and Nigeria.

Results.

The results of verification big data processing maturity level using The Big Data Processing Maturity Model show:

Russian Federation - 14 - Optimizing

USA - 23 - Transforming

Germany - 21 - Transforming

Singapore - 24 - Transforming

Nigeria - 9 - Exploring

Based on maturity level of initiatives and implementations of big data projects in different countries, public sector use of big data and big data analytics is wide-ranging; some government agencies and state-run enterprises have no experience with big data, while others have taken on small to moderate-sized projects.

The Big Data Processing Maturity is applicable for identification of big data processing maturity level of countries.

Objective 5: To define challenges on a way of big data penetration in governments and public agencies.

Results.

“Big data means new challenges involving complexity, security, and risks to privacy, as well as a need for new technology and human skills. Data sharing within a country among different government departments and agencies is another challenge. Governments, both local and national, in the process of implementing laws and regulations and performing public services and financial transactions accumulate an enormous amount of data with attributes, values, and challenges that differ from their counterparts in the business sector” (A.Kuraeva, 2015).

Besides main obstacle - ICT government budget, that slow down spreading big data penetration in public sector, there are number of common challenges for all countries:

High cost of big data implementation.

Lack of executive understanding of the core value of processing data.

Risk management.

Weak methodological background.

Complexity of data integration and data sharing.

Data security.

Existing workflows and business processes

Reviewing big data projects, initiatives, big data collaboration projects and different maturity levels in different countries, I identify next big data trends:

“Large and complex datasets are becoming a norm for public and private sector. Governments expect big data to enhance their ability to serve their citizens and address major national challenges involving the economy, health care, job creation, natural disasters, and terrorism. However, the majority of big data applications are in the citizen (participation in public affairs) and business sectors, rather than in the government sector” (A.Kuraeva, 2015).

...