The life cycle of establishing big data for enterprises should include these parts: big data organization, status evaluation, big data strategy formulation, data definition, data collection, data analysis, data governance and continuous improvement.

I. Organization of massive data

No one, everything is bullshit. The first step in the life cycle of big data should be to establish a "big data planning, construction and operation organization" with a special budget and independent KPI. Includes senior chief data officer as the responder, then the company data management committee or the big data implementation planning steering committee, and then the big data project group or the predecessor of the big data project group: the big data project pre-research group or the big data project preparation group. This team is the backbone of the future big data strategy formulation and implementation. Due to the large number of people, it is suggested to introduce RACI model to clarify the roles and responsibilities of all people.

Second, the status assessment and gap analysis of big data

Before making a strategy, we must first evaluate the current situation. The pre-evaluation survey includes three aspects: 1. External survey: What is the latest development of big data in the industry, and what is the application level of big data in the top enterprises in the industry? What is the average level of the industry, especially the application level of big data of major competitors? The second is to investigate internal customers. What do management, business departments, IT departments themselves and our end users expect from our big data business? The third is to find out your own situation and understand your own technology and personnel reserves. Finally, benchmarking, gap analysis, find out the gap.

After finding the gap, it is necessary to evaluate the maturity status. Generally speaking, the maturity of a company's big data application can be divided into four stages: the initial stage (only concept, no practice); Exploration period (I already know the basic concepts, and some people have discussed them, and I have basic big data technology reserves); Development period (having or establishing clear strategies, teams, tools and processes, and delivering initial results); Maturity (having a stable and mature strategy, team, tools and processes, and constantly delivering high-quality results).

Third, big data strategy.

With a big data organization and understanding the status quo, gaps and needs of our company's big data, we can set the strategic goals of big data. The formulation of big data strategy is the soul and core of the whole big data life cycle, and it will become a guide for the development of big data in the whole organization.

There is no uniform template for the content of big data strategy, but there are some basic requirements:

1. Concise and to the point, covering the needs of stakeholders inside and outside the company.

2. Be clear, so that we can clearly tell you what our goals and visions are.

Practically speaking, this goal can be achieved through hard work.

Fourth, the definition of big data

I think: "You can't collect data without defining data; If you don't receive it, you can't analyze it; If you cannot analyze it, you cannot measure it; If you cannot measure it, you cannot control it; If you can't manage it, you can't manage it; If you cannot manage it, you cannot use it. " Therefore, "after the requirements and strategies are clear, data definition is the premise of all data management".

Verb (abbreviation for verb) data collection

1. The data sources in the era of big data are very extensive, which may mainly come from three aspects: data generated by various application systems in the existing enterprise intranet (such as office and business production data), data from the enterprise external internet (such as social network data) and the Internet of Things.

2. There are many kinds of big data, which can be generally divided into: traditional structured data and a lot of unstructured data (such as audio and video).

3. There are many data collection and mining tools. Based on or integrated with the ETL platform of hadoop, data value mining tools represented by interactive exploration and data mining are becoming a trend.

4. Principles of data collection: Under the background of extensive data sources, huge data volume and numerous collection and mining tools, big data decision makers must clarify the principles of data collection: "The data that can be collected does not mean that it is worth collecting or necessary to collect. The "intersection" of the data that needs to be collected and the data that can be collected is the data that we determine to collect. "

Data processing and analysis of intransitive verbs

There are many tools in the industry that can help enterprises build an integrated "data processing and analysis platform". For enterprise big data managers and planners, the key is that "tools should meet the requirements of the platform, and the platform should meet the requirements of the business, not the business should meet the requirements of the platform, and the platform should meet the requirements of the tools of the manufacturers". So what kind of capabilities should this integrated platform have? It should be able to easily retrieve, classify, associate, push and realize metadata management. See the figure below:

Seven, data presentation

The value of big data management will eventually help management and business departments make business decisions through various forms of data presentation. Decision makers of big data need to integrate the system of big data with BI (Business Intelligence) system and KM (Knowledge Management) system. The following figure shows various presentation forms of big data.

Eight. Audit, governance and control

1. The audit, governance and control of big data refers to the management of big data, that is, setting up a special governance and control team, formulating a series of strategies, processes, systems and evaluation index systems, and supervising, inspecting and coordinating the objectives of several related functional departments, so as to optimize, protect and utilize big data and ensure its true value as a strategic asset of enterprises.

2. The governance of big data is an integral part of it governance, and the audit of big data is an integral part of IT audit. This system should be planned and implemented in a holistic way, not in a fragmented way.

3. The core of big data auditing, governance and control is data security, data quality and data efficiency.

Nine, continuous improvement

Based on the ever-changing business requirements and the problems exposed in the whole life cycle of big data found in audit and governance, PDCA and other methodologies are introduced to continuously optimize strategies, methods, processes and tools, and continuously improve the skills of relevant personnel, thus ensuring the continuous success of the big data strategy!