Shulou(Shulou.com)06/01 Report--

How to build big data platform and data analysis, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Everyone in the industry knows that the construction of big data analysis platform will help enterprises build unified data storage and data processing resources, carry out big data application construction around enterprise business, and finally form service-oriented data assets.

The general big data platform includes the following steps from platform construction to data analysis:

1. Linux system installation

CentOS, an open source version of the Redhat system, is generally used as the underlying platform. In order to provide a stable hardware foundation, when RAID the hard disk and mount the data storage node, it needs to be configured according to the situation. For example, you can choose to RAID2 the namenode of HDFS to improve its stability, and put the data storage and the operating system on different hard drives to ensure the normal operation of the operating system.

2. Distributed computing platform / component installation

Most of the current distributed systems use Hadoop series open source systems. The core of Hadoop is HDFS, a distributed file system. The common components based on it are Yarn, Zookeeper, Hive, Hbase, Sqoop, Impala, ElasticSearch, Spark and so on.

The advantages of using open source components: 1) there are many users, many bug can find the answer online (which is often the most time-consuming place in development); 2) open source components are generally free, and it is relatively convenient to learn and maintain; 3) open source components are generally constantly updated; 4) because the code is open source, if there is bug, the source code is free to modify and maintain.

The commonly used distributed data warehouses are Hive and Hbase. Hive can be queried with SQL, and Hbase can read rows quickly. Sqoop is required for external database import and export. Sqoop imports data from traditional databases such as Oracle and MySQL into Hive or Hbase. Zookeeper provides data synchronization service, and Impala is a supplement to hive, which can realize efficient SQL query.

3. Data import

As mentioned earlier, the tool for data import is Sqoop. It can import data from files or traditional databases to distributed platforms.

4. Data analysis

Data analysis generally includes two stages: data preprocessing and data modeling and analysis.

Data preprocessing is to prepare for the following modeling and analysis, the main work is to extract available features from massive data and establish a wide table. Hive SQL,Spark QL and Impala may be used in this process.

Data modeling analysis is to model the features / data extracted by preprocessing and get the desired results. As mentioned earlier, it is best to use Spark in this piece. Commonly used machine learning algorithms, such as naive Bayesian, logical regression, decision tree, neural network, TFIDF, collaborative filtering and so on, are already in ML lib and are convenient to use.

5. Result visualization and output API

Visualization is a general way to display the results or part of the original data. There are generally two cases, row data display and column lookup display.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.