Shulou(Shulou.com)06/01 Report--

This article mainly explains "Application Analysis of Hadoop Technology in Business Intelligence BI". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let Xiaobian take you to learn "Application Analysis of Hadoop Technology in Business Intelligence BI"!

Hadoop is a popular distributed computing solution, Apache is an open source project name, the core part includes HDFS and MapReduce. HDFS is a distributed file system and MapReduce is a distributed computing engine. Today, Hadoop has been verified, recognized and even mature in technology, and has also spawned a huge ecosystem, including HBase, Hive, Spark, etc. HBase is a distributed columnar database based on HDFS, HIVE is a data warehouse system based on HBase. Impala provides real-time SQL query capabilities for data stored in HDFS and HBase, is based on HIVE services, and shares HIVE metadata. Spark is a parallel computing framework similar to MapReduce, which also provides a similar Spark SQL query interface to HIVE, and Hive is a hadoop-based data analysis tool.

Many enterprises, such as banks, have a lot of pipeline operations, and the data is updated in real time and the amount of data is very large. Hadoop will be adopted as the underlying database, and the underlying data will be processed by middlemen, and then the BI system will be used to connect the intermediate tables of these intermediate data processing vendors to access the processed data, especially the Hadoop big data platform vendors such as Star Ring and Huawei, which are also widely used.

Here is a brief introduction to the combined application of Star Ring Big Data Platform and Sail Soft Big Data BI tool FineBI.

Because Star Ring is also dealing with Hive databases under Hadoop, its essence is similar, you can use the jdbc driver provided by Hive, this driver can also let FineBI connect to Star Ring databases and perform sql statement queries of relational databases, etc. Some special sql formulas need to confirm with Star Ring personnel whether they can be used.

First copy these drivers to Report Engineering and restart BI Server. After restart, you can establish a data connection with the Star Ring database, and finally query the data through the connection.

1. Local deployment

The following figure shows the jar package of hadoop used for FineBI internal test (place the following jar package under webinf-lib folder). For personal test, the connection can be successful, as shown in the following figure:

2. Data connection

Data connections are shown below:

After the test connection is successful, click OK to directly select the corresponding table in the database to add to the business package, similar to Mysql, the most common database table retrieval method.

3. Actual analysis of cases

Head Office Level of a Bank-Institutional Dimension-Quadrant Diagram

(2) Head Office Level-Institutional Dimension-Trend Analysis

(3) Head Office Level-Product Dimension-Profit Product

4. About FineBI's FineIndex and FineDirect features

Hadoop is the bottom layer, hive is the database, the above case is FineIndex (cube) connection, using hiveserver data connection; After the data connection is successful, add the tables in the hive database to the business package, that is, bring the data in the database to our multidimensional database.(FineIndex), of course, in the process of fetching, you can also read database association and escape, you can also manually escape and associate, and you can also do some etl operations such as adding formula columns/column conversion/join/union/filtering/grouping statistics/self-circulating columns/adding grouping columns/using some fields, etc. These processed data tables are used for front-end analysis.

In other words, database-FineIndex-front-end analysis, where FineIndex is equivalent to a form of intermediate library, used to store data tables, association escape indexes, etc. All of these greatly improve the efficiency of subsequent foreground analysis and processing of data (because direct sql access, efficiency is limited by the database itself, when the amount of data is large, general analysis tools are easy to get stuck and promoted, memory overflow causes the system to be unresponsive), which is also the original intention of FineIndex scheme. FineIndex exists for two purposes, one is to improve efficiency, and the other is to integrate data twice.

FineBI also has a connection mode FineDirect (database direct connection), which mainly meets the following requirements:

Real-time analysis of results

When enterprise users use BI tools, most of the time they perform OLAP analysis on a large amount of historical data, but some users need to present the results in real time. For example, the analysis of transaction risk in financial industry is to analyze each flow in real time. If it needs to go through the process of constructing multidimensional database, the arrival of data will be delayed, which will affect the accuracy of analysis results. But because the computation process is handed over to the database, the response speed depends more on the performance of the database.

Full utilization of big data platforms

With the continuous optimization of various distributed computing schemes, the computing performance of data has also developed rapidly, and the computing capacity has been significantly improved. Many enterprises have already had their own big data computing platforms, such as hadoop, kylin, greenplum, vertica, etc. The processing performance of these platforms for large amounts of data is sufficient to meet the use requirements, and there is no need for modeling. Therefore, FineBI Direct Engine provides the function of docking these data platforms.

At this point, I believe that everyone has a deeper understanding of "Hadoop technology in business intelligence BI application analysis", may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.