Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to use Spark to analyze cloud HBase data. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

1 current situation of cloud HBase query analysis

HBase native API:HBase native API is suitable for doing some query based on row key. This is the query scenario that HBase is best at.

Phoenix:Phoenix, as the SQL layer of HBase, uses secondary index technology and is good at multi-conditional combination queries. Phoenix does not have its own computing resources, and complex queries such as groupby need to be completed with the help of HBase coprocessors, which on the one hand has poor performance, and at the same time will affect the stability of HBase clusters.

Spark: with rich operators to support complex analysis, using the computing resources of Spark cluster, the performance can be improved by concurrency analysis without affecting the stability of HBase cluster.

2 comparison of the ways of analyzing HBase by Spark

There are three ways for Spark to analyze HBase data: "RDD API", "SQL API" and "HFILE". The correlation is as follows:

SQL API is recommended for small tables with dynamic data updates, which can effectively optimize the analysis and reduce the impact on the stability of HBase clusters; for static tables or full static tables, it is recommended to use the method of analyzing HFILE to read HDFS directly, so that it does not affect the stability of HBase clusters at all. RDD API is not recommended, which has no poor optimization performance. at the same time, when there is high concurrency and large amount of table data, it will seriously affect the stability of the HBase cluster, thus affecting the online business.

3 the specific use of the three ways

Cloud HBase team provides you with a github project for your reference to use the above three ways to develop Spark analysis HBase program, project address:

Https://github.com/lw309637554/alicloud-hbase-spark-examples?spm=a2c4e.11153940.blogcont573569.14.1b6077b4MNpI9X

Dependencies: you need to download the client package of Cloud HBase and Cloud Phoenix

Analyze HFILE:

You need to activate HDFS access to Cloud HBase first. Refer to the documentation.

Generate the snapshot table "snapshot 'sourceTable',' snapshotName'" for the table in hbase shell

Configure your own hdfs-sit.xml file in the project, and then analyze the snapshot table by reading HDFS directly

Specific example

RDD API correspondence: org.apache.spark.hbase.NativeRDDAnalyze

SQL API correspondence: org.apache.spark.sql.execution.datasources.hbase.SqlAnalyze

Analyze HFILE correspondence: org.apache.spark.hfile.SparkAnalyzeHFILE

This is the end of this article on "how to use Spark to analyze cloud HBase data". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.