Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how CDH uses Solr to achieve HBase secondary index. Many people may not know much about it. In order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

1. Simple explanation

The principle of using Solr to build a Hbase secondary index is very simple, but the space is exchanged for time, the index is built in Solr for the specified column, and finally the Hbase is accessed through rowkey, as shown below:

two。 Four major components

Hbase+Zookeeper+Solr+Key Value Store (Lily Hbase Indexer)

It can be seen that Hbase and Solr are deployed independently, and the core component that makes the relationship between them is Key Value Store.

Of course, we can implement this without using the existing Kmurv Store code. The logic is: pull data from Hbase and then initiate a POST request to save it in Solr. One or more rows of data in Hbase corresponds to one or more doc in Solr, as shown in the figure:

3. Three important configuration files

Schema.xml: build Solr schema corresponding to the component Solr, and define the columns to be indexed by Hbase, as shown in the figure:

Morphline-hbase-mapper.xml: let Hbase have a relationship with Lily, as shown in the figure:

The mapping relationship between morphlines.conf:Hbase and Solr is reflected in this file, as shown in the figure:

The diagram of the three files is as follows, and Hbase,Solr and Lyli are mapped one by one through these three files, as shown in the figure:

4. Understand Solr

See the logical view:

Index comparison figure:

Physical and logical views and indexed views:

Index creation process: Solr provides restful interface

5. Understanding Hbase structure diagrams

6. Learn how Lily acts as a bridge

Based on the Hbase replication feature:

Specific execution process: main class SepConsumer, multi-thread concurrent processing

In fact, this is the process:

After reading the above, do you have any further understanding of how CDH uses Solr to implement HBase secondary indexes? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.