Shulou(Shulou.com)05/31 Report--

How to ensure high availability of HBase? in view of this question, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible way.

HBase is a column-oriented non-relational distributed database (NoSQL) based on Hadoop. The design concept comes from Google's BigTable model and is oriented to real-time reading and writing and random access to large-scale data sets. It is a distributed storage system with high reliability, high performance and high scalability, which is widely used in big data related fields.

The HBase system supports transparent segmentation of the stored data, which makes the storage and computing of the system have good horizontal scalability.

Since 2017, Zhihu has gradually adopted HBase system to store all kinds of online business data, and built various application models and data computing tasks on top of HBase services.

With the development of Zhihu in the past two years, the core architecture team of Zhihu has created a complete set of HBase service platform management system based on open source container scheduling platform Kubernetes.

After nearly two years of research and development iteration, a relatively complete HBase automatic operation and maintenance service system has been formed, which can complete the rapid deployment of HBase cluster, smooth expansion and expansion, fine-grained monitoring of HBase components, fault tracking and other functions.

Background

Zhihu doesn't have much experience with HBase. In early 2017, HBase services were mainly used for offline algorithms, recommendation, anti-cheating, as well as storage and calculation of basic data warehouse data, and accessed through MapReduce and Spark. At that time, the online storage of Zhihu mainly used MySQL and Redis systems, among which:

MySQL: supports most of the business data storage. When the data size increases, there are some tables that need to be expanded, and sub-tables will bring certain complexity. Some businesses want to block this, and some businesses want to store some data that should be stored by columns in the form of rmsdb when designing tables for historical reasons, hoping to make a migration. In addition, MySQL is based on SSD, although it has good performance and high cost.

Redis: can provide large-scale caching, but also can provide some storage support. The performance of Redis is excellent, the main limitation is that it is more complicated to do data Resharding, and the second is the high cost of memory.

In view of the problems existing in the above two kinds of online storage, we hope to establish a set of online storage NoSQL service as a supplement to the above two kinds of storage.

During the selection period, we also considered Cassandra. In the early days, some businesses tried to use Cassandra as storage. After operating and maintaining the Cassandra system for a period of time, the team next door encountered a lot of problems. The operability of the Cassandra system did not meet expectations. At present, except for Tracing-related systems, other businesses have given up using Cassandra.

Starting from the existing offline storage systems, after measuring the stability, performance, code maturity, upstream and downstream system acceptance, industry usage scenarios and community activity, we selected HBase as one of the supporting components of Zhihu online storage.

1. HBase On Kubernetes

In the early days, Zhihu had only one cluster for offline computing, and all the businesses ran on one cluster, and the HBase cluster was mixed with other offline computing yarn and Impala. The daily offline computing and data reading and writing of HBase were seriously affected by other systems.

And the monitoring of HBase only stays at the host level, when there are operational problems, it is difficult to troubleshoot, and it takes a long time for the system to restore service. in this state, we need to rebuild a system suitable for online services.

In such a scenario, our need for online HBase services is clear:

Isolation

From the perspective of the business side, it is hoped that the relevant services will be isolated from the environment and the permissions will be returned to the business, so as to avoid misoperation and business interaction.

For response time and service availability, SLA can be specified according to business needs.

It can also be more adaptable to the allocation of resources and the configuration of parameters such as blockcache, providing business-level monitoring and alarm, quickly locating and responding to problems.

Resource utilization: from the point of view of operation and maintenance, the allocation of resources should be reasonable, the host cpu should be improved as much as possible, and the effective utilization of memory including disk should be improved.

Cost control: the team uses the minimum cost to get relatively large operation and maintenance benefits, so it needs to provide a convenient call interface, which can flexibly apply, expand, manage and monitor the HBase cluster. At the same time, the cost includes machine resources and engineers. At that time, the system we had online was independently maintained by an engineer.

Combining the above requirements and referring to our team's previous experience in infrastructure platform, the ultimate goal is to make HBase services into a basic component service platform for upstream business. This is also one of the ideas of Zhihu technology platform department, which is to blackbox, interface and service all the components to the business as far as possible. At the same time, the granularity of use and monitoring should be as accurate, meticulous and comprehensive as possible. This is our original intention to build an online HBase management operation and maintenance system.

2. Why Kubernetes?

The previous article said that we hope to service the entire HBase system platform, which involves how to manage and operate the HBase system. Zhihu has a wealth of work and experience in micro services and containers.

At that time, all our online businesses had completed the containerization migration, and the more than 10,000-level business containers ran smoothly on the mesos-based container management platform Bay (see [1])

At the same time, the team is also actively trying to Infrastructure containerization, and has successfully run the basic message queue component Kafka on the Kubernetes system (see [2]), so we decided to use HBase to manage and schedule resources through Kubernetes.

Kubernetes [3] is Google's open source container cluster management system, and it is the open source version of Google's large-scale container management technology Borg for many years. Kubernetes provides resource management and scheduling solutions for various dimensional components, isolating the use of container resources, the HA work of each component, as well as a relatively perfect network scheme.

Kubernetes is designed as an ecosystem platform for building components and tools that can easily deploy, extend, and manage applications. With the blessing of Kubernetes Dafa, we soon had the initial landing version ([4]).

III. The first generation

The initial landing version architecture is shown in the figure below. The platform establishes multiple logically isolated HBase clusters on the shared physical cluster through Kubernetes (hereinafter referred to as K8S) API. Each cluster is composed of a set of Master and several Regionserver (hereinafter referred to as RS). The cluster shares a set of HDFS storage clusters, each relying on an independent Zookeeper cluster; the cluster is managed through a set of management system Kubas services ([4]).

Primary architecture

Module definition

How to build a HBase cluster in K8S, first of all, you need to use the basic components of K8S to describe the composition of HBase. K8S has the following resource components:

Node: defines a host node, which can be a physical machine or a virtual machine

Pod: a set of closely related containers, which is the basic unit of K8S scheduling

ReplicationController: a set of pod controllers that can ensure the number and health of pod operations, and can scale flexibly.

Combined with the previous experience of Kafka on K8S, for the sake of high availability and scalability, we did not adopt the deployment method with multiple containers in one Pod, and defined a class of HBase components with a single ReplicationController, that is, the Master,Regionserver in the figure above and the Thriftserver; created on demand. Through the above concepts, we can define a minimum HBase cluster on K8S as follows:

2*MasterReplicationController

3*RegionserverReplicationController

2*ThriftserverReplicationController (optional)

IV. High availability and fault recovery

As a system oriented to online business services, high availability and failover must be considered in the design. in the overall design, we consider the availability and failure recovery of component level, cluster level and data storage level respectively.

1. Component level

HBase itself has considered a number of failover and recovery scenarios:

Zookeeper cluster: self-designed to ensure availability

Master: HA and update the master node through multiple Master registrations on the Zookeeper cluster

RegionServer: itself is stateless. When a node fails to go offline, the above Region will be automatically removed, which will not have much impact on service availability.

Thriftserver: at that time, most of the business was Python and Golang. Through the implementation of HBase with Thrift, Thriftserver itself is a single point. Here, we proxy a group of Thriftserver services through HAProxy.

HDFS: it is composed of Namenode and DataNode nodes. For Namenode, we enable the HA function to ensure the cluster availability of HDFS.

2. Cluster level

Pod container failure: Pod is maintained through Replication Controller. Controller Manager of K8S will listen for component failure in its storage etcd. If the copy is less than the default value, it will automatically be served by a new Pod container.

Kubernetes cluster crash: this scenario has occurred in the production environment. In view of this situation, we use a small number of physical machines with containers for mixed deployment of services with high SLA requirements. When extreme scenarios occur, we can ensure that the impact of important businesses can be controlled.

3. Data level

All HBase clusters built on K8S share a set of HDFS clusters, and the availability of data is provided by multiple copies of the HDFS cluster.

Fifth, implementation details

1. Resource allocation

In the initial stage, the physical node uses 2x12 core cpu,128G memory and 4T disk, in which the disk is used to build service HDFS,CPU and memory is used to establish HBase-related service nodes in K8S environment.

The main function of the Master component is to manage the HBase cluster, and the Thriftserver component mainly plays the role of agent, so the resources of these two components are allocated according to a fixed amount.

When designing resource allocation for Regionserver components, consider two ways to define resources:

Resource allocation mode

Allocate according to business needs:

According to the business side's description of its own services, evaluate the relevant QPS and SLA, and configure special parameters for the business, including the size and number of blockcache,region, etc.

The advantage is that it can optimize the business, make full use of resources and reduce the resource occupation cost of the business.

The management cost increases, each business needs to be evaluated, and the platform maintenance personnel are very unfriendly. At the same time, business students need to have an understanding of HBase.

Uniform resource allocation:

CPU and MEM are allocated according to pre-set quotas, provide multi-file configuration, and package the configuration of CPU and MEM.

The convenience is that the number of Regionserver is increased directly when the service capacity is expanded, the configuration is stable, the operation and maintenance cost is low, and it is convenient to troubleshoot when problems are encountered.

Special customization is required for some businesses with unique access methods, such as CPU computing, large KV storage, or businesses with MOB requirements.

In view of the fact that there were not many online services to be accessed at that time, the Regionserver was configured by service customization. In the formal environment, a set of Regionserver was configured uniformly for the same service, and there was no mixed configuration Regionserver group.

2. Parameter configuration

The basic image is built based on cdh6.5.0-hbase1.0.0:

# Example for hbase dockerfile # install cdh6.5.0-hbase1.0.0 ADD hdfs-site.xml / usr/lib/hbase/conf/ ADD core-site.xml / usr/lib/hbase/conf/ ADD env-init.py / usr/lib/hbase/bin/ ENV JAVA_HOME / usr/lib/jvm/java-8-oracle ENV HBASE_HOME / usr/lib/hbase ENV HADOOP_PREFIX / usr/lib/hadoop ADD env-init.py / usr/lib/hbase/bin/ ADD hadoop_xml_conf.sh / usr/lib/hbase/bin/

Fixed environment variables, such as JDK_HOME,HBASE_HOME, are injected into the container image through ENV

HDFS-related environment variables, such as hdfs-site.xml and core-site.xml, are pre-added to the Docker image, and the construction process is placed in the HBase-related directory to ensure that the HBase service can access the HDFS through the corresponding configuration.

The configuration information related to HBase, such as the Zookeeper cluster address, HDFS data directory path, heap memory and GC parameters that the component depends on, needs to be modified according to the information passed in KubasService, which is a typical example of incoming parameters.

REQUEST_DATA = {"name": 'test-cluster', "rootdir": "hdfs://namenode01:8020/tmp/hbase/test-cluster", "zkparent": "/ test-cluster", "zkhost": "zookeeper01,zookeeper02,zookeeper03", "zkport": 2181, "regionserver_num":' 3percent, "codecs": "snappy" "client_type": "java", "cpu": '1century, "memory":' 30th, "status": "running",}

When starting Docker with the above parameter KubasService, modify hbase-site.xml and hbase-env.sh with hadoop_xml_conf.sh and env-init.py in the startup command to complete the configuration injection, as shown below:

Source / usr/lib/hbase/bin/hadoop_xml_conf.sh & & put_config-- file / etc/hbase/conf/hbase-site.xml-- property hbase.regionserver.codecs-- value snappy & & put_config-- file / etc/hbase/conf/hbase-site.xml-- property zookeeper.znode.parent-- value / test-cluster & & put_config-- file / etc/hbase/conf/hbase-site.xml-- property hbase.rootdir-- value hdfs://namenode01:8020 / tmp/hbase/test-cluster & & put_config-file / etc/hbase/conf/hbase-site.xml-property hbase.zookeeper.quorum-value zookeeper01 Zookeeper02,zookeeper03 & & put_config-- file / etc/hbase/conf/hbase-site.xml-- property hbase.zookeeper.property.clientPort-- value 2181 & & service hbase-regionserver start & & tail-f / var/log/hbase/hbase-hbase-regionserver.log

3. Network communication

In terms of network, the native network mode on Kubernetes is adopted. Each Pod has its own IP address, and containers can communicate directly. At the same time, DNS automatic registration and anti-registration features are added to the Kubernetes cluster, using the identified name of Pod as the domain name, and the relevant information is synchronized to the global DNS when Pod is created, restarted and terminated.

We encountered a problem in this place. At that time, our DNS resolution could not solve the corresponding container domain name through IP in the Docker network environment, which made the service name registered with Master by Regionserver and registered with Zookeeper cluster inconsistent. As a result, the same Regionserver was registered twice in Master, resulting in the failure of normal communication between Master and Regionserver and the normal service provision of the whole cluster.

After our research and experiment on the source code, we modify the / etc/hosts file before the container starts the Regionserver service to shield the Kubernetes from the injected hostname information.

This modification enables the HBase cluster launched by the container to start smoothly and initialize successfully, but it also increases the complexity for the operation and maintenance staff, because the Regionserver now seen in the Master page provided by HBase is all in the form of IP records, which brings a lot of inconvenience to monitoring and fault handling.

VI. Existing problems

The initial architecture has successfully landed. After successfully connecting nearly ten cluster services, this architecture faces the following problems:

It is tedious to manage the operation business HBase cluster.

Need to manually determine the storage of the HDFS cluster in advance, and apply for an independent Zookeeper cluster. In the early days, multiple sets of HBase directly shared a set of Zookeeper cluster in order to save trouble, which is not in line with the original intention of our design.

The container identifier is inconsistent with the regionserver address registered in HBaseMaster, which affects the fault location

A single Regionserver runs in a separate ReplicationController (hereinafter referred to as RC), but the expansion and reduction of capacity is to make full use of the characteristics of RC, roughly using the way of increasing or decreasing RC to expand and reduce capacity.

HBase configuration

The initial design lacked flexibility, and the hbase-site.xml and hbase-env.sh related to the HBase service configuration were fixed in DockerImage. In this case, if you need to update a large number of configurations, you need to re-build the image.

Since it was originally designed to share a set of HDFS clusters as storage for multiple HBase clusters, the hdfs-site.xml and core-site.xml configuration files related to HDFS are also configured directly into the image. If you need to launch a HBase that relies on other HDFS clusters in Kubasservice, you also need to rebuild the image.

HDFS isolation

With the increase of access to HBase clusters, different HBase cluster services have different requirements for HDFS IO consumption, so there is a need to separate HBase dependent HDFS clusters.

The main problem is that the Docker image solidifies the relevant configuration files. The hdfs-site.xml and core-site.xml configuration files related to HDFS correspond to the relevant Docker images, while the versions of different Docker images are completely managed by the developers themselves. These problems are not taken into account in the implementation of the original version.

Monitor operation and maintenance

Insufficient metric data, changes in in-heap and out-of-heap memory, region and table access information have not been extracted or aggregated.

Region hot spot location is slow, can not locate hot spot Region in a short time.

New or offline components can only scan the KubasService database to find relevant changes, and component anomalies such as RegionServer disconnection or restart, Master switching can not be reported in time.

VII. Reconstruction

In order to further solve the problems existing in the initial version of the architecture and optimize the management and control process of HBase, we re-examine the existing architecture, and combine the new features of Kubernetes, upgrade the original architecture, and rewrite the services of the entire Kubas management system with Golang (the initial version uses Python for development).

Based on the Kubas management system, several basic micro services for monitoring and operation and maintenance are developed to improve the flexibility of HBase cluster deployment on Kubernetes. The architecture is shown in the following figure:

Second-generation architecture diagram

1 、 Deployment&ConfigMap

Deployment

Deployment (deployment) is a concept in Kubernetes and is a set of updated object descriptions for Pod or ReplicaSet to replace the previous ReplicationController. Deployment inherits all the functions of ReplicationController and has more new management features.

In the new Kubas management system, the new design uses Deployment instead of Replication Controller for Pod management, and uses a Deployment to deploy a group of Region Servers instead of a single Regionserver corresponding to a Replication Controller design, so as to improve the flexibility of cluster deployment and scaling management.

Each group of Deployment will inject tags of various information dimensions, such as the information of the relevant cluster, service type, application, and so on.

Deployment deployment

ConfigMap

ConfigMap is the resource object that Kubernetes uses to store configuration files. Through ConfigMap, you can mount the external configuration to a specified location in the container before starting the container, and provide configuration information for the programs running in the container.

In the management system after refactoring, all the component configurations of HBase are stored in ConfigMap, and the system manager will store several HBase configuration templates in ConfigMap of K8S system according to the need.

When the business side provides the HBase service application, the manager renders the specific hbase-site for the applied HBase cluster component by combining the needs of the business resources with the configuration template. Xml and hbase-env. Sh and other HBase configuration-related files are then stored in ConfigMap

When the container starts, k8s will Mount the configuration file in ConfigMap to the path specified in the configuration according to deployment

Similar to the operation of Deployment, each ConfigMap is tagged to associate the relevant ConfigMap with the corresponding cluster and application.

ConfigMap Archive

2. Configuration of component parameters

With the introduction of the ConfigMap feature, the request information for previously creating the cluster also changes.

RequestData {"name": "performance-test-rmwl", "namespace": "online", "app": "kubas", "config_template": "online-example-base.v1", "status": "Ready", "properties": {"hbase.regionserver.codecs": "snappy", "hbase.rootdir": "hdfs://zhihu-example-online:8020/user/online-tsn/performance-test-rmwl" "hbase.zookeeper.property.clientPort": "2181", "hbase.zookeeper.quorum": "zookeeper01,zookeeper02,zookeeper03", "zookeeper.znode.parent": "/ performance-test-rmwl"}, "client_type": "java", "cluster_uid": "k8s-example-hbase---performance-test-rmwl---example"}

Config_template specifies the configuration information template used by the cluster, and then all the component configurations related to the HBase cluster are rendered by this configuration template.

Config_template also pre-specifies the basic running configuration information of HBase components, such as component type, startup commands used, image files used, initial number of copies, and so on.

Servers: {"master": {"servertype": "master", "command": "service hbase-master start & & tail-f / var/log/hbase/hbase-hbase-master.log", "replicas": 1, "image": "dockerimage.zhihu.example/apps/example-master:v1.1", "requests": {"cpu": "500m" "memory": "5Gi"}, "limits": {"cpu": "4000m"},}

The Docker image file cooperates with the ConfigMap function to store the configuration file information in the pre-agreed path, while adding the soft chain file to the real HBase configuration path.

RUN mkdir-p / data/hbase/hbase-site RUN mv / etc/hbase/conf/hbase-site.xml / data/hbase/hbase-site/hbase-site.xml RUN ln-s / data/hbase/hbase-site/hbase-site.xml / etc/hbase/conf/hbase-site.xml RUN mkdir-p / data/hbase/hbase-env RUN mv / etc/hbase/conf/hbase-env.sh / data/hbase/hbase-env/hbase-env.sh RUN ln-s / data/hbase/hbase -env/hbase-env.sh / etc/hbase/conf/hbase-env.sh

3. Construction process

Combined with the previous introduction of Deployment and ConfigMap, as well as changes to Dockerfile, the entire HBase build process has also been improved:

HBaseonKubernetes construction process

Compile the relevant Dockerfile and build the basic HBase component image

Build the basic attribute configuration template for the HBase to be created, and customize the basic resources, which can be used to create a ConfigMap in the Kubernetes cluster through KubasAPI

When creating a deployment cluster, you can render the detailed ConfigMap of various components in the HBase cluster by calling KubasAPI and combining with the previously built ConfigMap template, and then build the Deployment in the Kubernetes cluster

Finally, a HBase component container running in KubernetesNode is completed through the configuration in the previously built image loading component ConfigMap.

By combining the configuration template of ConfigMap function of K8S and calling KubasAPI, we can deploy a set of available minimum HBase cluster (2Master + 3Region Server + 2Thriftserver) in a short time. In the scenario where all host Host have cached Docker image files, it takes no more than 15 seconds to deploy and start a whole HBase cluster.

At the same time, in the absence of dedicated front-end console, you can completely rely on Kubernetesdashboard to expand and reduce the capacity of HBase cluster components, as well as query, modify, update and redeploy component configuration.

VIII. Resource control

After refactoring, HBase service is open to Zhihu internal business. In a short period of time, Zhihu HBase cluster has increased by more than 30 + clusters. With the increase in the number of HBase clusters, two problems have gradually emerged:

The cost of operation and maintenance increases: the clusters that need operation and maintenance are gradually increasing.

Waste of resources: this is because the volume of many businesses is not high, but in order to ensure the high availability of HBase, we need to provide at least 2 Master+3 RegionServer, and often the load of Master is very low, which results in a waste of resources.

In order to solve the above two problems without breaking the requirement of resource isolation, we add HBaseRSGroup function to the management system of HBase platform.

The optimized architecture is as follows:

The use of RSGroup

Because the management of the business HBase cluster is isolated by the platform itself, the platform uses a degraded approach to manage the HBase cluster for further resource management.

By monitoring the metrics of each individual cluster, if the load of the business cluster falls below the threshold after a certain period of time, the platform will cooperate with the business side to migrate the HBase cluster to a set of MixedHBase clusters.

At the same time, if the load of a HBase business running in a MixedHBase cluster increases and continues to exceed the threshold for a period of time, the platform will consider promoting the relevant business to a separate cluster.

IX. Multi-IDC optimization

With the development and expansion of Zhihu business, the infrastructure of Zhihu has been gradually upgraded to multi-server room architecture. The management mode of Zhihu HBase platform has also been further upgraded in this process, and began to build a management mode of multi-server room management. The basic structure is shown below:

Multiple IDC access mode

The business HBase cluster runs on multiple IDC, and the master-slave mode of the IDC computer room is determined by the service. The slave IDC cluster data of the service is synchronized through the data synchronization components of the platform.

The Kubas services of each IDC are mainly responsible for the specific operations of the local Kubernetes cluster, including the management of the creation and deletion of the HBase cluster, the expansion of regionserver and other HBase components. The deployment of Kubas services is related to the computer room, and only connects with the K8S cluster in the computer room where the deployment is located.

The Kubas services of each IDC report the cluster information of the local computer room to the cluster discovery service, and update the relevant master and slave information of the relevant cluster.

The business side accesses the HBase cluster of multiple computer rooms through the ClientSDK encapsulated by the platform. The client can determine the master-slave relationship of the HBase cluster through the cluster discovery service, thus separating the relevant read and write operations, and the write modification access can point to the cluster of the master IDC through the client.

Data synchronization across computer rooms uses a self-developed HBaseReplicationWALTransfer to provide incremental data synchronization.

10. Data synchronization

In all kinds of business scenarios, there is a need for data synchronization across HBase clusters, such as data synchronization in offline HBase clusters and online clusters, multi-IDC cluster data synchronization, etc. For HBase data synchronization, there are two ways: full replication and incremental replication.

HBase data synchronization

In Zhihu HBase platform, we use two methods to synchronize data between HBase clusters:

HBase Snapshot

We use HBaseSnapshot for full data replication, which is mainly used in scenarios where offline data is synchronized with online data.

WALTransfer

It is mainly used for incremental data synchronization between HBase clusters; incremental replication we do not use HBaseReplication, we use self-developed WALTransfer components to incrementally synchronize HBase data

WALTransfer provides a list of WAL files by reading the source data HBase cluster, locates the corresponding WAL files in the HDFS cluster, and writes the incremental data of HBase to the destination cluster in order. We will analyze the relevant details in detail in later articles.

11. Monitoring

From the previous refactored architecture diagram, we can see that we have added a lot of modules to the Kubas service, which basically belong to the monitoring and management module of the HBase platform.

1. Kubas-Monitor component

The basic monitoring module discovers new HBase clusters by polling, and discovers HBase clusters, Master and Regionserver groups by subscribing to Zookeeper clusters.

Collect data from Regionserver Metric. The main data collection includes:

The information of Region, the number of region online, the number of store, the size of storefile, the size of storefileindex, the number of times memstore is missing when reading

Blockcache information, such as how much is used in blockcache, how much is idle, the cumulative loss rate, etc.

Statistics of read and write requests, such as read and write response time, table distribution of read and write, amount of read and write data, number of read and write failures, etc.

Operation information of compact and split, such as queue length, number of operations and time, etc.

Handler information, such as queue length, number of active handler, and number of active reader.

Metrics of other dimensions, such as container CPU and Mem occupancy, come from Kubernetes platform monitoring, disk IO, disk occupancy, etc., from host monitoring:

HBase partial monitoring

2. Kubas-Region-Inspector components

Collect the Region information of HBase table, obtain the data statistics of each HBaseRegion through HBaseAPI interface, and aggregate the Region data into data table information.

The Region distribution chart of HBase cluster is formed by calling open source components to locate Region hotspots.

HBaseRegion distributed monitoring

Through the monitoring information collected by the above modules, we can basically describe the status information of the HBase cluster running on Kubernetes, and can assist the operation and maintenance management staff to locate and troubleshoot the fault.

12. Future Work

With the rapid development of the company's business, Zhihu's HBase platform business is also constantly iterated and optimized. In the short term, we will further improve the management service capability of Zhihu HBase platform from the following directions:

Improve the security and stability of the cluster. Add HBase permission support to further improve security isolation under multi-tenant access

User cluster construction is customized. By providing a user data management system, the HBase construction interface is opened to business users, so that business users can build HBase clusters and add plug-ins such as Phoniex.

Automation of operation and maintenance inspection. Automatic cluster expansion, automatic hot spot detection and transfer, etc.

This is the answer to the question about how to ensure high availability for HBase. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.