Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what are the advantages of elasticsearch cluster architecture". In daily operation, I believe many people have doubts about the advantages of elasticsearch cluster architecture. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "what are the advantages of elasticsearch cluster architecture?" Next, please follow the editor to study!

The Evolution of ES Cluster Architecture

1. Initial stage

The initial stage of the order center ES is like a blank sheet of paper, there is basically no erection plan, and many configurations keep the default configuration of the cluster. The whole cluster is deployed on the elastic cloud of the group, and the nodes and machines of the ES cluster are chaotic. At the same time, according to the cluster dimension, an ES cluster will have a single point of problem, which is obviously not allowed for the order center business.

2. Cluster isolation phase

Like many businesses, ES clusters adopt a mixed distribution approach. However, because the order center ES stores online order data, occasionally mixed clusters seize a large number of resources of the system, resulting in abnormal ES service of the whole order center.

Obviously, any situation that affects the stability of the order query will not be tolerated, so for this case, the elastic cloud where the order center ES is located moves out of the cluster nodes that occupy high system resources, and the situation of the ES cluster is slightly improved. However, with the continuous increase of cluster data, the elastic cloud configuration can no longer satisfy the ES cluster, and for complete physical isolation, the order center ES cluster is simply deployed to high-configuration physical machines, and the performance of the ES cluster is improved.

3. Node replica tuning phase

The performance of ES has a lot to do with hardware resources. When an ES cluster is separately deployed to a physical machine, the nodes within the cluster do not monopolize the resources of the whole physical machine. When the cluster is running, the nodes on the same physical machine will still have the problem of preempting resources. So in this case, in order to maximize the use of machine resources for a single ES node, each ES node is deployed on a separate physical machine.

But then the question arises again, what if a single node has a bottleneck? How should we optimize it?

The principle of ES query: when a request reaches a certain number of shards, if the query does not specify a shard type (Preference parameter), the request will be loaded on each node of the corresponding shard number. The default replica configuration of the cluster is one master and one deputy. In view of this situation, we think of the way to expand the replica capacity, from the default one master and one pair to one master and two pairs, while adding corresponding physical machines.

Schematic diagram of setting up ES cluster in order center

As shown in the figure, the entire erection method uses VIP to load balance external requests:

The whole cluster has a set of primary shards and two sets of secondary shards (one primary and two deputy). Requests forwarded from the gateway node will be balanced by polling before hitting the data node. The way that the cluster adds a set of replicas and expands the machine capacity increases the cluster throughput, thus improving the query performance of the whole cluster.

Of course, the number of fragments and the number of copies is not the more the better, at this stage, we made a further exploration on the selection of the appropriate number of fragments. The number of fragments can be understood as a database table in MySQL, while the current order center ES query is mainly divided into two categories: single ID query and paging query.

The larger the number of shards, the larger the scale of horizontal expansion of the cluster, and the throughput of single ID query based on shard routing can also be greatly improved, but the performance of aggregated paging query will be reduced; the smaller the number of shards, the smaller the scale of horizontal expansion of cluster, and the query performance of single ID will also decline, but the performance of paged query will be improved.

So how to balance the number of shards and the existing query business, we made many adjustments to stress testing, and finally chose the number of shards with better cluster performance.

4. Master-slave cluster adjustment phase

At this point, the ES cluster of the order center has begun to take shape, but because the order center business requires high timeliness and high ES query stability, if any node in the cluster is abnormal, the query service will be affected, thus affecting the entire order production process. It is obvious that this exception is fatal, so in order to deal with this situation, our initial idea is to add a standby cluster, when an exception occurs in the primary cluster, the query traffic can be downgraded to the standby cluster in real time.

How should the backup cluster be set up? How to synchronize data between master and backup? What kind of data should the standby cluster store?

Considering that the ES cluster does not have a good master / slave scheme for the time being, and in order to better control ES data writing, we use the business double write method to set up the master / slave cluster. Every time the business operation needs to write ES data, the master cluster data is written synchronously, and then the standby cluster data is written asynchronously. At the same time, because most of the ES query traffic comes from orders in recent days, and the database data of the order center has a set of archiving mechanism, the orders that have been closed before the specified number of days are transferred to the historical order database.

Therefore, the logic of deleting standby cluster documents is added in the archiving mechanism, so that the order data stored in the newly built standby cluster is consistent with the amount of data in the database on the order centerline. At the same time, ZK is used to make a flow control switch in the query service to ensure that the query traffic can be degraded to the standby cluster in real time. Here, the order center master-slave cluster is completed, and the stability of ES query service is greatly improved.

5. Today: real-time mutual backup dual cluster stage

During this period, because the ES version of the main cluster is lower than 1.7, while the current stable version of ES has been iterated to 6.x, the new version of ES not only optimizes the performance greatly, but also provides some new and useful features, so we have carried out a version upgrade to the main cluster, directly from 1.7 to 6.x.

The process of cluster upgrade is tedious and lengthy. It is not only necessary to ensure no impact on online business and smooth upgrade without perception, but also that ES cluster does not support data migration across multiple versions from 1.7 to 6.x, so you need to upgrade the main cluster by rebuilding the index. The specific upgrade process will not be described here.

Unavailability is inevitable when the primary cluster is upgraded, but this is not allowed for the order center ES query service. Therefore, in the upgrade phase, the slave cluster temporarily acts as the master cluster to support all online ES queries to ensure that the upgrade process does not affect the normal online services. At the same time, for online business, we have redefined the planning of the two clusters and re-divided the online query traffic.

The backup cluster stores hot data online in recent days, and the data scale is much smaller than that of the main cluster, which is about 1/10 of the number of documents in the main cluster. The amount of cluster data is small, and under the same cluster deployment scale, the performance of the standby cluster is better than that of the main cluster.

However, in the online real scene, most of the online query traffic also comes from hot data, so the standby cluster is used to carry the query of these hot data, and the standby cluster gradually evolves into a hot data cluster. The previous main cluster stores full data, and uses this cluster to support the rest of the query traffic. This part of the query is mainly the special scenario query that needs to search the full order and the internal query of the order center system, and the main cluster has gradually evolved into a cold data cluster.

At the same time, the standby cluster can be downgraded to the main cluster with one click. The two clusters are equally important, but each can be demoted to another cluster. The double-write strategy is also optimized as follows: assuming that there is an AB cluster, the normal synchronous mode (A cluster) and asynchronous mode (B cluster). When an exception occurs in cluster A, cluster B is written synchronously (master) and cluster An asynchronously (standby).

Synchronization Scheme of ES order data

When MySQL data is synchronized to ES, it can be summarized in two ways:

Scenario 1: monitor the Binlog of MySQL and analyze the Binlog to synchronize the data to the ES cluster.

Scenario 2: write the data to the ES cluster directly through ES API.

Considering the business particularity of ES service in order system, the real-time performance of order data is high. Obviously, the way of monitoring Binlog is equivalent to asynchronous synchronization, which may produce a large delay. And scheme 1 is essentially similar to scheme 2, but a new system is introduced, and the maintenance cost is also increased. Therefore, the order center ES adopts the way of writing order data directly through ES API, which is simple and flexible, and can well meet the needs of order center data synchronization to ES.

Because the synchronization of ES order data is written in the business, when an exception occurs in the new or updated document, if you retry, the response time of the normal operation of the business will be affected.

Therefore, the ES is updated only once for each business operation. If an error or exception occurs, a remedial task is inserted into the database. Some Worker tasks will scan the data in real time and update the ES data again based on the database order data. Through this compensation mechanism, the ultimate consistency between ES data and database order data can be ensured.

Some of the pits encountered

1. Queries with high real-time requirements go to DB.

Students with knowledge of the ES writing mechanism may know that the new documents will be collected into Indexing Buffer, then written to the file system cache, and can be indexed like other files in the file system cache.

By default, however, documents from Indexing Buffer to the file system cache (that is, Refresh operations) are automatically refreshed in pieces per second, so that's why we say ES is a near-real-time search rather than real-time: document changes are not immediately visible to the search, but become visible within a second.

The current order system ES uses the default Refresh configuration, so for those services with high real-time order data, directly go to the database query to ensure the accuracy of the data.

2. Avoid deep paging queries

The paging query of ES cluster supports from and size parameters. When querying, each shard must construct a priority queue with the length of from+size, and then send it back to the gateway node, and then the gateway node sorts these priority queues to find the correct size documents.

Suppose that in an index with six main shards, the from is 10000 and the score size is 10, each shard must produce 10010 results, aggregate and merge 60060 results in the gateway node, and finally find 10 documents that meet the requirements.

Thus, when the from is large enough, even if OOM does not occur, it will affect the CPU and bandwidth, thus affecting the performance of the whole cluster. So you should avoid deep paging queries and try not to use them.

3. FieldData and Doc Values

FieldData

The online query times out occasionally. By debugging the query statement, it is located that it has something to do with sorting. Sorting in the es1.x version uses the FieldData structure, FieldData consumes JVM Heap memory, JVM memory is limited, and a threshold is set for FieldData Cache.

If you run out of space, the LRU algorithm removes the FieldData and loads the new FieldData Cache, which consumes system resources and takes a lot of time. As a result, the response time of this query soars, and even affects the performance of the entire cluster. To solve this problem, the solution is to use Doc Values.

Doc Values

Doc Values is a column data storage structure, similar to FieldData, but its storage location is in the Lucene file, that is, it does not occupy JVM Heap. With the iteration of the ES version, Doc Values is more stable than FieldData, and Doc Values is the default setting since 2.x.

At this point, the study on "what are the advantages of elasticsearch cluster architecture" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.