Shulou(Shulou.com)06/01 Report--

How to Elasticsearch 7.7 asynchronous search principle analysis, I believe many inexperienced people are helpless, for this reason this article summarizes the causes and solutions of the problem, through this article I hope you can solve this problem.

Elasticsearch version 7.7 brings a new feature, the search process allows asynchronous execution, after the client sends the search request, Elasticsearch server returns an id to the client, and the client uses this id to obtain the search progress, and supports returning "partial" results, which is very friendly to UI interaction related query requests, such as the drawing process can be displayed gradually.

basic usage

Asynchronous search is very simple to use, just use the new API, and the rest is the same as the_search request:

POST /_async_search

There will be several new fields in the returned results:

id: Get the subsequent query progress based on this id.

is_partial: When query is finished, this field indicates whether query is executed successfully on all shards or fails.

is_running: Indicates whether the search process is still running.

If the search runs quickly, the Response of_async_search will contain the complete search results. By default, it will wait for 1 second, controlled by the parameter wait_for_completion_timeout. If this time exceeds, the search progress (or results) will be obtained according to id:

GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=

This is very thoughtful. In the case of user authentication, everyone can only GET to the asynchronous search results submitted by themselves, and will not be seen by others. Search results are saved for 5 days by default, controlled by the keep_alive parameter.

Similarly, you can manually Delete this asynchronous search request, and if the search is still in progress, it will be cancelled.

DELETE /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=

realization principle

The implementation principle of asynchronous search is not much different from synchronous search. The process of searching performed by data nodes is the same. The only difference is that the coordinator node originally waits for the whole process to be completed before returning to the client. Now it only waits for 1 second (controlled by the wait_for_completion_timeout parameter). If the whole process is completed within 1 second, the final result is returned to the client, and after 1 second, an id is generated and returned to the client.

As usual, the data node returns a complete Response at fragment level after executing a query for a fragment, and does not calculate one point to return one point, so the unit of Response is still at fragment level. Batched_reduce_size is set to 5 by default, which defaults to 512 in synchronous search. A smaller value returns partial results to the client earlier.

To save query results for 5 days, es creates a system index named.async-search into which the query results are saved, but if the search process ends within the wait_for_completion_timeout, all result sets are returned in the current request and not saved to the.async-search index.

Start a search from Node A, what about GET from Node B?

Asynchronous search returns an id, and then obtains progress according to this id. How to obtain this task information? Where the progress and result sets need to be stored is a question that needs to be considered. ES will put it in two places:

If the search process has not been completed, the progress information is obtained from the taskmanager of the coordination node.

If the search is finished, the progress and result sets are saved to an index called.async-search.

In the first case, the progress information is stored in the coordinator node's memory. This information exists only on a single node of the whole cluster. Therefore, when your asynchronous search request is sent to node1 and the progress request is sent to node2 (for example, forwarded by the Load Balancer or polled by the client itself), how can you get the progress information at node2? The answer is that node2 receives a GET request for progress and forwards it to node1. So how does node2 know that asynchronous search requests are sent to node1? This information is actually stored in the id returned by the asynchronous search request, so you now know why it is so long:

"id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc="

This id is a base64 code that contains the following information:

docId When search results are saved in.async-search, the asynchronous search task results docId in.async-search

nodeId The nodeId of the node that received the asyncsearch request

id Task id in taskManager

With this information, the process of GET search progress is relatively clear:

After receiving the GET /_async_search/id request, the coordinator node decodes the above three pieces of information according to the id, and first determines whether the node executing_async_search is its own node. If it is not its own node, it directly sends an RPC request to the target node according to the decoded nodeId to obtain this information; if it is its own node, it obtains this information from its own taskManager according to the task id, or executes an ordinary GET doc request according to the docId to obtain it from the.async-search index.

.async-search Expired deletion of index

The data in index.async-search is stored for 5 days by default, but everyone knows that there is no TTL concept in es, so how to delete the expired data? In fact, es internally performs DeleteByQuery on this index periodically.

DeleteByQueryRequest toDelete = new DeleteByQueryRequest(INDEX)

.setQuery(QueryBuilders.rangeQuery(EXPIRATION_TIME_FIELD).lte(nowInMillis));

When a node receives the cluster status, it drives a periodic thread in clusterChanged to perform cleanup. By default, it is performed once every hour, controlled by the parameter async_search.index_cleanup_interval. This cleanup operation is performed by the GENERIC thread pool, and it will only be performed on the node where the primary shard 0 of the.async-search index is located, and will not be performed on all nodes.

After reading the above, do you know how to parse the asynchronous search principle of Elasticsearch 7.7? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.