Shulou(Shulou.com)06/03 Report--

This article mainly introduces "what are the problems of Elasticsearch paging query". In the daily operation, I believe that many people have doubts about the problems of Elasticsearch paging query. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the questions of "what are the questions of Elasticsearch paging query?" Next, please follow the editor to study!

A feature of Elasticsearch paging queries is that if you write a query statement like this:

{"from": 10, "size": 10, "query": {}}

Elasticsearch will query out the first 20 pieces of data, then truncate the first 10 pieces of data and return only 10-20 pieces of data.

The side effects of this are obvious. If the amount of data is large, the query will be slower and slower.

So for queries with a large amount of data, use scroll. This approach is equivalent to creating a cursor that marks the current reading position and ensures that the next query fetches the data quickly.

But there is also a small hole to pay attention to in both ways, which is explained in detail below.

From + size mode

Problems that may arise:

Result window is too large, from + size must be less than or equal to: [10000] but was [10010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

In fact, this error message has been made very clear, through this paging query of the maximum value is 10000, more than 10000 will report an error.

The solution is also very simple, one is to use scroll for large amount of data query, the other is to increase the size of index.max_result_window value to support the query range.

Scroll is recommended.

Scroll mode

Problems that may arise:

Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.

The reason for this error is:

When there are a large number of requests that need to use scroll to request data from Elasticsearch, the default maximum number of scroll_id is 500. when the maximum number is reached, some requests do not have scroll_id available, resulting in an error.

This problem may be more common, especially in high concurrency scenarios.

The solution can increase the size of the search.max_open_scroll_context value.

But this is not a good solution, a better way is to clean up the scroll_id in time after the query.

# pythonfrom elasticsearch import Elasticsearchclient = Elasticsearch (host, http_auth= (username, password), timeout=3600) es_data = client.search (es_index, query_body, scroll='1m', size=100) scroll_id= es_ data ['_ scroll_id'] client.clear_scroll (scroll_id=scroll_id) # cleanup method

In fact, even if we don't clean it manually, the cursor will release itself after it expires, which has something to do with the parameters when using it.

For example, scroll='1m' stands for 1min and will be released.

But just like we use other resources, release in time after use, develop good coding habits, the system can be more robust.

At this point, the study of "what are the problems of Elasticsearch paging query" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.