Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to deeply understand batched_reduce_size. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

In the search process, after receiving the query request from the client, the coordinator node forwards the request to other data nodes. In this process, the query request will be sent in the unit of shard, and all requests will be processed in the next step after receiving the response (QuerySearchResult).

The batched_reduce_size parameter is used to control the execution of a reduce during the Search process when the coordinator node receives the specified result.

The official website explains him as follows:

(Optional, integer) The number of shard results that should be reduced at once on the coordinating node. This value should be used as a protection mechanism to reduce the memory overhead per search request if the potential number of shards in the request can be large. Defaults to 512.

This explanation seems difficult to understand, so refer to the code and reinterpret the meaning of this parameter: after the coordinator node sends the query request in parallel, when the response QuerySearchResult received reaches the number of batched_reduce_size, execute reduce,reduce to perform the following operations:

If the request contains an aggregation, the aggregation is performed on the batch of results

If the TopN needs to be calculated in the request, the calculation is performed on the batch of results

So the main purpose of this parameter is to prevent the orchestrating node from taking up too much memory in order to summarize all the results.

The main implementation code is as follows. BufferSize is the value set by batched_reduce_size. The default is 512.

Private synchronized void consumeInternal (QuerySearchResult querySearchResult) {

If (index = = bufferSize) {

If (hasAggs) {

InternalAggregations reducedAggs = InternalAggregations.reduce (Arrays.asList (aggsBuffer), reduceContext)

Arrays.fill (aggsBuffer, null)

AggsBuffer [0] = reducedAggs

}

If (hasTopDocs) {

TopDocs reducedTopDocs = mergeTopDocs (Arrays.asList (topDocsBuffer))

QuerySearchResult.from () + querySearchResult.size (), 0)

Arrays.fill (topDocsBuffer, null)

TopDocsBuffer [0] = reducedTopDocs

}

NumReducePhases++

Index = 1

}

Final int i = index++

If (hasAggs) {

AggsBuffer [I] = (InternalAggregations) querySearchResult.consumeAggs ()

}

If (hasTopDocs) {

TopDocsBuffer [I] = topDocs.topDocs

}

}

The above is the editor for you to share how to understand batched_reduce_size in depth, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.