Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is the principle of Elasticsearch Multi Get and Bulk API". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

This article will introduce batch acquisition API (Multi Get API) and Bulk API in detail.

1 、 Multi Get API

Public final MultiGetResponse mget (MultiGetRequest multiGetRequest, RequestOptions options) throws IOException

Public final void mgetAsync (MultiGetRequest multiGetRequest, RequestOptions options, ActionListener listener)

Its core needs to focus on MultiGetRequest.

As we know above, mget and batch acquisition of documents, add multiple Item through add method, each item represents a file acquisition request, and its related fields have been described in detail in get API, so I won't go into too much detail here.

Example of using Mget API

Public static void testMget () {RestHighLevelClient client = EsClient.getClient (); try {MultiGetRequest request = new MultiGetRequest (); request.add ("twitter", "_ doc", "10"); request.add ("twitter", "_ doc", "11"); request.add ("twitter", "_ doc", "12") Request.add ("gisdemo", "_ doc", "10"); MultiGetResponse result = client.mget (request, RequestOptions.DEFAULT); System.out.println (result);} catch (Throwable e) {e.printStackTrace ();} finally {EsClient.close (client);}}

The returned result is essentially an array of GetResponse, and the entire request will not fail because one of them fails, but the result will indicate whether each is successful or not. The returned result class diagram is as follows:

Its field filtering (Source filtering), routing and other mechanisms are the same as Get API, so it will not be explained repeatedly.

2. Detailed explanation of Bluk API

Bulk API can include multiple index operations in one API call, such as updating the index, deleting the index, and so on. Its API is defined as follows:

Public final BulkResponse bulk (BulkRequest bulkRequest, RequestOptions options) throws IOException

Public final void bulkAsync (BulkRequest bulkRequest, RequestOptions options, ActionListener listener)

Its core needs to focus on BulkRequest.

2.1BulkRequest detailed explanation

List requests: a single command container. Subclasses of DocWriteRequest include: IndexRequest, UpdateRequest, DeleteRequest.

The index involved in private final Set indices:requests.

List payloads: payload, version 6.4.0. It seems that this field doesn't make much sense. Usually, the request body (payload data) of a command is stored in a DocWriteRequest object, such as the source field of IndexRequest.

Protected TimeValue timeout:timeout mechanism, which takes effect for a Bulk request.

ActiveShardCount waitForActiveShards: valid for the entire Bulk request.

Private RefreshPolicy refreshPolicy = RefreshPolicy.NONE: refresh policy.

Private long sizeInBytes = 0: the size of the entire Bulk request.

Add a request for BulkRequest through add api.

2.2 detailed explanation of Bulk API request format

The Bulk Rest request protocol is based on the following format:

POST _ bulk {"index": {"_ index": "test", "_ type": "_ doc", "_ id": "1"} {"field1": "value1"} {"delete": {"_ index": "test", "_ type": "_ doc" "_ id": "2"} {"create": {"_ index": "test", "_ type": "_ doc", "_ id": "3"} {"field1": "value3"} {"update": {"_ id": "1", "_ type": "_ doc" "_ index": "test"} {"doc": {"field2": "value2"}}

The request format is defined as follows (restfull):

POST request whose Content-Type is application/x-ndjson.

Each command occupies two lines, with the end character of each line\ r\ n.

First behavior metadata, "opType": {metadata}.

The second behavior payload (optional), such as the Index operation, whose payload is the IndexRequest#source field.

Available values for opType: index, create, update, delete.

Common metadata (index, create, update, delete) is as follows

1) _ index: index name

2) _ type: type name

3) _ id: document ID

4) routing: route valu

5) parent

6) version: data version number

7) version_type: version type

Operation-specific metadata

1. Index | create

1) pipeline

2 、 update

1) retry_on_conflict: the number of retries in case of update conflicts.

2) _ source: field filtering.

Payload description

1. Index | create

Its payload is the _ source field.

2 、 update

Its payload is partial doc, upsert and script.

3 、 delete

There is no payload.

The main reason why the request format should be designed as an effective carrier of metdata+ is that at the receiving end node (the so-called receiving end node refers to the first node that receives the command), it is only necessary to parse the metadata and then forward the request directly to the corresponding data node.

2.3 Analysis of the general characteristics of bulk API

2.3.1 version Management

Each Bulk entry has its own version, which exists in the metadata of the item of the request entry.

2.3.2 routing

Each Bulk entry takes effect separately.

2.3.3 Wait For Active Shards

You can usually set BulkRequest#waitForActiveShards to require the minimum number of copies that Bulk needs to be active before batch execution.

2.3.4 Bulk Demo

Public static final void testBulk () {RestHighLevelClient client = EsClient.getClient (); try {IndexRequest indexRequest = new IndexRequest ("twitter", "_ doc", "12") .source (buildTwitter ("dingw", "2009-11-18T14:12:12", "test bulk")) UpdateRequest updateRequest = new UpdateRequest ("twitter", "_ doc", "11") .doc (new IndexRequest ("twitter", "_ doc", "11") .source (buildTwitter ("dingw", "2009-11-18T14:12:12", "test bulk update")); BulkRequest request = new BulkRequest () Request.add (indexRequest); request.add (updateRequest); BulkResponse bulkResponse = client.bulk (request, RequestOptions.DEFAULT); for (BulkItemResponse bulkItemResponse: bulkResponse) {if (bulkItemResponse.isFailed ()) {BulkItemResponse.Failure failure = bulkItemResponse.getFailure (); System.out.println (failure); continue } DocWriteResponse itemResponse = bulkItemResponse.getResponse (); if (bulkItemResponse.getOpType ()) = = DocWriteRequest.OpType.INDEX | | bulkItemResponse.getOpType () = = DocWriteRequest.OpType.CREATE) {IndexResponse indexResponse = (IndexResponse) itemResponse; System.out.println (indexRequest) } else if (bulkItemResponse.getOpType () = = DocWriteRequest.OpType.UPDATE) {UpdateResponse updateResponse = (UpdateResponse) itemResponse; System.out.println (updateRequest);} else if (bulkItemResponse.getOpType () = = DocWriteRequest.OpType.DELETE) {DeleteResponse deleteResponse = (DeleteResponse) itemResponse; System.out.println (deleteResponse) } catch (Exception e) {e.printStackTrace ();} finally {EsClient.close (client);}} "what is the principle of Elasticsearch Multi Get and Bulk API"? thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.