Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "what is the Elasticsearch component library". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Introduction of Elasticsearch component library

Before I explain, I would like to mention again what full-text retrieval is:

Full-text retrieval: an information retrieval technology that takes all text information as the object of retrieval. However, the database we use, such as Mysql,MongoDB, is not as powerful as ES in text information retrieval, especially in Chinese. So let's take a look at how ES works instead of SQL in the project.

The Elasticsearch service I use is version 7.4.2, and then I use the official Elastiscsearch-Rest-Client library to operate ES, and the API of the official library is easy to use.

The official document address of the component library:

Https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html

In addition, this component library supports multiple languages:

Support for multilingual

Note: Elasticsearch Clients refers to how to use API to operate the component library of ES services.

Some students may ask, "JavaScript API is written in the component library of Elasticsearch. Can you access ES services directly at the front end?" Yes, but the port and IP address of the ES service will be exposed, which can be very insecure. So we still use the back-end service to access the ES service.

Our project is a Java project, so we naturally use the above two: Java Rest Client or Java API. Let's take a look at Java API first, but we will find that it has been abandoned. As shown in the following figure:

Java API has been abandoned.

So we have to use Java REST Client. And it is divided into two kinds: high-level and low-level. Advanced contains more functions, and if high is compared to MyBatis, then low-level is equivalent to JDBC. So we use the advanced Client.

Advanced and low-level Client

II. Integration of retrieval services

We treat the retrieval service as a separate service. Call it the passjava-search module.

1.1 add search service module

Create a passjava-search module.

First we create a search service module passjava-search in the PassJava-Platform module. Then check the spring web service. This is shown in the following figure.

Step 1: select Spring Initializr, and then click Next.

Select Spring Initializr

Step 2: fill in the module information and click Next.

Passjava-search service module

Step 3: select Web- > Spring Web dependency, and then click Next.

1.2 configure Maven dependencies

Refer to the ES official website for configuration.

When you go to the official website of ES, you can see that there are low-level and high-level Rest Client, and we choose high-level (High Level Rest Client). Then enter the Maven warehouse of the high-end Rest Client. The address on the official website is as follows:

Https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.9/index.html

Rest Client official documentation

Plus Maven dependency.

Corresponding file path:\ passjava-search\ pom.xml

Org.elasticsearch.client elasticsearch-rest-high-level-client 7.4.2

Configure version 7.4.2 of elasticsearch

Due to the addition of Maven dependency, the elasticsearch version is 7.6.2, so when you encounter this version inconsistency, you need to manually change it.

Corresponding file path:\ passjava-search\ pom.xml

7.4.2

After refreshing Maven Project, you can see that the introduced elasticsearch is version 7.4.2, as shown in the following figure:

Set version to 7.4.2

Common module dependencies introduced by PassJava.

Common module is an independent public module of the PassJava project, which introduces a lot of common component dependencies. After other modules introduce Common module dependencies, it is very convenient to introduce these common components separately.

Corresponding file path:\ passjava-search\ pom.xml

Com.jackson0714.passjava passjava-common 0.0.1-SNAPSHOT

After adding the dependencies, we can register the search service with the Nacos registry. The use of the Nacos registry has also been explained in detail in previous articles, and it is important to note that you need to start the Nacos registry before you can register the passjava-search service properly.

1.3 register the search service to the registry

Modify the configuration file: src/main/resources/application.properties. Configure the application name, registry address, and naming middle of the registry.

Spring.application.name=passjava-searchspring.cloud.nacos.config.server-addr=127.0.0.1:8848spring.cloud.nacos.config.namespace=passjava-search

Add service discovery annotations to the startup class: @ EnableDiscoveryClient. So that the passjava-search service can be discovered by the registry.

Because the Common module depends on the data source, but the search module does not, the search module needs to remove the data source dependency:

Exclude = DataSourceAutoConfiguration.class

The above two annotations are as follows:

@ EnableDiscoveryClient@SpringBootApplication (exclude = DataSourceAutoConfiguration.class) public class PassjavaSearchApplication {public static void main (String [] args) {SpringApplication.run (PassjavaSearchApplication.class, args);}}

Next, we add a dedicated configuration class for the ES service, the main purpose of which is to automatically load an ES Client for subsequent ES API use, instead of new one ES Client at a time.

1.4 add ES configuration class

Configuration class: PassJavaElasticsearchConfig.java

The core method is the RestClient.builder method, which can set the IP address, port number and transport protocol of the ES service. Finally, RestHighLevelClient is loaded automatically.

Package com.jackson0714.passjava.search.config;import org.apache.http.HttpHost;import org.elasticsearch.client.RestClient;import org.elasticsearch.client.RestHighLevelClient;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration / * * @ Author: official account | Wukong chat Architecture * @ Date: 17:02 on 2020-10-8 * @ Site: www.passjava.cn * @ Github: https://github.com/Jackson0714/PassJava-Platform * / @ Configurationpublic class PassJavaElasticsearchConfig {@ Bean / / register a RestHighLevelClient for the container Used to operate ES / / refer to the official documentation: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.9/java-rest-high-getting-started-initialization.html public RestHighLevelClient restHighLevelClient () {return new RestHighLevelClient (RestClient.builder (new HttpHost ("192.168.56.10", 9200, "http") }}

Next, let's test whether the ES Client automatically loads successfully.

1.5 Test ES Client auto-load

Write the test method in the test class PassjavaSearchApplicationTests and print out the automatically loaded ES Client. The expected result is a RestHighLevelClient object.

Package com.jackson0714.passjava.search;import org.elasticsearch.client.RestHighLevelClient;import org.junit.jupiter.api.Test;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.beans.factory.annotation.Qualifier;import org.springframework.boot.test.context.SpringBootTest;@SpringBootTestclass PassjavaSearchApplicationTests {@ Qualifier ("restHighLevelClient") @ Autowired private RestHighLevelClient client; @ Test public void contextLoads () {System.out.println (client);}}

The result of the run is shown below, and the RestHighLevelClient is printed. Indicates that the custom ES Client autoload is successful.

ES test results

1.6 Test ES simple insert data

Test the method testIndexData, omitting the User class. The users index is not recorded in my ES, so the expected result is a new piece of users data in ES.

/ * Test stores data to ES. * * / @ Testpublic void testIndexData () throws IOException {IndexRequest request = new IndexRequest ("users"); request.id ("1"); / / document's id / / construct User object User user = new User (); user.setUserName ("PassJava"); user.setAge ("18"); user.setGender ("Man"); / / User object is converted to JSON data String jsonString = JSON.toJSONString (user) / / put JSON data into request request.source (jsonString, XContentType.JSON); / / perform insert operation IndexResponse response = client.index (request, RequestOptions.DEFAULT); System.out.println (response);}

By executing the test method, we can see the following output from the console, indicating that the data was successfully inserted into the ES. It is also important to note that the result field in the result is updated, because I have performed several more insert operations locally in order to take screenshots, but because id = 1, all the operations are updated rather than created.

Console output result

Let's take a look at the data in the users index in ES. Query the users index:

GET users/_search

The results are as follows:

Query users index results

You can see from the figure that a record is queried, and the _ id of the queried data is 1, which is the same as the inserted document id. The values of several other fields are also consistent. Indicates that there is no problem with the inserted data.

"age": "18", "gender": "Man", "userName": "PassJava" 1.7 Test ES query complex statements

Example: search the bank index, and the address field contains the age distribution of all big owners (top 10 items), as well as the average age, as well as the average salary.

1.7.1 construct search conditions

We can follow the example given in the official documentation to create a SearchRequest object, specify the index to be queried as bank, and then create a SearchSourceBuilder to assemble the query conditions. There are three conditions that need to be assembled:

The address contains the owner of the road.

Aggregation was carried out according to age distribution.

Calculate the average salary.

The code is shown below. If you need the source code, please download it on my Github/PassJava.

Example of querying complex statements

Copy out the printed retrieval parameters, then format them in the JSON formatting tool, and then paste them to the ES console for execution. It is found that the execution results are correct.

Print out the retrieval parameters

Format the JSON string with an online tool, and the result is as follows:

Then we remove some of the default parameters, and finally put the simplified retrieval parameters into Kibana.

The search statement executed in the Kibana Dev Tools console is shown below, and the retrieval result is shown below:

Execute the retrieval statement in the console

Total number of records found: 29.

Details of the first hit record are as follows:

Average balance:13136.

Average age: 26.

The address contains Road's: 263 Aviation Road.

It is consistent with the test results performed in IDEA, indicating that the function of complex retrieval has been successfully implemented.

17.2 get the details of the hit record

To obtain the detailed data of the hit record, you need to get it through the getHists () method twice, as shown below:

/ / 3.1) obtain the data found. SearchHits hits = response.getHits (); / / 3.2) get the real hit result SearchHit [] searchHits = hits.getHits ()

We can print out the details of all the hits by traversing the searchHits.

For (SearchHit hit: searchHits) {String hitStr = hit.getSourceAsString (); BankMember bankMember = JSON.parseObject (hitStr, BankMember.class);}

The hitStr that gets each record is JSON data, as shown below:

{"account_number": 431, "balance": 13136, "firstname": "Laurie", "lastname": "Shaw", "age": 26, "gender": "F", "address": "263 Aviation Road", "employer": "Zillanet", "email": "laurieshaw@zillanet.com", "city": "Harmon", "state": "WV"}

BankMember is a JavaBean defined based on the details of the returned results. It can be generated automatically through the tool. The websites that generate JavaBean online are as follows:

Https://www.bejson.com/json2javapojo/new/

Add this JavaBean to the PassjavaSearchApplicationTests class:

@ ToString@Datastatic class BankMember {private int account_number; private int balance; private String firstname; private String lastname; private int age; private String gender; private String address; private String employer; private String email; private String city; private String state;}

Then print out the bankMember:

System.out.println (bankMember)

BankMember

The result is indeed the BankMember object we encapsulated, and we got all the attribute values in it.

1.7.3 get aggregate information of age distribution

In the response returned by ES, the age distribution data is returned in ES format. If you want to return it in our own format, you need to process response.

As shown in the figure below, this is the result of the age distribution of the query, and we need to take out some of the fields, such as buckets, which represents four of the 21-year-olds.

Age distribution information returned by ES

Here is the code implementation:

Aggregations aggregations = response.getAggregations (); Terms ageAgg1 = aggregations.get ("ageAgg"); for (Terms.Bucket bucket: ageAgg1.getBuckets ()) {String keyAsString = bucket.getKeyAsString (); System.out.println ("user age: + keyAsString +" number: "+ bucket.getDocCount ());}

The final printed results are as follows: 4 at the age of 21, 4 at the age of 26, and so on.

Print results: user age distribution

1.7.4 get average salary aggregation information

Now let's see how the average salary is returned in the desired format. The result returned by ES is shown in the figure below. We need to get the value value of the balanceAvg field.

Average salary information returned by ES

Code implementation:

Avg balanceAvg1 = aggregations.get ("balanceAvg"); System.out.println ("average salary:" + balanceAvg1.getValue ())

The printed results are as follows, with an average salary of 28578 yuan.

Print results: average salary

III. Practice: synchronizing ES data 3.1 to define a retrieval model

The PassJava project can be used to configure the item bank. What should we do if we want to search the item bank through keywords?

Similar to Baidu search, enter a few keywords to find the related results, our function is similar, through the Elasticsearch search engine, the background management interface and Mini Program as the search entry, only need to enter keywords on Mini Program, you can retrieve relevant questions and answers.

First of all, we need to save the questions and answers to ES. Before saving, the first step is to define the model of the index, as shown below. There are title and answer fields in the model, which represent the questions and answers.

"id": {"type": "long"}, "title": {"type": "text", "analyzer": "ik_smart"}, "answer": {"type": "text", "analyzer": "ik_smart"}, "typeName": {"type": "keyword"} 3.2 create an index in ES

We have defined the index structure above, and then create the index in ES.

Execute the following statement in the Kibana console:

PUT question {"mappings": {"properties": {"id": {"type": "long"}, "title": {"type": "text", "analyzer": "ik_smart"}, "answer": {"type": "text", "analyzer": "ik_smart"} TypeName: {"type": "keyword"}

The execution result is as follows:

Create a question index

We can see if the question index is in ES with the following command:

GET _ cat/indices

The execution result is shown in the following figure:

View all indexes in ES

3.3.Defining ES model

Above we define the index of ES, then define the model corresponding to the index, store the data in this model, and then store it in ES.

The ES model is as follows, with four fields: id, title, answer, and typeName. And the ES index correspond to each other.

@ Datapublic class QuestionEsModel {private Long id; private String title; private String answer; private String typeName;} 3.4 the time to trigger the save

When we create or save a topic in the background, we save the data to the mysql database and then to ES.

As shown in the following figure, when the administrative background creates a topic, it triggers the saving of data to ES.

The first step is to save the data to mysql. Since this feature is already included in the project, we will no longer explain it, but go directly to the second step: save the data to ES.

To save the data to ES, you need to assemble the data into the data corresponding to the ES index, so I used an ES model to save the data to ES model first.

3.5 assemble data with model

The key code here is copyProperties, which takes the data from the question object and assigns it to the ES model. However, there are some fields in ES model that are not available in question, so you need to pick out and assign values separately, such as the typeName field, which is not found in the question object, and its corresponding field is question.type, so we take out the type and assign it to the typeName field of ES model. As shown in the following figure:

Assemble data with model

3.6 Save data to ES

I wrote an api to save the title in the passjava-search microservice to save the data to ES.

Save data to ES

Then call the search micro-service's save ES method in the passjava-question micro-service.

/ / call the passjava-search service to send the data to ES for storage. SearchFeignService.saveQuestion (esModel); 3. 7 verify that the ES is created successfully

We can view the documents in the question index through the kibana console. View it with the following command:

GET question/_search

The execution result is shown in the following figure, with a record:

In addition, do you have any questions: can you update the title again?

The answer is yes, the data saved to ES is idempotent because it is saved with an id similar to the database primary key.

IV. Practice: querying ES data

We have synchronized the data into ES, and now it is how the front end queries the ES data. Here we still use Postman to simulate the front-end query request.

4.1 define request parameters

I defined three request parameters:

Keyword: used to match a question or answer.

Id: used to match the title id.

PageNum: used for paging query data.

Here I define these three parameters as a class:

@ Datapublic class SearchParam {private String keyword; / / full-text matching keywords private String id; / / title id private Integer pageNum; / / query page data} 4.2 define the return parameters

I also defined four fields for the returned response:

QuestionList: a list of topics found.

PageNum: how many pages of data.

Total: the total number of entries queried.

TotalPages: total number of pages.

The defined classes are as follows:

@ Datapublic class SearchQuestionResponse {private List questionList; / / topic list private Integer pageNum; / / which page of the query data private Long total; / / total number of private Integer totalPages; / / total number of pages} 4.3 assemble ES query parameters

When calling the query API of ES, you need to build the query parameters.

The core code for assembling query parameters is as follows:

Assemble query parameters

Step 1: create a retrieval request.

Step 2: set which fields need to be blurred. There are three fields: title,answer,typeName.

Step 3: set up how to page. The page size here is five.

Step 4: call the query api.

4.4 the result returned by formatting ES

The data returned by ES is in the format defined by ES, and the real data is nested in the response of ES, so you need to format the returned data.

The core code is shown in the following figure:

Formatting ES returns the result

Step 1: get the data found.

Step 2: get the result of the real hit.

Step 3: format the returned data.

Step 4: assemble the paging parameters.

4.5Test ES query 4.5.1 Lab 1: test title matching

We now want to verify that the title field matches the request parameter keyword = 111and matches the data with title = 111and there is only one entry. The page number pageNum I passed is 1, indicating that the first page of data is returned. As shown in the following figure:

Test matching title

4.5.2 Lab 2: test answer matching

We now want to verify whether the answer field can match. The request parameter keyword = test answer passed matches the data of title = test answer, and there is only one entry, indicating that the query is successful. As shown in the following figure:

Test matching answer

4.5.2 Lab 3: test id matching

If we want to match the title id, we need to pass the request parameter id, and the id is an exact match. In addition, id and keyword are union, so the keyword field cannot be passed.

The request parameter id = 5 and the returned result is also the data with id = 5, which means the query is successful. As shown in the following figure:

Test id match

This is the end of the content of "what is Elasticsearch component Library". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.