Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to optimize the redis batch query interface", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "how to optimize the redis batch query interface".

Retrieve the data from OpenSearch, then populate and assemble the data, and finally return

The logic seems very simple. I thought so at first, so I estimated that it would be completed in 5 days. Finally, it took almost 10 days to develop, debug, and change bug until it was launched (of course, these 10 days are not the only thing to do).

The complexity is that there are many factors that affect the return structure. To solve the problem, you need to check the configuration, check the database, check the cache, check the OpenSearch, and check the code.

To get to the point, no matter how complex the logic is, it is not an interface for you to evade the problem, let alone a reason for you not to optimize. This is not the focus of this article, but the optimization process.

Required. The interface provided to APP generally requires that the response time be within 100ms.

The first pressure test

Horrible, the average response time 150ms, and in this stress test process also found other problems, the background error, the check is the number of OpenSearch queries per second limit

Optimize code and configuration

1. Modify the OpenSearch configuration and change the OpenSearch connection address in the pressure test environment to the private network address.

2. Change the cyclic query cache in the code to one-time batch query return.

3. Get rid of the useless code in the project after confirming with the relevant students

The second pressure test

Although the code has been optimized and the configuration has been modified, the situation has become worse and new problems have been changed.

At that time, the code was checked repeatedly to make sure that the number of query caches was minimal, and the parameters related to the connection thread pool were adjusted to a relatively large and reasonable value.

If the stress test still fails to meet the requirements, there is only one last trick: cache the result set

That is, the user ID and the keyword searched by the user are key, and the result of the query is value, cached for 5 minutes

The third pressure test

Finally met the requirements, the response time reached 32ms at 60, and I found a new optimization point.

Unexpectedly, there is still the operation of checking the database in the interface, which is unbearable. After the investigation, some unnecessary dependencies have been removed.

Growth

Learned to use RedisTemplate's executePipelined to do redis batch queries.

A summary of this optimization

1. Absolutely avoid cycling the database and cache (PS: there can be no query cache in the loop, let alone the operation of querying the database, because the number of loops cannot be controlled)

2. For API interfaces, the cache is usually checked directly, but not the database.

3. Use more batch queries, less single queries, and try to find out at once

4. For using Aliyun, you should pay attention to the configuration of the corresponding product, and you still have to spend the money. At the same time, you must remember to use the private network address of the corresponding product in the formal environment.

5. Pay attention to the size of connection pool (including database connection pool, Redis cache connection pool, thread pool)

6. Do not deploy other services on the stress testing machine, but only run the services to be tested to avoid being affected by other items. For online environments, it is best to deploy only one important service on one machine.

7. Useless and commented-out code, useless dependencies had better be cleaned up in time

8. The cluster needless to say

9. Some monitoring tools can help us better locate problems, such as link tracking. PinPoint is used in this project.

10. If the room for technical optimization is already very small, you can try to start with the business, speak with the actual data, and convince the test from the daily and historical traffic data.

11. Every code change may introduce new problems, so every time you modify the code, you have to go back and test it (PS: after each modification, I will search with several different sets of keywords, and then compare whether the data returned before and after the modification are consistent, then postman and Beyond compare will come in handy.)

12. more logs must be added in key places to facilitate troubleshooting in the future, because the most important way to troubleshoot online problems is to rely on logs

At this point, I believe you have a deeper understanding of "how to optimize the redis batch query interface". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.