Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to deal with repeated requests / concurrent requests". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought. Let's study and learn how to deal with repeated requests / concurrent requests.

Use the unique request number to remove duplicates

You might think that as long as the request has a unique request number, you can use Redis to do this-- as long as the unique request number exists in redis and proves that it has been processed, it is considered duplicate.

The code is roughly as follows:

String KEY = "REQ12343456788"; / / the unique number of the request long expireTime = 1000X / 1000 milliseconds expires, and the repeated request in 1000ms will be considered as duplicate long expireAt = System.currentTimeMillis () + expireTime; String val = "expireAt@" > business parameter.

The above solution can solve the scenario with a unique request number. For example, before each request is written, the server returns a unique number to the client, and the client uses this request number to make the request, and the server can complete the re-interception.

However, in many scenarios, the request does not have such a unique number! So can we use the parameter of the request as the identity of a request?

Consider a simple scenario first. Assuming that the request parameter has only one field, reqParam, we can use the following flag to determine whether the request is repeated. User ID: API name: request parameter

String KEY = "dedup:U=" + userId + "M =" + method + "P =" + reqParam

So when the same user visits the same interface and comes with the same reqParam, we can locate that he is duplicated.

But the problem is that our interface is not usually that simple, and in the current mainstream, our parameter is usually a JSON. So for this kind of scene, how can we repeat it?

A summary of the calculation request parameters is used as the parameter identification

Suppose we sort the request parameters (JSON) in ascending order by KEY, sort them and spell them into a string as the key value? But this can be very long, so we can consider getting a summary of the string with a MD5 as an argument, replacing the position of reqParam with this summary.

String KEY = "dedup:U=" + userId + "M =" + method + "P =" + reqParamMD5

In this way, the unique logo of the request is typed!

Note: MD5 may be repeated in theory, but deduplication is usually deduplicated in a short time window (for example, one second). It is almost impossible for the same user interface to spell out different parameters to cause the same MD5 in a short time.

Continue to optimize and consider eliminating some time factors

The above problem is actually a very good solution, but when it is actually put into use, you may find some problems: some request users click repeatedly in a short period of time (for example, sent three requests in 1000 milliseconds). But bypassed the above de-rejudgment (different key values).

The reason is that there is a time field in the fields of these request parameters, which marks the time of the user's request, which allows the server to discard some old requests (for example, 5 seconds ago). As in the following example, the other parameters of the request are the same, except that the request time is one second apart:

/ / the two requests are the same, but the request time difference is String req = "{\ n" + "\" requestTime\ ":\" 20190101120001\ ",\ n" + "\" requestValue\ ":\" 1000\ ",\ n" + "\" requestKey\ ":\" key\ "\ n" + "}" String req2 = "{\ n" + "\" requestTime\ ":\" 20190101120002\ ",\ n" + "\" requestValue\ ":\" 1000\ ",\ n" + "requestKey\":\ "key\"\ n "+"} "

For this kind of request, we may also need to block subsequent repeated requests. Therefore, this kind of time field needs to be removed before obtaining the summary of business parameters. A similar field may be the latitude and longitude field of GPS (there may be a slight difference between repeated requests).

Request deduplication tool class. Java implements the parameters of public class ReqDedupHelper {/ * * @ param reqJSON request. Here, it usually refers to the fields to be removed from the JSON * @ param excludeKeys request parameters and then the MD5 summary of * @ return removal parameters * / public String dedupParamMD5 (final String reqJSON, String...) ExcludeKeys) {String decreptParam = reqJSON; TreeMap paramTreeMap = JSON.parseObject (decreptParam, TreeMap.class); if (roomdeKeyscards null) {List dedupExcludeKeys = Arrays.asList (excludeKeys); if (! dedupExcludeKeys.isEmpty ()) {for (String dedupExcludeKey: dedupExcludeKeys) {paramTreeMap.remove (dedupExcludeKey) } String paramTreeMapJSON = JSON.toJSONString (paramTreeMap); String md5deDupParam = jdkMD5 (paramTreeMapJSON); log.debug ("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString (excludeKeys), paramTreeMapJSON); return md5deDupParam;} private static String jdkMD5 (String src) {String res = null Try {MessageDigest messageDigest = MessageDigest.getInstance ("MD5"); byte [] mdBytes = messageDigest.digest (src.getBytes ()); res = DatatypeConverter.printHexBinary (mdBytes);} catch (Exception e) {log.error ("", e);} return res;}}

Here are some test logs:

Public static void main (String [] args) {/ / two requests are the same, but the request time difference is one second String req = "{\ n" + "\" requestTime\ ":\" 20190101120001\ ",\ n" + "\" requestValue\ ":\" 1000\ ",\ n" + "\" requestKey\ ":\" key\ "\ n"} " String req2 = "{\ n" + "\" requestTime\ ":\" 20190101120002\ ",\ n" + "\" requestValue\ ":\" 1000\ ",\ n" + "requestKey\":\ "key\"\ n "+"} "; / / full parameter alignment, so the two parameters MD5 are different String dedupMD5 = new ReqDedupHelper (). DedupParamMD5 (req) String dedupMD52 = new ReqDedupHelper (). DedupParamMD5 (req2); System.out.println ("req1MD5 =" + dedupMD5+ ", req2MD5=" + dedupMD52); / / removal time parameter comparison, MD5 same String dedupMD53 = new ReqDedupHelper (). DedupParamMD5 (req, "requestTime"); String dedupMD54 = new ReqDedupHelper (). DedupParamMD5 (req2, "requestTime"); System.out.println ("req1MD5 =" + dedupMD53+ ", req2MD5=" + dedupMD54);}

Log output:

Req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267, req2MD5=A2D20BAC78551C4CA09BEF97FE468A3Freq1MD5 = C2A36FED15128E9E878583CAAAFEFDE9, req2MD5=C2A36FED15128E9E878583CAAAFEFDE9

Log description:

At the beginning, the two parameters are different because the requestTime is different, so when you remove the duplicate parameter summary, you can find that the two values are different.

On the second call, the requestTime is removed ("requestTime" is passed in the second parameter), and it is found that the two feeds are the same, as expected.

Summary

At this point, we can get a complete de-duplication solution, as follows:

String userId= "12345678"; / user String method = "pay"; / / API name String dedupMD5 = new ReqDedupHelper (). DedupParamMD5 (req, "requestTime"); / / calculate the summary of request parameters, in which the interference of request time String KEY = "dedup:U=" + userId + "M =" + method + "P =" + dedupMD5;long expireTime = 1000 exchange / 1000 millisecond expiration will be considered as repeated requests within 1000ms long expireAt = System.currentTimeMillis () + expireTime;String val = "expireAt@" + expireAt / / NOTE: direct SETNX does not support expiration time, so setting + expiration is not an atomic operation. In extreme cases, it may not expire if it is set. Later, the same request may be mistaken for need to be duplicated, so the underlying API is used here. Ensure that the expiration time of SETNX+ is atomic operation Boolean firstSet = stringRedisTemplate.execute ((RedisCallback) connection-> connection.set (KEY.getBytes (), val.getBytes (), Expiration.milliseconds (expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)) Final boolean isConsiderDup;if (firstSet! = null & & firstSet) {isConsiderDup = false;} else {isConsiderDup = true;} Thank you for your reading. The above is the content of "how to deal with repeated requests / concurrent requests". After the study of this article, I believe you have a deeper understanding of how to deal with repeated requests / concurrent requests, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.