Shulou(Shulou.com)06/01 Report--

Editor to share with you what are the abnormal points for attention when using returnResource thread pool. I hope you will get something after reading this article. Let's discuss it together.

The online environment found an abnormal termination of a worker thread. Look at the log. First there is some SocketTimeoutException, then suddenly there is a ClassCastException

Redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out...java.lang.ClassCastException: [B cannot be cast to java.lang.Long at redis.clients.jedis.Connection.getIntegerReply (Connection.java:208) at redis.clients.jedis.Jedis.sismember (Jedis.java:1307)

After simulating the abnormal situation of the network manually, the online anomaly is finally reproduced. After in-depth analysis (putting forward the hypothesis-> verifying the hypothesis), we finally find out the cause of this problem. See the following sample code:

JedisPool pool =...; Jedis jedis = pool.getResource (); String value = jedis.get ("foo"); System.out.println ("Make SocketTimeoutException"); System.in.read (); / / waiting for manufacturing SocketTimeoutExceptiontry {value = jedis.get ("foo"); System.out.println (value);} catch (JedisConnectionException e) {e.printStackTrace ();} System.out.println ("Recover from SocketTimeoutException"); System.in.read () / / wait for the recovery of Thread.sleep (5000); / / continue to sleep for a period of time and wait for the network to fully recover boolean isMember = jedis.sismember ("urls", "baidu.com")

And log output:

BarMake SocketTimeoutExceptionredis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed outRecover from SocketTimeoutException at redis.clients.util.RedisInputStream.ensureFill (RedisInputStream.java:210) at redis.clients.util.RedisInputStream.readByte (RedisInputStream.java:47) at redis.clients.jedis.Protocol.process (Protocol.java:131) at redis.clients.jedis.Protocol.read (Protocol.java:196) at redis.clients.jedis.Connection.readProtocolWithCheckingBroken (Connection.java:283) at redis .clients.jedis.Connection.getBinaryBulkReply (Connection.java:202) at redis.clients.jedis.Connection.getBulkReply (Connection.java:191) at redis.clients.jedis.Jedis.get (Jedis.java:101) at com.tcl.recipevideohunter.JedisTest.main (JedisTest.java:23) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0 (Native Method) at java.net.SocketInputStream.read (SocketInputStream.java:152) at java.net .SocketInputStream.read (SocketInputStream.java:122) at java.net.SocketInputStream.read (SocketInputStream.java:108) at redis.clients.util.RedisInputStream.ensureFill (RedisInputStream.java:204)... 8 moreException in thread "main" java.lang.ClassCastException: [B cannot be cast to java.lang.Long at redis.clients.jedis.Connection.getIntegerReply (Connection.java:208) at redis.clients.jedis.Jedis.sismember (Jedis.java:1307) at com.tcl .uplopevideohunter.JedisTest.main (JedisTest.java:32)

Analysis:

When the second get ("foo") is executed, the network times out and no get foo command is actually sent. When sismember is executed, the network has returned to normal and is the same jedis instance, so the previous get foo command (which is already in the output stream cache) is sent together.

The order of execution is as follows:

127.0.0.1 get foo 9379 > get foo "bar" 127.0.0.1 sismember urls baidu.com (integer) 1127.0.1 sismember urls baidu.com > get foo "bar" 127.0.0.1

So the result of the final sismember in the above sample code is the result of get foo, that is, a string, while sismember requires a long type, which leads to ClassCastException.

Why is this problem online? The reason is that the logic for performing redis is similar to this:

While (true) {Jedis jedis = null; try {jedis = pool.getResource (); / / some redis operation here. } catch (Exception e) {logger.error (e);} finally {pool.returnResource (jedis);}}

Because if the network is abnormal, pool.returnResource (jedis) can still be executed successfully, that is, it can be returned to the pool (jedis is not empty at this time). After the network is restored, it is a multithreaded environment, which causes some other thread to get the same Jedis instance (pool.getResource ()).

If the return type of the jedis operation in this thread does not match the return type of the first unsuccessful jedis operation of the jedis instance during the network exception (for example, one is get and the other is sismember), a ClassCastException exception will occur.

Fortunately, I can't imagine if it returns the same type (such as lpop ("queue_order_pay_failed"), lpop ("queue_order_pay_success").

If you insert a get ("nonexist-key") before the sismember in the above sample code (the key does not exist in the redis, it should return empty).

Value = jedis.get ("nonexist-key"); System.out.println (value); boolean isMember = jedis.sismember ("urls", "baidu.com"); System.out.println (isMember)

The actual log output is:

BarException in thread "main" java.lang.NullPointerException at redis.clients.jedis.Jedis.sismember (Jedis.java:1307) at com.tcl.recipevideohunter.JedisTest.main (JedisTest.java:37)

Analysis:

Get ("nonexist-key") gets the result of the previous get ("foo"), while sismember gets the result of get ("nonexist-key"), and get ("nonexist-key") returns null, so the null pointer exception is reported.

Solution: you can't use returnResource regardless of the situation. A more robust, reliable and elegant approach is as follows:

While (true) {Jedis jedis = null; boolean broken = false; try {jedis = jedisPool.getResource (); return jedisAction.action (jedis); / / template method} catch (JedisException e) {broken = handleJedisException (e); throw e;} finally {closeResource (jedis, broken);}} / * * Handle jedisException, write log and return whether the connection is broken. * / protected boolean handleJedisException (JedisException jedisException) {if (jedisException instanceof JedisConnectionException) {logger.error ("Redis connection" + jedisPool.getAddress () + "lost.", jedisException) } else if (jedisException instanceof JedisDataException) {if ((jedisException.getMessage ()! = null) & & (jedisException.getMessage (). IndexOf ("READONLY")! =-1) {logger.error ("Redis connection" + jedisPool.getAddress () + "are read-only slave.", jedisException);} else {/ / dataException, isBroken=false return false }} else {logger.error ("Jedis exception happen.", jedisException);} return true;} / * * Return jedis connection to the pool, call different return methods depends on the conectionBroken status. * / protected void closeResource (Jedis jedis, boolean conectionBroken) {try {if (conectionBroken) {jedisPool.returnBrokenResource (jedis);} else {jedisPool.returnResource (jedis);}} catch (Exception e) {logger.error ("return back jedis failed, will fore close the jedis.", e); JedisUtils.destroyJedis (jedis);}}

Add:

Ubuntu local simulation access redis network timeout:

Sudo iptables-An INPUT-p tcp-- dport 6379-j DROP

Restore the network:

Sudo iptables-F

Add:

If the jedis operation logic is similar to that shown below

Jedis jedis = null;try {jedis = jedisSentinelPool.getResource (); return jedis.get (key);} catch (JedisConnectionException e) {jedisSentinelPool.returnBrokenResource (jedis); logger.error (", e); throw e;} catch (Exception e) {logger.error (", e); throw e;} finally {jedisSentinelPool.returnResource (jedis);}

If a JedisConnectionException occurs, such as a network exception, returnBrokenResource will be executed first, and the jedis has already been destroy. When you enter finally and execute returnResource again, an error will be reported:

Redis.clients.jedis.exceptions.JedisException: Could not return the resource to the pool at redis.clients.util.Pool.returnResourceObject (Pool.java:65) at redis.clients.jedis.JedisSentinelPool.returnResource (JedisSentinelPool.java:221)

Temporary solution:

JedisSentinelPool.returnBrokenResource (jedis); jedis=null; / / does not actually perform the relevant actions in returnResource at this time

However, this is not recommended, and a more rigorous method of releasing resources is described above.

After reading this article, I believe you have a certain understanding of "what are the abnormal precautions of using returnResource thread pool?". If you want to know more about it, welcome to follow the industry information channel. Thank you for your reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.