Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to solve j2Cache online anomaly troubleshooting problem", the content is easy to understand, clear, hope to help you solve doubts, the following let the editor lead you to study and learn "how to solve j2Cache online anomaly troubleshooting problem" this article.

Problem background

According to the development feedback, an online service will throw an exception after running for a period of time, causing the redis cache to become unavailable. The project uses j2Caceh, and the exception is thrown by j2Cache's RedisCacheProvider, such as:

Exception in thread "main" redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool at redis.clients.util.Pool.getResource (Pool.java:51) at redis.clients.jedis.JedisPool.getResource (JedisPool.java:99) at net.oschina.j2cache.redis.RedisCacheProvider.getResource (RedisCacheProvider.java:51) at com.xczysoft.ltl.core.support.j2cache.J2CacheRedisCacheChannel.main (J2CacheRedisCacheChannel.java:66) Caused by: java. Util.NoSuchElementException: Timeout waiting for idle object at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject (GenericObjectPool.java:447) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject (GenericObjectPool.java:361) at redis.clients.util.Pool.getResource (Pool.java:49)... 3 more

J2Cache: sweet potato open source 2-stage caching framework: https://gitee.com/ld/J2Cache

Analysis of problems

From the appearance of the exception log, it is obvious that there are no resources in jedis pool. When jedis pool has no resources and the client applies for a connection, the framework reserves a policy controlled by the user to handle it. The specific policies are as follows:

Connection pool parameter: blockWhenExhausted, there are two strategies

True: block wait time for maxWaitMillis (default). This is the default policy. When pool has no available resources, block wait time for maxWaitMillis. This value is infinitely long by default. Connection pool should set an appropriate wait time.

False: fails immediately when no connection is available.

Our service does not set the parameters for whenExhaustedAction, but maxWait sets 1500. That is, when the jedis pool has no available resources, the thread that acquires the connection waits for 1.5 seconds, and throws an exception after 1.5 seconds when no resources are available.

Going back to the above question, what are the reasons for jedis pool? There are no more than two points, as follows:

1. Normal situation: high program concurrency, resulting in sporadic connection pooling with no available resources

2. Exception: the connection pool is used improperly. After obtaining the resources from the connection pool, there is no normal release of the resources when the connection pool is finished. As a result, the connection pool takes one less than one, and the initial exception will be thrown inevitably.

Hypothetical problem

Combined with the analysis of jedis pool, but the concurrency of our service is not high, the maximum number of connections in the default connection pool is 8, and the thread that acquires the connection does not get the thread after waiting for 1.5 seconds. The most important point is that when the program runs to the end, the exception that does not get the connection is no longer accidental.

It becomes an event of necessity, so based on the above analysis, assume that the problem is caused by the improper use of connection pooling in the program. The place where the program uses jedis is j2Cache, an open source 2-level cache framework for sweet potatoes, which is likely to be the pot of sweet potatoes.

Verify carefully.

Through the assumption of the problem, we need to find the code in the program to get resources from jedis pool, then we first need to find the place to initialize the connection pool, where the jedis pool is maintained through RedisCacheProvider in j2Cache. The following is the code in j2Cache to operate redis through the connection of jedis pool. As you can see, it is very standardized, wrapping the operation of resources through try,catch,finally, and releasing resources in finally, ensuring that resources must be released.

Sweet potato means that I can't memorize this pot, so it's definitely not the fault of j2Cache. You can see that after RedisCacheProvider initializes the connection pool, it provides a static method getResource () to obtain the connection, most likely through this entry, the business level gets the connection in the RedisCacheProvider. Later, we continue to look, and locate a very suspicious method, with the following code:

/ * send broadcast command to clear cache * * @ param region: Cache region name * / private void _ sendClearCmd (String region) {/ / send broadcast Command cmd = new Command (Command.OPT_CLEAR_KEY, region, "") Try (Jedis jedis = RedisCacheProvider.getResource ()) {jedis.publish (SafeEncoder.encode (config.getProperty ("redis.channel_name")), cmd.toBuffers ();} catch (Exception e) {log.error ("Unable to clear cache,region=" + region, e);}}

As you can see, this is a piece of code related to j2Cache, but not within the framework of sweet potato, but a piece of code in a cache channel that we configured when we connected to j2Cache. The problem is passing.

After RedisCacheProvider.getResource () gets the jedis object, it finishes using it and does not release it.

Problem recurrence

The above basically locates the problem. Let's simulate the scenario under which the problem occurs. The code logic is similar to the above. After we initialize a connection pool, in a for loop, we simulate to acquire the connection many times but not release it, such as:

Public static void main (String [] args) throws Exception {Properties properties = ResourceUtil.getResourceAsProperties ("app.properties", true); new J2CacheRedisCacheChannel ("j2Cache 666", properties); for (int I = 1; I

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.