Shulou(Shulou.com)05/31 Report--

This article introduces how the process of application stuck bugs caused by improper use of Redis is very detailed. Interested friends can refer to it and hope to help everyone.

First of all, let's talk about the problem phenomenon: the application of the intranet sandbox environment API has been stuck for 1 week, and all APIs have no response phenomenon.

At first, when the test complained that the environment was slow, we restarted the application and the application returned to normal, so we did nothing. But then the frequency of problems became more and more frequent, and more and more colleagues began to complain, so they felt that there might be problems with the code and began to investigate.

First of all, no problems were found in the native ide developed, the database and redis were normal when the application card died, and there were no special error logs. Start to suspect that it is a sandbox environment machine problem (the test environment itself is very fragile!_!)

So ssh goes to the server and executes the following command

top

At this time found that the machine is still normal, so I intend to look at the jvm stack information

Let's look at the problem first. Apply more resource-intensive threads.

Execute top-H-p 12798

Find the first 3 relatively expensive threads

jstack View heap memory

jstack 12798 |grep 12799 hexadecimal 31ff

I didn't see any problems. I also looked at the upper and lower 10 lines, so I executed them.

See some threads are in lock state. However, no business-related code appears and is ignored. There was no clue at the moment. Think about it. Decided to give up this stuck machine

In order to protect the accident site, dump all heap memory of the problem process, and then restart the test environment in debug mode. Apply it directly to remotely debug the problem machine when the problem appears again.

The next day, the problem reappeared, so he notified the operation and maintenance nginx to forward and remove the problem application, and debugged tomcat remotely.

He randomly found an interface, and the breakpoint was at the interface entrance. The tragedy began, and nothing happened! API waits for service response, no breakpoint entered. At this time a little confused, calm down for a while, in front of the entrance aop place under a breakpoint, debug again, this time into the breakpoint, f8N times found in the execution of the redis command stuck in the main. Continue to follow, and finally find a problem in a place to jedis:

/** * Returns a Jedis instance to be used as a Redis connection. The instance can be newly created or retrieved from a * pool. * * @return Jedis instance ready for wrapping into a {@link RedisConnection}. */ protected Jedis fetchJedisConnector() { try { if (usePool && pool != null) { return pool.getResource(); } Jedis jedis = new Jedis(getShardInfo()); // force initialization (see Jedis issue #82) jedis.connect(); return jedis; } catch (Exception ex) { throw new RedisConnectionFailureException("Cannot get Jedis connection", ex); } }

After pool.getResource() above, the thread starts waiting

public T getResource() { try { return internalPool.borrowObject(); } catch (Exception e) { throw new JedisConnectionException("Could not get a resource from the pool", e); } }

return internalPool.BorrowObject(); This code should be a lease code followed by

public T borrowObject(long borrowMaxWaitMillis) throws Exception { this.assertOpen(); AbandonedConfig ac = this.abandonedConfig; if (ac != null && ac.getRemoveAbandonedOnBorrow() && this.getNumIdle()

this.getMaxTotal() - 3) { this.removeAbandoned(ac); } PooledObject p = null; boolean blockWhenExhausted = this.getBlockWhenExhausted(); long waitTime = 0L; while(p == null) { boolean create = false; if (blockWhenExhausted) { p = (PooledObject)this.idleObjects.pollFirst(); if (p == null) { create = true; p = this.create(); } if (p == null) { if (borrowMaxWaitMillis < 0L) { p = (PooledObject)this.idleObjects.takeFirst(); } else { waitTime = System.currentTimeMillis(); p = (PooledObject)this.idleObjects.pollFirst(borrowMaxWaitMillis, TimeUnit.MILLISECONDS); waitTime = System.currentTimeMillis() - waitTime; } } if (p == null) { throw new NoSuchElementException("Timeout waiting for idle object"); }

其中有段代码

if (p == null) { if (borrowMaxWaitMillis < 0L) { p = (PooledObject)this.idleObjects.takeFirst(); } else { waitTime = System.currentTimeMillis(); p = (PooledObject)this.idleObjects.pollFirst(borrowMaxWaitMillis, TimeUnit.MILLISECONDS); waitTime = System.currentTimeMillis() - waitTime; } }

borrowMaxWaitMillis

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.