What if the application gets stuck due to improper use of Redis?
This article introduces the relevant knowledge of "what to do if the application gets stuck due to improper use of Redis". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!
First of all, let's talk about the problem: the API in the intranet sandbox environment has been jammed for one week, and all api are unresponsive.
At first, when the test complained about the slow response of the environment, we restarted the application and the application returned to normal, so we didn't do anything about it. But then the problem occurred more and more frequently, and more and more colleagues began to complain, so they felt that there might be something wrong with the code and began to troubleshoot.
First of all, it is found that there is no problem with the local ide developed, the database and redis are normal when the application is stuck, and there is no special error log. Began to suspect that it was the sandbox environment machine problem (the test environment itself is brittle!)
So ssh went to the server and executed the following command
Top
At this time, I found that the machine was normal, but it was still inside, so I planned to take a look at the jvm stack information.
First, let's take a look at the resource-consuming threads of the problem application.
Execute top-H-p 12798
Find the first three relatively resource-consuming threads
Jstack view heap memory
Jstack 12798 | hexadecimal 31ff of grep 12799
I don't see any problem, so let's take a look at the top and bottom 10 lines and execute it.
See that some threads are in the lock state. But there is no business-related code, ignored. I don't have a clue at this time. Think about it. The machine that decided to give up this jammed state
In order to protect the accident site, dump all the heap memory of the problem process first, then restart the test environment application in debug mode, and directly remotely debug the problem machine when the problem is ready to reappear.
The problem reappeared the next day, so notify the operation and maintenance nginx to forward and remove the problem application and remotely debug tomcat yourself.
I randomly found an interface, and the breakpoint was at the entrance of the interface. When the tragedy began, nothing happened! API waits for the service to respond and does not enter the breakpoint. At this time a little confused, calm for a while, in the entrance before the aop place under a breakpoint, debug again, this time into the breakpoint, f8 N times later found in the execution of the redis command card owner. Continue to follow and finally find a problem in a place where you go to jedis:
/ * *
* Returns a Jedis instance to be used as a Redis connection. The instance can be newly created or retrieved from a
* pool.
*
* @ return Jedis instance ready for wrapping into a {@ link RedisConnection}.
, /
Protected Jedis fetchJedisConnector () {
Try {
If (usePool & & pool! = null) {
Return pool.getResource ()
}
Jedis jedis = new Jedis (getShardInfo ())
/ / force initialization (see Jedis issue # 82)
Jedis.connect ()
Return jedis
} catch (Exception ex) {
Throw new RedisConnectionFailureException ("Cannot get Jedis connection", ex)
}
}
The thread starts wait after pool.getResource () above
Public T getResource () {
Try {
Return internalPool.borrowObject ()
} catch (Exception e) {
Throw new JedisConnectionException ("Could not get a resource from the pool", e)
}
}
Return internalPool.borrowObject (); this code should be a rental code followed by
Public T borrowObject (long borrowMaxWaitMillis) throws Exception {
This.assertOpen ()
AbandonedConfig ac = this.abandonedConfig
If (ac! = null & & ac.getRemoveAbandonedOnBorrow () & & this.getNumIdle ()
< 2 && this.getNumActive() >This.getMaxTotal ()-3) {
This.removeAbandoned (ac)
}
PooledObject p = null
Boolean blockWhenExhausted = this.getBlockWhenExhausted ()
Long waitTime = 0L
While (p = = null) {
Boolean create = false
If (blockWhenExhausted) {
P = (PooledObject) this.idleObjects.pollFirst ()
If (p = = null) {
Create = true
P = this.create ()
}
If (p = = null) {
If (borrowMaxWaitMillis < 0L) {
P = (PooledObject) this.idleObjects.takeFirst ()
} else {
WaitTime = System.currentTimeMillis ()
P = (PooledObject) this.idleObjects.pollFirst (borrowMaxWaitMillis, TimeUnit.MILLISECONDS)
WaitTime = System.currentTimeMillis ()-waitTime
}
}
If (p = = null) {
Throw new NoSuchElementException ("Timeout waiting for idle object")
}
There is a code if (p = = null) {
If (borrowMaxWaitMillis < 0L) {
P = (PooledObject) this.idleObjects.takeFirst ()
} else {
WaitTime = System.currentTimeMillis ()
P = (PooledObject) this.idleObjects.pollFirst (borrowMaxWaitMillis, TimeUnit.MILLISECONDS)
WaitTime = System.currentTimeMillis ()-waitTime
}
}
BorrowMaxWaitMillis
