Shulou(Shulou.com)05/31 Report--

This article is to share with you about Arthas how to achieve CPU troubleshooting and hot code updates, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article. Let's take a look at it.

1. Preface background

Online code often takes up too much CPU. According to past experience, I will use top instructions to further check specific information with the help of jstack to troubleshoot problems, but basically they can not escape the situation that they need to be reissued, even if it is an incremental package, and the application also needs to be temporarily suspended. Later, the operation and maintenance brother asked me to try Arthas, saying that the code can be hot updated, just to give it a try.

Environment

JDK1.8

SPringBoot 2.2.2

Arthas

Linux

Test the code:

@ RequestMapping (value = "/ bigThread") @ ResponseBodypublic String bigThread (int id) {ArthasService.test (); while (true) {Thread T2 = new Thread (); t2.start (); id + +; if (100000 = = id) {return String.valueOf (id);}

2.thread-b to see if there are any blocked threads

Thread-b, find out which thread is currently blocking other threads, but do not find it after execution, indicating that the thread is not always blocked and executing all the time.

4.thread id to view specific information

Based on the previous step, let's take a closer look at thread 15 (because of the ID=15 above).

It roughly means that the thread is waiting for a condition to continue execution, and you can see that when the method is executing the LinkedBlockingQueue.take method, the API prompt for this method is as follows:

Public E take () throws InterruptedException {Ex; int c =-1 count.get final AtomicInteger count = this.count;final ReentrantLock takeLock = this.takeLock; takeLock.lockInterruptibly (); try {while (count.get () = = 0) {notEmpty.await ();} x = dequeue (); c = count.getAndDecrement (); if (c > 1) notEmpty.signal ();} finally {takeLock.unlock () } if (c = = capacity) signalNotFull (); return x;}

Among them: AtomicInteger is to ensure the atomicity in the case of high concurrency, and the ReentrantLock logo can be reentered into the lock, which is what you need to know under the JUC package.

The key point of this code is: notEmpty.await (), consuming data from the queue, when the queue is empty, thread blocking, so we roughly know that the problem is thread blocking, but we still don't know which line of code.

If you can clearly know which code has been changed this time, you can directly perform step 6, and if you do not know, you can use step 5 to locate the problem.

5.watch to see which Controller executed the code watch org.springframework.web.servlet.DispatcherServlet getHandler returnObj

This script can detect all methods that match Handler through DispatcherServlet, that is, requests to enter Controller, as follows:

After finding the corresponding code, let's take a closer look at the exception message, and there may be a question: why do I need to operate so tediously when I can view the error message through the log? My business scenario is: the log is still very large, and it was brushed over as soon as it was caught. At this time, the location log is not very easy to operate. Of course, it is also possible and intuitive to fish out the log. I usually go to check the log to locate the problem. Here is also to provide an idea.

6.watch the method exception information watch class full-path method name "{params [0], throwExp}"-e-x 2

As above, the error is intuitively prompted, and the following can be fixed and solved. Here, we can also check the execution time through the trace instruction:

Trace class full path method name "{params [0], throwExp}"-e-x 2

The return information is as follows, and you can also see the error message and how long each method has been executed.

[arthas@10999] $trace com.arthas.controller.OrderController bigThreadPress Q or Ctrl+C to abort.Affect (class count: 1, method count: 1) cost in 53 ms, listenerId: 10`-ts=2020-08-19 14 14-ts=2020-08-19 1414-1414 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `- [1452.684406ms] com.arthas.controller.OrderController:bigThread () [throws Exception] +-[0.168814ms] com.arthas.service.ArthasService:test () # 20`-throw:java.lang.OutOfMemoryError #-2 [unable to create new native thread] 7.jad anti-compilation hot update

After we know the problem above, we will locate the problem.

Command: jad class full path method name

[arthas@13190] $jad com.arthas.controller.OrderControllerClassLoader: +-org.springframework.boot.loader.LaunchedURLClassLoader@17f052a3 +-sun.misc.Launcher$AppClassLoader@3d4eac69 +-sun.misc.Launcher$ExtClassLoader@45f45fa1 Location: FileVOTRAGOTRAGUP softwareUniverse arthasUnix.jarSplink BOOTRANINFAccord classesless / / * Decompiled with CFR. * * Could not load the following classes: * com.arthas.service.ArthasService * org.springframework.stereotype.Controller * org.springframework.web.bind.annotation.RequestMapping * org.springframework.web.bind.annotation.ResponseBody * / package com.arthas.controller;import com.arthas.service.ArthasService;import org.springframework.stereotype.Controller;import org.springframework.web.bind.annotation.RequestMapping;import org.springframework.web.bind.annotation.ResponseBody @ Controllerpublic class OrderController {@ RequestMapping (value= {"/ bigThread"}) @ ResponseBodypublic String bigThread (int id) {ArthasService.test (); do {Thread T2 = new Thread (); t2.start ();} while (100000! = + + id); return String.valueOf (id);} Affect (row-cnt:1) cost in 1513 ms.

At this point, the code is decompiled, so we need to output it to a java file in order to be able to change it.

Instruction: jad com.arthas.controller.OrderController > / tmp/OrderController.java

That is: jad class full path method name > storage path / storage name

Then go to vi under the tmp path to modify the java file. After the modification is completed, check the corresponding classloader to prepare for compilation.

Sc-d * OrderController | grep classLoaderHashmc-c 17f052a3 / tmp/OrderController.java-d / tmp

But there is an error in the compilation here, official hint:

Before compilation, call [arthas@13190] $trace com.arthas.controller.OrderController bigThreadPress Q or Ctrl+C to abort.Affect (class count: 1, method count: 1) cost in 77 ms, listenerId: 2`-ts=2020-08-19 1515-ts=2020-ts=2020 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `- [6734.666529ms] com.arthas.controller.OrderController:bigThread () [throws Exception] +-[0.786517ms] com.arthas.service.ArthasService:test () # 20`-throw:java.lang.OutOfMemoryError #-2 [unable to create new native thread] Code @ RequestMapping (value = "/ bigThread") @ ResponseBodypublic String bigThread (int id) {ArthasService.test () While (true) {Thread T2 = new Thread (); t2.start (); id + +; if (100000 = = id) {return String.valueOf (id);}} @ RequestMapping (value = "/ bigThread") @ ResponseBodypublic String bigThread (int id) {ArthasService.test (); Thread T2 = new Thread (); t2.start (); return "success" } compilation directive [arthas@13190] $redefine / tmp/OrderController.classredefine success, size: 1, classes:com.arthas.controller.OrderController is called three times after compilation. `- ts=2020-08-19 1515 size-08-19 1515 size-08-19 after compilation, call`-ts=2020-08-19 1515 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `--[5.609405ms] com.arthas.controller.OrderController:bigThread ()`-- [0.204675ms] com.arthas.service.ArthasService:test () # 20`-ts=2020-08-19 15Groupe 52purse 04tress threadbare namehttp Mustang Mustang 0.0.0.0-8080 Mustang execlashi 4sideshift Idlides 10 isobasium daemonships truepriority5 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `--[3.900149ms] com.arthas.controller.OrderController:bigThread ()`-- [0.14636ms] com.arthas.service.ArthasService:test () # 20`-ts=2020-08-19 1515 Groupe 52Rich 04th threadbare namehttp Mustco Mustang 0.0.0.0-8080 Mustang execlashi 5sideshift isdaemontrueuspriority5 TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `- [1.90945ms] com.arthas.controller.OrderController:bigThread ()`-[0.147353ms] com.arthas.service.ArthasService:test () # 20

You can find that the time changed from 6734.666529ms to 3ms or so, indicating that the hot updated code is in effect.

This is how Arthas implements CPU troubleshooting and hot code updates. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.