Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to use Uber jvm profiler. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Background

Uber jvm profiler is used to collect jvm-related indicators in distributed monitoring, such as cpu/memory/io/gc information, etc.

Installation

Make sure that maven and JDK > = 8 are installed, and direct mvn clean package

Java application

Description

Can be used directly in the deployment of java agent

Use

Java-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.KafkaOutputReporter,brokerList='kafka1:9092',topicPrefix=demo_,tag=tag-demo,metricInterval=5000,sampleInterval=0-cp target/jvm-profiler-1.0.0.jar

Option interpretation

Parameter description reporterreporter category, where the default is com.uber.profiling.reporters.KafkaOutputReporter. If reporter is com.uber.profiling.reporters.KafkaOutputReporter, then brokerList is the kafka list, and topicPrefix is separated by commas. If reporter is com.uber.profiling.reporters.KafkaOutputReporter, the prefix tagkey of topicPrefix is kafka topic is metric of tag, and the frequency of metricIntervalmetric report in reporter will be output to the frequency of mssampleIntervaljvm stack metrics report according to the actual situation, and ms according to the actual situation.

Result display

"nonHeapMemoryTotalUsed": 11890584.0, "bufferPools": [{"totalCapacity": 0, "name": "direct", "count": 0, "memoryUsed": 0}, {"totalCapacity": 0, "name": "mapped", "count": 0, "memoryUsed": 0}] "heapMemoryTotalUsed": 24330736.0, "epochMillis": 1515627003374, "nonHeapMemoryCommitted": 13565952.0, "heapMemoryCommitted": 257425408.0, "memoryPools": [{"peakUsageMax": 251658240, "usageMax": 251658240, "peakUsageUsed": 1194496, "name": "Code Cache", "peakUsageCommitted": 2555904, "usageUsed": 1173504, "type": "Non-heap memory" "usageCommitted": 2555904}, {"peakUsageMax":-1, "usageMax":-1, "peakUsageUsed": 9622920, "name": "Metaspace", "peakUsageCommitted": 9830400, "usageUsed": 9622920, "type": "Non-heap memory", "usageCommitted": 9830400} {"peakUsageMax": 1073741824, "usageMax": 1073741824, "peakUsageUsed": 1094160, "name": "Compressed Class Space", "peakUsageCommitted": 1179648, "usageUsed": 1094160, "type": "Non-heap memory", "usageCommitted": 1179648}, {"peakUsageMax": 1409286144 "usageMax": 1409286144, "peakUsageUsed": 24330736, "name": "PS Eden Space", "peakUsageCommitted": 67108864, "usageUsed": 24330736, "type": "Heap memory", "usageCommitted": 67108864}, {"peakUsageMax": 11010048, "usageMax": 11010048, "peakUsageUsed": 0 "name": "PS Survivor Space", "peakUsageCommitted": 11010048, "usageUsed": 0, "type": "Heap memory", "usageCommitted": 11010048}, {"peakUsageMax": 2863661056, "usageMax": 2863661056, "peakUsageUsed": 0, "name": "PS Old Gen", "peakUsageCommitted": 179306496 "usageUsed": 0, "type": "Heap memory", "usageCommitted": 179306496}], "processCpuLoad": 0.0008024004394748531, "systemCpuLoad": 0.23138430784607697, "processCpuTime": 496918000, "appId": null, "name": "24103@machine01", "host": "machine01", "processUuid": "3c2ec835-749d-45ea-a7ec-e4b9fe17c23a", "tag": "mytag" "gc": [{"collectionTime": 0, "name": "PS Scavenge", "collectionCount": 0}, {"collectionTime": 0, "name": "PS MarkSweep", "collectionCount": 0}]} spark application

Description

Unlike java applications, jvm-profiler.jar needs to be distributed to each node

Use

-jars hdfs:///public/libs/jvm-profiler-1.0.0.jar-conf spark.driver.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.KafkaOutputReporter,brokerList='kafka1:9092',topicPrefix=demo_,tag=tag-demo,metricInterval=5000,sampleInterval=0-conf spark.executor.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.KafkaOutputReporter,brokerList='kafka1:9092',topicPrefix=demo_,tag=tag-demo,metricInterval=5000,sampleInterval=0

Option interpretation

Parameter description reporterreporter category, where the default is com.uber.profiling.reporters.KafkaOutputReporter. If reporter is com.uber.profiling.reporters.KafkaOutputReporter, then brokerList is the kafka list, and topicPrefix is separated by commas. If reporter is com.uber.profiling.reporters.KafkaOutputReporter, the prefix tagkey of topicPrefix is kafka topic is metric of tag, and the frequency of metricIntervalmetric report in reporter will be output to the frequency of mssampleIntervaljvm stack metrics report according to the actual situation, and ms according to the actual situation.

Result display

"nonHeapMemoryTotalUsed": 11890584.0, "bufferPools": [{"totalCapacity": 0, "name": "direct", "count": 0, "memoryUsed": 0}, {"totalCapacity": 0, "name": "mapped", "count": 0, "memoryUsed": 0}] "heapMemoryTotalUsed": 24330736.0, "epochMillis": 1515627003374, "nonHeapMemoryCommitted": 13565952.0, "heapMemoryCommitted": 257425408.0, "memoryPools": [{"peakUsageMax": 251658240, "usageMax": 251658240, "peakUsageUsed": 1194496, "name": "Code Cache", "peakUsageCommitted": 2555904, "usageUsed": 1173504, "type": "Non-heap memory" "usageCommitted": 2555904}, {"peakUsageMax":-1, "usageMax":-1, "peakUsageUsed": 9622920, "name": "Metaspace", "peakUsageCommitted": 9830400, "usageUsed": 9622920, "type": "Non-heap memory", "usageCommitted": 9830400} {"peakUsageMax": 1073741824, "usageMax": 1073741824, "peakUsageUsed": 1094160, "name": "Compressed Class Space", "peakUsageCommitted": 1179648, "usageUsed": 1094160, "type": "Non-heap memory", "usageCommitted": 1179648}, {"peakUsageMax": 1409286144 "usageMax": 1409286144, "peakUsageUsed": 24330736, "name": "PS Eden Space", "peakUsageCommitted": 67108864, "usageUsed": 24330736, "type": "Heap memory", "usageCommitted": 67108864}, {"peakUsageMax": 11010048, "usageMax": 11010048, "peakUsageUsed": 0 "name": "PS Survivor Space", "peakUsageCommitted": 11010048, "usageUsed": 0, "type": "Heap memory", "usageCommitted": 11010048}, {"peakUsageMax": 2863661056, "usageMax": 2863661056, "peakUsageUsed": 0, "name": "PS Old Gen", "peakUsageCommitted": 179306496 "usageUsed": 0, "type": "Heap memory", "usageCommitted": 179306496}], "processCpuLoad": 0.0008024004394748531, "systemCpuLoad": 0.23138430784607697, "processCpuTime": 496918000, "appId": null, "name": "24103@machine01", "host": "machine01", "processUuid": "3c2ec835-749d-45ea-a7ec-e4b9fe17c23a", "tag": "mytag" "gc": [{"collectionTime": 0, "name": "PS Scavenge", "collectionCount": 0}, {"collectionTime": 0, "name": "PS MarkSweep", "collectionCount": 0}]} analysis

Existing reporter

Reporter explains that ConsoleOutputReporter defaults to repoter, which is generally used to debug FileOutputReporter file-based reporter, but is not applicable in distributed environments. It is necessary to set outputDirKafkaOutputReporter based on kafka reporter, which is often used in formal environments. You have to set brokerList,topicPrefixGraphiteOutputReporter based on Graphite reporter, need to set graphite.host and other configuration RedisOutputReporter based on redis reporter, build commands mvn-P redis clean packageInfluxDBOutputReporter based on InfluxDB reporter, build commands mvn-P influxdb clean package, need to set influxdb.host and other configuration recommendations to use KafkaOutputReporter in the production environment. High flexibility in operation, can be combined with clickhouse grafana for indicator display

Source code analysis

The jvm-profiler as a whole is based on java agent implementation, and the project pom file specifies the Premain-Class entry and Agent-Class in MANIFEST.MF as the specific implementation class of com.uber.profiling.Agent, AgentImpl.

Analyze the run method of the specific AgentImpl class

Public void run (Arguments arguments, Instrumentation instrumentation, Collection objectsToCloseOnShutdown) {if (arguments.isNoop ()) {logger.info ("Agent noop is true, do not run anything"); return;} Reporter reporter = arguments.getReporter (); String processUuid = UUID.randomUUID (). ToString (); String appId = null; String appIdVariable = arguments.getAppIdVariable () If (appIdVariable! = null & &! appIdVariable.isEmpty ()) {appId = System.getenv (appIdVariable);} if (appId = = null | | appId.isEmpty ()) {appId = SparkUtils.probeAppId (arguments.getAppIdRegex ()) } if (! arguments.getDurationProfiling (). IsEmpty () | |! arguments.getArgumentProfiling () .isEmpty ()) {instrumentation.addTransformer (new JavaAgentFileTransformer (arguments.getDurationProfiling (), arguments.getArgumentProfiling ();} List profilers = createProfilers (reporter, arguments, processUuid, appId); ProfilerGroup profilerGroup = startProfilers (profilers); Thread shutdownHook = new Thread (new ShutdownHookRunner (profilerGroup.getPeriodicProfilers (), Arrays.asList (reporter), objectsToCloseOnShutdown)) Runtime.getRuntime () .addShutdownHook (shutdownHook);}

Arguments.getReporter () gets the reporter. If it is not set, it is set to reporterConstructor, otherwise it is set to the specified reporter

String appId, set appId, first look it up in the configuration, if there is no setting, then look it up in env. For spark applications, take the value of spark.app.id.

List profilers = createProfilers (reporter, arguments, processUuid, appId), create profilers. Default is CpuAndMemoryProfiler,ThreadInfoProfiler,ProcessInfoProfiler.

1. Where CpuAndMemoryProfiler,ThreadInfoProfiler,ProcessInfoProfiler reads data from JMX, ProcessInfoProfiler also reads data from / pro

two。 If durationProfiling,argumentProfiling,sampleInterval,ioProfiling is set, the corresponding MethodDurationProfiler (time spent by the output method call), MethodArgumentProfiler (the value of the output method parameter), StacktraceReporterProfiler,IOProfiler will be increased.

3.MethodArgumentProfiler and MethodDurationProfiler use the javassist third-party bytecode compilation tool to rewrite the corresponding class. The specific implementation refers to JavaAgentFileTransformer.

4.StacktraceReporterProfiler reads data from JMX

5.IOProfiler reads the directory data corresponding to the / pro file on the local machine.

ProfilerGroup profilerGroup = startProfilers (profilers) starts timing report of profiler

It will also distinguish between oneTimeProfilers and periodicProfilers,ProcessInfoProfiler, which belong to oneTimeProfilers, because the information of process will not change during operation and does not require reporter of periodic lines.

At this point, the whole process is over

This is the end of the article on "how to use Uber jvm profiler". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.