Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "Java Web's Tomcat monitoring selection example analysis". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Monitoring and selection

Tomcat Manager and Psi-probe are good visual monitoring tools that can view Tomcat status information (such as requests per unit time, thread status, etc.). In the production environment, with the continuous increase of Tomcat instances, it is a bit tedious to maintain Manager consoles on different instances. at the same time, in order to combine with enterprise monitoring systems and operation and maintenance dashboards, it is necessary to choose monitoring tools with better compatibility and adaptability.

Nowadays, the popularity of enterprise micro-services and the strong demand of CI/CD require that our services have the characteristics of strong scalability and fast delivery while being easy to develop and maintain. In this environment, Tomcat+Prometheus+Grafana can well adapt to the existing architecture of the enterprise, and can be well integrated with other middleware service monitoring. As a new generation of cloud native monitoring system, Prometheus encourages users to monitor the internal status of the service, so that users can get the real running status of the service and application.

The schematic diagram of Tomcat+Prometheus+Grafana in container environment is as follows:

In the actual deployment, it can be divided into the following steps:

1. Download jmx_prometheus_javaagent (https://github.com/prometheus/jmx_exporter)). For container environment, you can enter jmx_prometheus_javaagent and Tomcat into the running image together.

2. When Tomcat starts, JAVA_OPTS adds Javaagent and Tomcat config.yaml (https://github.com/prometheus/jmx_exporter/blob/master/example_configs/Tomcat.yml) configurations, such as JAVA_OPTS= "- javaagent:/var/tomcat/jmx_prometheus_javaagent-0.3.1.jar=port:/vat/tomcat/conf/config.xml" (port is the actual port).

3. Prometheus add job_name of Tomcat.

Monitoring index carding

In the book "SRE: Google operation and maintenance decryption", it is pointed out that the monitoring system needs to be able to effectively support white-box monitoring and black-box monitoring. Through the white box monitoring, we can understand its internal actual operation status, and through the observation of the monitoring indicators, we can predict the possible problems, so as to optimize the potential uncertain factors. Black-box monitoring, such as HTTP probe, TCP probe and so on, can quickly notify the relevant personnel to deal with when the system or service fails.

The four golden metrics monitored can help us better measure end-user experience, service disruption, business impact, and other issues at the service level. Focus on the following four types of metrics: error, delay, traffic, and saturation.

Through the understanding of black box and white box monitoring, combined with the four gold indicators of monitoring, aiming at the important monitoring indicators of Tomcat, we can consider the following aspects:

Monitoring practice

Error

Error monitoring focuses on error requests that occur in the current application or system. For Tomcat error monitoring, the main monitoring indicator is the number of Tomcat error requests and the ratio to the total number of requests, most of which are explicit errors. For implicit errors (such as status code response 200, but the actual business process fails), you usually need to add hooks and business logic monitoring to the service according to the business scenario.

Figure 1: the ratio of the number of requests to errors of a Tomcat instance under 1000+qps

Of course, we can also use tools such as ELK to count the overall error status codes of Tomcat upstream agents or LB, so that we can better locate errors and consider how to "fail quickly" when 5xx errors occur, so as to reduce the impact of errors on system performance.

Figure 2: status code statistics collected by the Tomcat upstream agent

Delay

Delay monitoring focuses on the time required for service requests. For Tomcat delay monitoring, the main monitoring metric is the Response time of Tomcat instances. At the same time, we can also use HTTP semantic detection in black box monitoring to collect the full link delay of the domain name corresponding to Tomcat, and send specified GET, POST, HEAD and other requests to match the status code and the returned content of the server.

Figure 3: response time summary of multiple Tomcat instances under a product line

Figure 4: average request time and Tomcat average response time collected by the Tomcat upstream agent

Flow

Traffic monitoring focuses on the traffic of the current system or application in order to better measure the carrying capacity of the service. For Tomcat traffic monitoring, we need to pay attention to the traffic sent and received by Tomcat instances.

Figure 5: traffic sent and received by a Tomcat instance

Saturation degree

Saturation monitoring focuses on the utilization of resources by the system or application, and emphasizes the limited resources that can most affect the status of the service.

Different systems or applications have different requirements for saturation monitoring, but the commonness is that it plays a decisive role in the state index when the service performance decreases obviously. For Tomcat saturation monitoring, we can quickly locate the cause of the problem by looking at the change curve of each index before and after the alarm from the aspects of heap memory, number of threads, GC times and time consuming.

Tomcat thread pool

The collection of the initial configuration of Tomcat thread pool can be adjusted according to the pressure of different business scenarios and different instances, such as maxThreads, acceptCount and so on.

Figure 6: default configuration related to thread pool for a Tomcat instance

JVM monitoring

For Tomcat JVM monitoring, we usually need to pay attention to the usage of heap memory, the number and time of GC, and the monitoring of the number of JVM threads.

JVM heap memory monitoring

Java Heap is a piece of memory managed by JVM, which is shared by all threads and used to store object instances. Therefore, the use of heap memory is critical.

Figure 7: usage of the heap memory of a Tomcat instance

Number of GC and time-consuming

By looking at the number of GC and time consumed per unit time, we can select different garbage collectors for different business scenarios, thus further improving performance.

Figure 8: a Tomcat instance default garbage collector total GC degree curve

Figure 9: default garbage collector total GC time curve of a Tomcat instance

JVM thread count monitoring

The curve statistics of the number of JVM threads can give us a more intuitive view of the current operation of JVM threads.

Figure 10: thread-related statistics of a Tomcat instance JVM

This is the end of the content of "example Analysis of Tomcat Monitoring selection of Java Web". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.