Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to customize Prometheus monitoring indicators". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

At present, most of the companies that use Spring Boot to build micro-service system are using Prometheus to build micro-service measurement index (Metrics) monitoring system. The general approach is to integrate Prometheus index collection SDK in micro-service applications, so that Spring Boot exposes the relevant Metrics collection endpoints.

However, generally speaking, the number and types of Metrics exposed by Spring Boot by default are limited. If you want to establish richer monitoring dimensions (such as TP90/TP99 quantile metrics, etc.) for micro-service applications, you also need to configure other metric types provided by the Prometheus class library (micrometer-registry-prometheus) based on the Metrics that has been opened by Spring Boot by default.

But how can it be implemented in a more elegant way in the Spring Boot framework? Do you need to write the exposure logic of various custom monitoring indicator codes in the business code? In the following content, we will demonstrate how to customize Prometheus monitoring metrics in a more elegant way through @ annotations + AOP!

Notes on custom monitoring indicator configuration

It should be noted that in Spring Boot applications, the collection of program running information (such as metrics, logs) is more commonly achieved through Spring's AOP agent interception, but the logic of the interceptor running process will somehow damage the system performance, so in the process of customizing Prometheus monitoring metrics, the developers can choose whether to report metrics or not, and from the point of view of ease of use. This can be done through annotations. For example:

Package com.wudimanong.monitor.metrics.annotation; import java.lang.annotation.ElementType; import java.lang.annotation.Inherited; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; @ Target ({ElementType.METHOD}) @ Retention (RetentionPolicy.RUNTIME) @ Inherited public @ interface Tp {String description () default ";}

As shown in the code above, we have defined an annotation to mark the type of reporting timer metrics. If you want to count quantile metrics such as TP90 and TP99 for the API, you can use this annotation. In addition, you can define comments for other types of metrics to be reported, such as:

Package com.wudimanong.monitor.metrics.annotation; import java.lang.annotation.ElementType; import java.lang.annotation.Inherited; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; @ Target ({ElementType.METHOD}) @ Retention (RetentionPolicy.RUNTIME) @ Inherited public @ interface Count {String description () default ";}

As shown above, we have defined an annotation for reporting counter type metrics! If you want to count indicators such as the average response time of the interface and the number of requests for the interface, you can mark it through this annotation!

If you find it troublesome to define different types of metrics respectively, and you want to report to Prometheus for some APIs, you can also define a general annotation that can be used to report multiple metric types at the same time, for example:

Package com.wudimanong.monitor.metrics.annotation; import java.lang.annotation.ElementType; import java.lang.annotation.Inherited; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; @ Target ({ElementType.METHOD}) @ Retention (RetentionPolicy.RUNTIME) @ Inherited public @ interface Monitor {String description () default ";}

In short, whether it is to define specific indicator annotations separately or to define a general metric annotation, the goal is to expand the monitoring indicator types of Spring Boot micro-service applications in a more flexible way.

AOP Agent Logic implementation of Custom Monitoring Index Annotation

Above, we flexibly define annotations for reporting different metric types, and the specific implementation logic of the above annotation can be implemented by defining a general AOP proxy class. The specific implementation code is as follows:

Package com.wudimanong.monitor.metrics.aop; import com.wudimanong.monitor.metrics.Metrics; import com.wudimanong.monitor.metrics.annotation.Count; import com.wudimanong.monitor.metrics.annotation.Monitor; import com.wudimanong.monitor.metrics.annotation.Tp; import io.micrometer.core.instrument.Counter; import io.micrometer.core.instrument.MeterRegistry; import io.micrometer.core.instrument.Tag; import io.micrometer.core.instrument.Tags; import io.micrometer.core.instrument.Timer; import java.lang.reflect.Method Import java.util.function.Function; import org.aspectj.lang.ProceedingJoinPoint; import org.aspectj.lang.annotation.Around; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.reflect.MethodSignature; import org.springframework.stereotype.Component; @ Aspect @ Component public class MetricsAspect {/ * Prometheus Metrics Management * / private MeterRegistry registry; private Function tagsBasedOnJoinPoint Public MetricsAspect (MeterRegistry registry) {this.init (registry, pjp-> Tags. Of) (new String [] {"class", pjp.getStaticPart (). GetSignature (). GetDeclaringTypeName (), "method", pjp.getStaticPart (). GetSignature (). GetName ()});} public void init (MeterRegistry registry, Function tagsBasedOnJoinPoint) {this.registry = registry; this.tagsBasedOnJoinPoint = tagsBasedOnJoinPoint } / * * logical implementation of @ Tp indicator configuration annotations * / @ Around ("@ annotation (com.wudimanong.monitor.metrics.annotation.Tp)") public Object timedMethod (ProceedingJoinPoint pjp) throws Throwable {Method method = ((MethodSignature) pjp.getSignature ()) .getMethod (); method = pjp.getTarget () .getClass () .getMethod (method.getName (), method.getParameterTypes ()) Tp tp = method.getAnnotation (Tp.class); Timer.Sample sample = Timer.start (this.registry); String exceptionClass = "none"; try {return pjp.proceed ();} catch (Exception ex) {exceptionClass = ex.getClass () .getSimpleName (); throw ex } finally {try {String finalExceptionClass = exceptionClass / / create a definition counter And set the Tags information of the indicator (name can be customized) Timer timer = Metrics.newTimer ("tp.method.timed", builder-> builder.tags (new String [] {"exception", finalExceptionClass}) .tags (this.tagsBasedOnJoinPoint.apply (pjp)) .tag ("description") Tp.description () .publishPercentileHistogram () .register (this.registry)) Sample.stop (timer) } catch (Exception exception) {} / * logical implementation of @ Count indicator configuration annotations * / @ Around ("@ annotation (com.wudimanong.monitor.metrics.annotation.Count)") public Object countMethod (ProceedingJoinPoint pjp) throws Throwable {Method method = ((MethodSignature) pjp.getSignature ()) .getMethod () Method = pjp.getTarget (). GetClass (). GetMethod (method.getName (), method.getParameterTypes ()); Count count = method.getAnnotation (Count.class); String exceptionClass = "none"; try {return pjp.proceed ();} catch (Exception ex) {exceptionClass = ex.getClass (). GetSimpleName (); throw ex } finally {try {String finalExceptionClass = exceptionClass / / create a definition counter And set the Tags information of the indicator (name can be customized) Counter counter = Metrics.newCounter ("count.method.counted", builder-> builder.tags (new String [] {"exception", finalExceptionClass}) .tags (this.tagsBasedOnJoinPoint.apply (pjp)) .tag ("description") Count.description () .register (this.registry)) Counter.increment () } catch (Exception exception) {} / * logical implementation of @ Monitor generic indicator configuration annotations * / @ Around ("@ annotation (com.wudimanong.monitor.metrics.annotation.Monitor)") public Object monitorMethod (ProceedingJoinPoint pjp) throws Throwable {Method method = ((MethodSignature) pjp.getSignature ()) .getMethod () Method = pjp.getTarget (). GetClass (). GetMethod (method.getName (), method.getParameterTypes ()); Monitor monitor = method.getAnnotation (Monitor.class); String exceptionClass = "none"; try {return pjp.proceed ();} catch (Exception ex) {exceptionClass = ex.getClass (). GetSimpleName (); throw ex } finally {try {String finalExceptionClass = exceptionClass / / timer Metric Timer timer = Metrics.newTimer ("tp.method.timed", builder-> builder.tags (new String [] {"exception", finalExceptionClass}) .tags (this.tagsBasedOnJoinPoint.apply (pjp)) .tag ("description") Monitor.description () .publishPercentileHistogram () .register (this.registry)) Timer.Sample sample = Timer.start (this.registry); sample.stop (timer) / / counter Metric Counter counter = Metrics.newCounter ("count.method.counted", builder-> builder.tags (new String [] {"exception", finalExceptionClass}) .tags (this.tagsBasedOnJoinPoint.apply (pjp)) .tag ("description") Monitor.description () .register (this.registry)) Counter.increment ();} catch (Exception exception) {}

The above code fully implements the logic of the metric configuration annotations we defined earlier, where the logic for @ Monitor annotations is the integration of @ Tp and @ Count annotation logic. If you need to define other metric types, you can continue to expand on this basis!

It should be noted that in the above logic implementation, the construction of indicator types such as "Timer" and "Counter" is not directly used by the construction object in the "micrometer-registry-prometheus" dependency package, but is implemented through a custom Metrics.newTimer (), which is mainly intended to report metrics in a more concise and flexible way. The code definition is as follows:

Package com.wudimanong.monitor.metrics; import io.micrometer.core.instrument.Counter; import io.micrometer.core.instrument.Counter.Builder; import io.micrometer.core.instrument.DistributionSummary; import io.micrometer.core.instrument.Gauge; import io.micrometer.core.instrument.MeterRegistry; import io.micrometer.core.instrument.Timer; import io.micrometer.core.lang.NonNull; import java.util.function.Consumer; import java.util.function.Supplier; import org.springframework.beans.BeansException; import org.springframework.context.ApplicationContext Import org.springframework.context.ApplicationContextAware; public class Metrics implements ApplicationContextAware {private static ApplicationContext context; @ Override public void setApplicationContext (@ NonNull ApplicationContext applicationContext) throws BeansException {context = applicationContext;} public static ApplicationContext getContext () {return context;} public static Counter newCounter (String name, Consumer consumer) {MeterRegistry meterRegistry = context.getBean (MeterRegistry.class); return new CounterBuilder (meterRegistry, name, consumer). Build () } public static Timer newTimer (String name, Consumer consumer) {return new TimerBuilder (context.getBean (MeterRegistry.class), name, consumer). Build ();}}

The above code obtains the MeterRegistry instance by accessing the context of the Spring container, and uses it to build metric type objects such as Counter and Timer. The reason why the acquisition method is defined as static here is that it is easy to reference in the business code!

The CounterBuilder and TimerBuilder constructor code definitions involved in the above code are as follows:

Package com.wudimanong.monitor.metrics; import io.micrometer.core.instrument.Counter; import io.micrometer.core.instrument.Counter.Builder; import io.micrometer.core.instrument.MeterRegistry; import java.util.function.Consumer; public class CounterBuilder {private final MeterRegistry meterRegistry; private Counter.Builder builder; private Consumer consumer; public CounterBuilder (MeterRegistry meterRegistry, String name, Consumer consumer) {this.builder = Counter.builder (name); this.meterRegistry = meterRegistry This.consumer = consumer;} public Counter build () {consumer.accept (builder); return builder.register (meterRegistry);}}

The above code is the CounterBuilder constructor code! the TimerBuilder constructor code is as follows:

Package com.wudimanong.monitor.metrics; import io.micrometer.core.instrument.MeterRegistry; import io.micrometer.core.instrument.Timer; import io.micrometer.core.instrument.Timer.Builder; import java.util.function.Consumer; public class TimerBuilder {private final MeterRegistry meterRegistry; private Timer.Builder builder; private Consumer consumer; public TimerBuilder (MeterRegistry meterRegistry, String name, Consumer consumer) {this.builder = Timer.builder (name); this.meterRegistry = meterRegistry This.consumer = consumer;} public Timer build () {this.consumer.accept (builder); return builder.register (meterRegistry);}}

The reason why the constructor code is specifically defined separately is mainly from the elegance of the code! If the construction of other metric types is involved, it can be extended in a similar way!

Custom metric annotation configuration class

In the above code, we have defined several custom metric annotations and their implementation logic code. In order to make them run in the Spring Boot environment, we also need to write the following configuration class, as follows:

Package com.wudimanong.monitor.metrics.config; import com.wudimanong.monitor.metrics.Metrics; import io.micrometer.core.instrument.MeterRegistry; import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer; import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.env.Environment @ Configuration public class CustomMetricsAutoConfiguration {@ Bean @ ConditionalOnMissingBean public MeterRegistryCustomizer meterRegistryCustomizer (Environment environment) {return registry-> {registry.config () .commonTags ("application", environment.getProperty ("spring.application.name"));}; @ Bean @ ConditionalOnMissingBean public Metrics metrics () {return new Metrics ();}}

The above configuration code mainly specifies the application name carried in the reporting Prometheus metric information, and configures the custom Metrics class with Bean!

The use and effect of business code

Next, let's demonstrate how to report Prometheus monitoring metrics in the business code as follows:

Package com.wudimanong.monitor.controller; import com.wudimanong.monitor.metrics.annotation.Count; import com.wudimanong.monitor.metrics.annotation.Monitor; import com.wudimanong.monitor.metrics.annotation.Tp; import com.wudimanong.monitor.service.MonitorService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController @ RestController @ RequestMapping ("/ monitor") public class MonitorController {@ Autowired private MonitorService monitorServiceImpl; / / Monitoring Metrics Notes use / / @ Tp (description = "/ monitor/test") / / @ Count (description = "/ monitor/test") @ Monitor (description = "/ monitor/test") @ GetMapping ("/ test") public String monitorTest (@ RequestParam ("name") String name) {monitorServiceImpl.monitorTest (name) Return "Monitoring demonstration project test interface returned-> OK!";}}

As shown in the above code, in the actual business programming, it is relatively simple to configure the Prometheus monitoring metrics uploaded by the interface through annotations! When you start the program locally, you can view the relevant metrics by visiting the "/ actuator/prometheus" metric collection endpoint of the microservice application, as shown in the following figure:

With these custom reported monitoring metrics, after Promethues is collected, we can use visualization tools such as Grafana to build a friendly monitoring view with a multi-dimensional interface, such as TP90/TP99:

As shown above, multiple PromeQL can be defined at the same time in Grafana to define different monitoring metrics information. Here, we count the TP90 and TP95 quantiles of the interface method "monitorTest" through the "histogram_quantile" function provided by Prometheus. And the indicator used is the custom "tp_method_timed_xx" indicator type!

"how to customize Prometheus monitoring indicators" content is introduced here, thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.