Shulou(Shulou.com)05/31 Report--

This article mainly analyzes the relevant knowledge points of how to analyze the principle of non-invasive micro-service probe, the content is detailed and easy to understand, the operation details are reasonable, and has a certain reference value. If you are interested, you might as well follow the editor and learn more about "how to analyze the principle of non-invasive microservice probe".

Preface

With the rise of micro-service architecture, the complexity of application behavior increases significantly. In order to improve the observability of services, distributed monitoring system becomes very important.

Based on Google's Dapper paper, many famous monitoring systems have been developed: Zipkin, Jaeger, Skywalking and OpenTelemetry, which wants to unify the rivers and lakes. A large number of manufacturers and open source enthusiasts have made a lot of excellent designs around the collection, collection, storage and display of monitoring data.

Today, even individual developers can rely on open source products to easily build a complete monitoring system. However, as a monitoring service provider, we must do a good job of unbinding with the business to reduce the cost of user access, version update, problem repair and business stop loss. Therefore, a pluggable, non-intrusive collector has become a necessary killer mace for many manufacturers.

In order to obtain call chain information between services, collectors usually need to bury points before and after the method. In Java ecology, there are two common burial methods: relying on SDK manual burial point, and using Javaagent technology to do non-invasive burial point. The following around the non-invasive burial technology and principle for everyone to do a comprehensive introduction.

Non-invasive collector (probe)

In the distributed monitoring system, the module can be divided into: collector (Instrument), transmitter (TransPort), collector (Collector), storage (Srotage), display (API&UI).

Architectural illustration of zipkin

The collector sends the collected monitoring information from the application side to the collector, stores it, and finally provides it to the front-end query.

The information collected by the collector is called Trace (call chain). A Trace has a unique identity traceId, consisting of a top-down tree-like span. In addition to spanId, each span also has a traceId and a parent spanId, so that a complete call chain relationship can be restored.

In order to generate a span, we need to put a burying point before and after the method call. For example, in a http call, we add a burying point before and after the execute () method to get the complete calling method information and generate a span unit.

In Java ecology, there are two common burial methods: relying on SDK manual burial point, and using Javaagent technology to do non-invasive burial point. Many developers come into contact with distributed monitoring systems, starting from Zipkin. The most classic thing is to understand the X-B3 trace protocol, use Brave SDK, and manually generate trace. But the way of burying the SDK is undoubtedly deeply dependent on the business logic. When upgrading the burying point, the code must be changed.

So how to unbind it from the business logic?

Java also provides another way: rely on Javaagent technology, modify the bytecode of the target method to achieve non-invasive burial point. This collector using Javaagent is also called a probe. Using-javaagent when the application is started, or using the attach (pid) mode at run time, you can import the probe package into the application and complete the implantation of the burial point. Non-intrusive way, can achieve senseless thermal upgrade. Users do not need to understand the underlying principles to use a complete monitoring service. At present, many open source monitoring products have provided a rich java probe library, as a monitoring service provider, further reduce the development cost.

To develop a non-invasive probe, it can be divided into three parts: Javaagent, bytecode enhancement tool, and trace generation logic. The following will introduce these contents for you.

Basic concept

Let's take a look at the knowledge about JavaAgent before using Java.

What is a bytecode?

Since the c-like language Java was invented by sun in 1994, it has become very popular all over the world, relying on the feature of "compile once, run everywhere". Unlike C++, Java first compiles all the source code into class (bytecode) files, and then relies on JVM (virtual machines) on different platforms to interpret and execute bytecode, thus unbinding it from the hardware. The structure of the class file is an table table, which is made up of a number of struct objects.

Types

Name

Description

Length

U4magic magic number Identify Class file format 4 bytes u2minor_version minor version number 2 bytes u2major_version major version number 2 bytes u2constant_pool_count constant pool calculator 2 bytes cp_infoconstant_pool constant pool n bytes u2access_flags access flag 2 bytes u2this_class class index 2 bytes u2super_class parent class index 2 bytes u2interfaces_count interface counter 2 bytes u2interfaces interface index collection 2 byte u2fields_count fields Number of 2-byte field_infofields field collection n-byte u2methods_count method counter 2-byte method_infomethods method set n-byte u2attributes_count additional attribute counter 2-byte attribute_infoattributes additional attribute set n-byte

Field properties of bytecode

Let's compile a simple class `Demo.java`

Package com.httpserver;public class Demo {private int num = 1; public int add () {num = num + 2; return num;}}

Open the Demo.class file in hexadecimal, and the parsed field is also composed of many struct fields: constant pool, parent class information, method information, and so on.

JDK's own parsing tool, javap, can print class files in a human-readable way, and the results are consistent with the above.

What is JVM?

JVM (Java Virtual Machine), a virtual machine capable of running Java bytecode, is part of the Java architecture. JVM has its own perfect hardware architecture, such as processors, stacks, registers, etc., as well as corresponding instruction systems. JVM shields the information related to the specific operating system platform, so that Java programs only need to generate object code (bytecode) running on JVM to run on a variety of platforms without modification, which is the real meaning of "compile once, run everywhere".

Virtual machine, as a programming language, is not only dedicated to Java language, but any language can be compiled and run by JVM as long as the generated compilation file meets the requirements of JVM for loading and compiling file format.

At the same time, the JVM specification does not define the garbage collection algorithm and the internal algorithm to optimize the instructions of the Java virtual machine, which only describes the functions that should have, mainly in order not to bring too much trouble and restrictions to the implementer. It is precisely because of the right description, this leaves room for various manufacturers to develop.

Wikipedia: a comparison of existing JVM

Among them, HotSpot (Orcale) and OpenJ9 (IBM) with better performance are loved by the majority of developers.

Memory Model of JVM

After JVM deployment, each Java application starts, it will call JVM's lib library to apply for resources to create a JVM instance. JVM divides the memory into different areas. The following is the memory model of the JVM runtime:

Method area: used to store class information, constants, static variables, code compiled by just-in-time compiler, etc.

Heap: all threads share and place object objects and arrays, which is also the GC (the main area of the garbage collector)

Virtual machine stack & program counter: threads are private and each new thread allocates a corresponding memory object. Each method is called until the completion of execution, corresponding to a stack frame in the virtual machine stack from the stack to the stack process.

Parent delegation loading mechanism

When a Java application starts and runs, an important action is to load the definition of the class and create an instance. This depends on JVM's own ClassLoader mechanism.

Parental appointment

A class must be loaded by a ClassLoader, and the corresponding ClassLoader has a parent ClassLoader. Looking for a class definition will be looked up from the bottom up, which is the parent delegation model.

To save memory, JVM does not put all class definitions in memory, but

At startup: load the necessary classes into memory through ClassLoader

Runtime: when creating a new instance, look for it in memory first, otherwise load it into memory

Execution method: find the definition of the method, put the local variables and the bytecode of the method into the virtual machine stack, and finally return the calculation result. Of course, static methods are different.

This design reminds us that magical enhancements can be achieved if you can replace the loaded class definition at load time or directly.

JVM tool Interface

The obscure JVM shields the underlying complexity and allows developers to focus on business logic. In addition to booting with memory parameters through java-jar, there is actually a set of special interfaces for developers, and that is JVM tool Interface.

JVM TI is a two-way interface. JVM TI Client, also known as agent, is based on the event event mechanism. It accepts events, exercises control over JVM, and responds to events.

It has an important feature-Callback (callback function) mechanism: JVM can generate a variety of events, in the face of various events, it provides a Callback array. The Callback function is called when each event is executed, so the core of writing JVM TI Client is to place the Callback function.

It is this mechanism that allows us to send instructions to JVM to load new class definitions.

JavaAgent

Now let's try to think about this: how do you change the definition of the method in the application?

It's a bit like putting an elephant in the fridge:

Generate a new class according to the bytecode specification

Using JVM TI, command JVM to load the class into the corresponding memory.

After replacement, the system will use the method we enhanced.

This is not easy, but fortunately, jdk has prepared such an upper interface instructment package for us. It is also very easy to use, let's use a simple example of agent to illustrate the key design of the instructment package.

Simple example of Javaagent

Javaagent can be used in two ways:

Add parameters to configure agent package path at startup:-javaagent:/$ {path} / agent.jar

Run attach to the pid of the JVM instance, attaching the jar package to: VirtualMachine.attach (pid); VirtualMachine.loadAgent ("/ / agent.jar")

Use demo in the first way

Public class PreMainTraceAgent {public static void premain (String agentArgs, Instrumentation inst) {inst.addTransformer (new DefineTransformer (), true);} static class DefineTransformer implements ClassFileTransformer {@ Override public byte [] transform (ClassLoader loader, String className, Class classBeingRedefined, ProtectionDomain protectionDomain, byte [] classfileBuffer) throws IllegalClassFormatException {System.out.println ("premain load Class:" + className); return classfileBuffer;}

Manifest-Version: 1.0

Can-Redefine-Classes: true

Can-Retransform-Classes: true

Premain-Class: PreMainTraceAgent

Then create a new directory under the resources directory: META-INF, and a new file under that directory: MANIFREST.MF:

Finally, it is packed into an agent.jar package.

Premain (): the entrance to the entry mode of javaagent. As the name implies, it is executed before the main function. When making a jar package, you need to name the entry Premain-Class: PreMainTraceAgent in the MF file.

Handle to the Instrumentation:JVM instance. Whether you go to-javaagent or attach, you will eventually get an instance-related Instrumentation. The two more important functions in inst are redefineClasses (ClassDefinition...) Definitions) and retransformClasses (Class...) Classes) through these two functions, we can add the enhanced bytecode to the JVM

What's the difference between redefineClasses () and retransformClasses ()? redefineClasses () is suitable for modifying newly added classes, while retransformClasses () can replace which class definitions have been loaded into memory.

ClassFileTransformer: there is an important method transform () in this interface, and the consumer needs to implement this class. When this class is added to the Transformer array in inst, this method is called when each class is loaded or modified. Information about the definition of the class, such as the class binary definition classfileBuffer

AddTransformer (): you can add classes that implement ClassFileTransformer to an array built into Instrumentation. Just like a processing factory, the class processed by the previous ClassFileTransformer will be used as an argument to the next ClassFileTransformer.

When you get here, you'll find that enhanced bytecode is so simple.

Bytecode generation tool

Through the previous understanding, there is no more than such a feeling of modifying bytecode ^ _ ^! But we have to pay attention to another problem, how to generate bytes?

Boss: I am familiar with the JVM specification, understand the meaning of each bytecode, I can manually change the class file, for which I wrote a library.

Master: I know the customer's framework. I modify the source code, recompile, and replace the binary.

Xiaobai: I can't understand the bytecode. I can use the library written by the boss.

Here are several common bytecode generation tools

ASM

ASM is a pure bytecode generation and analysis framework. It has complete syntax analysis, semantic analysis, and can be used to generate class bytecode dynamically. But this tool is still too professional, the user must be very familiar with the JVM specification, must be clear about the replacement of a function to make exactly what changes in the class file. ASM provides two sets of API:

CoreAPI event-based formal representation class

TreeAPI represents classes in an object-based way

With a preliminary knowledge of bytecode and JVM memory model, you can simply generate classes according to the official documentation.

ASM is very powerful and is used in 1. OpenJDK's lambda syntax 2. Groovy and Koltin's compiler 3. Test coverage statistics tools Cobertura and Jacoco 4. Single test mock tools, such as Mockito and EasyMock 5. CGLIB, ByteBuddy these dynamic class generation tools. BYTEBUDDY

ByteBuddy is an excellent runtime bytecode generation tool based on ASM implementation and provides easier-to-use API. It is used by many distributed monitoring projects such as Skywalking, Datadog and so on as probes of Java applications to collect monitoring information.

The following is a performance comparison with other tools.

Java Proxy:JDK has its own agent mechanism, which can host users' classes to facilitate expansion. But an interface must be given, and the function is limited.

Cglib: famous, but developed too early and not updated with the features of JDK. Although its library is still useful, it has been slowly removed from the project by users.

Javassit: this library attempts to mimic the javac compiler by converting the source code at run time. This is very ambitious, but this difficulty is very challenging, so far there is still a considerable gap with javac.

In our actual use, ByteBuddy's API is really friendly and basically meets all bytecode enhancement requirements: modifications to interfaces, classes, methods, static methods, constructor methods, annotations, and so on. In addition, the built-in Matcher interface supports fuzzy matching, and you can modify qualified types according to name matching.

But there are also some disadvantages: the official documents are relatively old and there are few Chinese documents. Many important features, such as aspects, are not described in detail, and you often need to look at code comments and test cases to understand the true meaning. If you are interested in ByteBuddy, you can follow our official account. The following article will share ByteBuddy specifically.

Generation of Trace data

Through bytecode enhancement, we can achieve a non-invasive burial point, so the connection with the generation logic of trace is injected into the soul. Let's use a simple example to show how such a combination can be done.

Tracer API

This is a simple API that is used to generate trace messages.

Public class Tracer {public static Tracer newTracer () {return newTracer ();} public Span newSpan () {return newSpan ();} public static class Span {public void start () {System.out.println ("start a span");} public void end () {System.out.println ("span finish") / / todo: save span in db}

There is only one method sayHello (String name) target class Greeting

Public class Greeting {public static void sayHello (String name) {System.out.println ("Hi!" + name);}}

To generate trace messages manually, we need to add manual burial points before and after the method.

... Public static void main (String [] args) {Tracer tracer = Tracer.newTracer (); / / generate a new span Tracer.Span span = tracer.newSpan (); / / start and end of span span.start (); Greeting.sayHello ("developer"); span.end ();}.

Non-invasive burial point

Byte enhancement allows us to avoid modifying the source code. Now we can define a simple section, put the span generation logic into the section, and then use Bytebuddy to implant the burial point.

TraceAdvice

Put the trace generation logic into the section

Public class TraceAdvice {public static Tracer.Span span = null; public static void getCurrentSpan () {if (span = = null) {span = Tracer.newTracer () .newSpan () }} / * @ param target target class instance * @ param clazz target class class * @ param method target method * @ param args target method parameter * / @ Advice.OnMethodEnter public static void onMethodEnter (@ Advice.This (optional = true) Object target, @ Advice.Origin Class clazz @ Advice.Origin Method method, @ Advice.AllArguments Object [] args) {getCurrentSpan () Span.start () } / * @ param target target class instance * @ param clazz target class class * @ param method target method * @ param args target method parameter * @ param result returns result * / @ Advice.OnMethodExit (onThrowable = Throwable.class) public static void onMethodExit (@ Advice.This (optional = true) Object target @ Advice.Origin Class clazz, @ Advice.Origin Method method, @ Advice.AllArguments Object [] args, @ Advice.Return (typing = Assigner.Typing.DYNAMIC) Object result) {span.end () Span = null;}}

OnMethodEnter: called when the method enters. Bytebuddy provides a series of annotations with @ Advice.OnMethodExit static methods that can be populated with the node where the method starts. We can get the details of the method and even modify the incoming parameters to skip the execution of the target method.

OnMethodExit: called when the method ends. Similar to onMethodEnter, but can catch the exception thrown by the method body and modify the return value.

Implant Advice

Pass the Instrumentation handle obtained by Javaagent to AgentBuilder (API of Bytebuddy)

Public class PreMainTraceAgent {public static void premain (String agentArgs, Instrumentation inst) {/ / Bytebuddy's API is used to modify AgentBuilder agentBuilder = new AgentBuilder.Default () .with (AgentBuilder.PoolStrategy.Default.EXTENDED) .with (AgentBuilder.InitializationStrategy.NoOp.INSTANCE) .with (AgentBuilder.RedefinitionStrategy.RETRANSFORMATION) .with (new WeaveListener ()) .ClassFormatChanges () AgentBuilder = agentBuilder / / matches the full class name of the target class .type (ElementMatchers.named ("baidu.bms.debug.Greeting")) .transform (new AgentBuilder.Transformer () {@ Override public DynamicType.Builder transform (DynamicType.Builder builder) TypeDescription typeDescription, ClassLoader classLoader JavaModule module) {return builder.visit (/ / woven into section Advice.to (TraceAdvice.class) / / match the method of the target class .on (ElementMatchers.named ("sayHello") }}); agentBuilder.installOn (inst);} / / Local launch public static void main (String [] args) throws Exception {ByteBuddyAgent.install (); Instrumentation inst = ByteBuddyAgent.getInstrumentation (); / / enhanced premain (null, inst); / / call Class greetingType = Greeting.class. GetClassLoader () .loadClass (Greeting.class.getName ()); Method sayHello = greetingType.getDeclaredMethod ("sayHello", String.class); sayHello.invoke (null, "developer");}

Local debugging

In addition to making agent.jar, we can start it in the main function when we debug locally, as prompted above.

Print the result

WeaveListener onTransformation: baidu.bms.debug.Greetingstart a spanHi! Developerspan finishDisconnected from the target VM, address: '127.0.0.1 transport 61646, transport:' socket' can see that we have added the trace generation logic before and after the target method.

In real business, we often only need to capture the boxes used by the application, such as the RestTemplate method of Spring, to obtain the accurate call information of the Http method. This way of relying on this bytecode enhancement is decoupled from the business to the greatest extent.

This is the end of the introduction on "how to analyze the principle of non-invasive microservice probe". More related content can be searched for previous articles, hoping to help you answer questions and questions, please support the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.