Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to repair bug when submitting gRPC-spring-boot-starter projects". In daily operation, I believe many people have doubts about how to repair bug problems when submitting gRPC-spring-boot-starter projects. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "how to repair bug when submitting gRPC-spring-boot-starter projects". Next, please follow the editor to study!

What is gRPC-spring-boot-starter?

This is a spring-boot-starter project, used in the spring boot framework, quick and convenient use of grpc technology, out of the box. It provides the following functional features:

In spring boot applications, an embedded gRPC service is automatically configured and run through @ GrpcService.

Use @ GrpcClient to automatically create and manage your gRPC Channels and stubs

Support for Spring Cloud (registering services with Consul or Eureka or Nacos and getting gRPC server information)

Support Spring Sleuth as a distributed link tracking solution (if brave-instrument-grpc exists)

Support for global and custom gRPC server / client interceptors

Support for Spring-Security

Support for metric (based on micrometer/actuator)

Also applicable to (non-shaded) grpc-netty

Type selection gRPC-spring-boot-starter

The project undertaken by the blogger's new company uses grpc as the micro-service communication framework, the underlying framework of the project uses spring boot, and then the use of grpc is manually configured, the code is cumbersome to write, and this tedious templated code is filled with every micro-service project that uses grpc. So after the technology selection found the gRPC-spring-boot-starter this open source project, this project code quality is good, very standardized, the documentation is also relatively complete. However, in view of the fact that previous work experience has encountered problems with open source projects (bloggers' selection principle, if there is a suitable wheel, figure out the wheel, and then based on this wheel, without building a wheel), and generally the solution cycle is relatively long, so in the end, we did not directly adopt their distribution package, but after fork finished the project, we intend to maintain it ourselves. It is because of this that it is possible to quickly solve the problem later. It is also verified that the choice of second opening is correct.

Bug appears, grpc is not elegantly offline.

Fengfengzhuo refactored all the code, all changed to gRPC-spring-boot-starter and went online, and everything went well after launch, but there were some problems when the project was put into production for the second time. At this time, it is not certain that the problem is caused by switching between grpc implementations. The phenomenon is that there are a large number of request exceptions online. After the launch is complete, the exception disappears. A similar exception occurs with each subsequent scrolling update. At this time, it is easy to contact whether to switch the grpc implementation, grpc is not elegantly offline, resulting in rolling updates, a large number of ongoing requests are not processed properly, resulting in abnormal traffic? Because our online traffic is relatively large, there are a large number of requests almost all the time, so we require online services to support seamless rolling updates. If the traffic is relatively small, the problem may not be exposed, which explains the point previously discussed with colleagues, why such an obvious problem has not been detected early. But so far, it's all just speculation, and the truth goes on.

Locate bug and find the real reason

With the above guess, we directly find the class GrpcServerLifecycle that manages and maintains the GrpcServer lifecycle of gRPC-spring-boot-starter. This class implements spring's SmartLifecycle interface, which is used to register the hook of SpringContextShutdownHook. Its implementation is as follows:

@ Slf4jpublic class GrpcServerLifecycle implements SmartLifecycle {private static AtomicInteger serverCounter = new AtomicInteger (- 1); private volatile Server server; private volatile int phase = Integer.MAX_VALUE; private final GrpcServerFactory factory; public GrpcServerLifecycle (final GrpcServerFactory factory) {this.factory = factory;} @ Override public void start () {try {createAndStartGrpcServer () } catch (final IOException e) {throw new IllegalStateException ("Failed to start the grpc server", e);} @ Override public void stop () {stopAndReleaseGrpcServer ();} @ Override public void stop (final Runnable callback) {stop (); callback.run () } @ Override public boolean isRunning () {return this.server! = null & &! this.server.isShutdown (); @ Override public int getPhase () {return this.phase;} @ Override public boolean isAutoStartup () {return true;} / * Creates and starts the grpc server. * * @ throws IOException If the server is unable to bind the port. * / protected void createAndStartGrpcServer () throws IOException {final Server localServer = this.server; if (localServer = = null) {this.server = this.factory.createServer (); this.server.start (); log.info ("gRPC Server started, listening on address:" + this.factory.getAddress () + ", port:" + this.factory.getPort ()) / / Prevent the JVM from shutting down while the server is running final Thread awaitThread = new Thread (()-> {try {this.server.awaitTermination ();} catch (final InterruptedException e) {Thread.currentThread () .interrupt () }, "grpc-server-container-" + (serverCounter.incrementAndGet ()); awaitThread.setDaemon (false); awaitThread.start ();}} / * Initiates an orderly shutdown of the grpc server and releases the references to the server. This call does not * wait for the server to be completely shut down. * / protected void stopAndReleaseGrpcServer () {final Server localServer = this.server; if (localServer! = null) {localServer.shutdown (); this.server = null; log.info ("gRPC server shutdown.");}

That is to say, when the spring container is closed, ShutdownHook will be triggered and the GrpcServer service will be shut down. This is where the problem arises. As we can see from the stopAndReleaseGrpcServer () method, Grpc returns almost instantly after shudown (). This causes the Grpc service to be instantly recycled when the process receives the kill command, rather than waiting for the processing in execution to be completed. This judgment can be further confirmed from the document description of shutdown (), such as:

/ * Initiates an orderly shutdown in which preexisting calls continue but new calls are rejected. * After this call returns, this server has released the listening socket (s) and may be reused by * another server. * *

Note that this method will not wait for preexisting calls to finish before returning. * {@ link # awaitTermination ()} or {@ link # awaitTermination (long, TimeUnit)} needs to be called to * wait for existing calls to finish. * * @ return {@ code this} object * @ since 1.0.0 * / public abstract Server shutdown ()

The document points out that after calling shutdown (), new request traffic is no longer received, and ongoing requests will continue to be processed, but note that it does not wait for the existing call request to complete, but must use the awaitTermination () method to wait for the request to complete, that is to say, the logic of handling closure here lacks the logic of awaitTermination () waiting for the request in processing to complete.

Simulated environment, repeated verification

Verification method:

The problem with this scenario is very easy to verify, it just takes a little longer to simulate business blocking on the server side, and then kill the java process to see if the program will be kill immediately. If the normal elegant offline is turned off, it will wait for the blocking time after the process kill. Otherwise, no matter how long the business is blocked, the process will immediately kill.

Verify the located bug

First verify that, as mentioned above, without awaitTermination (), the process dies immediately. Directly use the demo program included in gRPC-spring-boot-starter, and add the following time-consuming code to simulate business execution in the server method:

@ GrpcService public class GrpcServerService extends SimpleGrpc.SimpleImplBase {@ Override public void sayHello (HelloRequest req, StreamObserver responseObserver) {HelloReply reply = HelloReply._newBuilder_ () .setMessage ("Hello = >"\ + req.getName ()) .build (); try {System._err_.println ("request received, blocking waiting"); TimeUnit._MINUTES_.sleep (1); System._err_.println ("blocking completed, request terminated") } catch (InterruptedException e) {e.printStackTrace ();} responseObserver.onNext (reply); responseObserver.onCompleted ();}}

The above code simulates the execution of the method for one minute, and then triggers the grpc client call. Then find the process number on the server side and kill it directly. It is found that the process is indeed kill immediately. Continue to increase the blocking time, from one minute to six minutes, repeat the test, or immediately kill down, without any wait.

Verify the effect after repair

First fix the above code, the correct shutdown logic should be as follows: after Grpc issues the shutdown instruction, the block waits for all requests to finish normally. At the same time, the blocking will tamp down the main process and not hang up inside.

Protected void stopAndReleaseGrpcServer () {final Server localServer = this.server; if (localServer! = null) {localServer.shutdown (); try {this.server.awaitTermination ();} catch (final InterruptedException e) {Thread.currentThread () .interrupt ();} this.server = null Log.info ("gRPC server shutdown.");}}

Similarly, as verified by the above steps, when the java process is dropped by kill, the java process is not immediately kill, but the thread is blocked by awaitTermination (), and the java process is not dropped by kill until the simulated business blocking in the business method ends, which is exactly the elegant offline shutdown effect we want to achieve. When being kill, the thread stack is as follows:

Even if it is kill, you can still print the following log [blocking completed, request terminated], which further verifies that the problem has been solved after repair:

At this point, the study of "how to repair bug when submitting gRPC-spring-boot-starter projects" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.