Shulou(Shulou.com)05/31 Report--

Microservice call response returned to the Times ClassCastException case analysis, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Problem description

The problem is repeated. Demo is here.

I was dragged to see a problem the other day. A service (later called A service) runs in synchronous mode, and other micro services are invoked in RPC mode. There is no problem with local debugging. When running online, the interface of this service calling another service (later called Service B) will report an error. According to the log of a HttpClientFilter that they customize and expand, the called provider service has returned a reply message normally, but it will report ClassCastException later, and cannot transform InvocationException into the return value type of the business code. The log is as follows:

/ / the business logic is called [INFO] test () is called! Com.github.yhs0092.blogdemo.javachassis.service.ConsumerService.test (ConsumerService.java:20) / / the message returned by provider [INFO] get response is printed in the user-defined HttpClientFilter, status [200], content is [{"content": "returnOK"}] com.github.yhs0092.blogdemo.javachassis.filter.PrintResponseFilter.afterReceiveResponse (PrintResponseFilter.java:26) / / ClassCastException is thrown [ERROR] invoke failed Invocation=PRODUCER rest client.consumer.test org.apache.servicecomb.swagger.invocation.exception.DefaultExceptionToResponseConverter.convert (DefaultExceptionToResponseConverter.java:35) java.lang.ClassCastException: org.apache.servicecomb.swagger.invocation.exception.InvocationException cannot be cast to com.github.yhs0092.blogdemo.javachassis.service.TestResponse at com.sun.proxy.$Proxy30.test (Unknown Source) at com.github.yhs0092.blogdemo.javachassis.service.ConsumerService.test (ConsumerService.java:21) at sun.reflect.NativeMethodAccessorImpl. Invoke0 (NativeMethod) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.apache.servicecomb.swagger.engine.SwaggerProducerOperation.doInvoke (SwaggerProducerOperation.java:160) at org.apache.servicecomb.swagger.engine.SwaggerProducerOperation.syncInvoke (SwaggerProducerOperation.java:148) at org .apache.servicecomb.swagger.engine.SwaggerProducerOperation.invoke (SwaggerProducerOperation.java:115) at org.apache.servicecomb.core.handler.impl.ProducerOperationHandler.handle (ProducerOperationHandler.java:40)

In the process of analyzing the problem, they mentioned that since the online B service was still not upgraded from the old version, they replaced the interface jar package of the B service on which A service depended with the lower version.

Analysis process

Initial contact with this problem gives people a very strange feeling. If a consumer has already received a reply when calling provider, it will directly return the reply to the consumer's business logic code; in case there is a real error in the middle, the resulting InvocationException should also be "thrown" instead of trying to "return" to the consumer's business logic as shown in the log.

There is so little information available for analysis that you can only look back at the logic of the sdk code to see if you can reproduce the problem.

In the micro-service of RPC invocation mode, when business logic is called through provider interface, it is actually called through a proxy of provider interface type generated by ServiceComb. Behind this proxy, the source of the actual invocation process is in the org.apache.servicecomb.provider.pojo.Invoker class. In synchronous invocation mode, the key code that distinguishes how the reply is returned to the business logic is in the syncInvoke method:

Protected Object syncInvoke (Invocation invocation, SwaggerConsumerOperation consumerOperation) {Response response = InvokerUtils.innerSyncInvoke (invocation); if (response.isSuccessed ()) {/ / here, the result in the response is returned to the business logic return consumerOperation.getResponseMapper (). MapResponse (response) as a normal reply;} / / here is the exception logic, and the result in response is the error message, which will be thrown to the business logic throw ExceptionFactory.convertConsumerException (response.getResult ()) by InvocationException;}

An error in the online log indicates that the method did not go to the throw statement, but returned to the return statement.

The main process triggered in the InvokerUtils.innerSyncInvoke () method is Handler- > HttpClientFilter- > network thread. Since the reply message returned by the B service has been printed in the afterReceiveResponse () method of the user-defined HTTPClientFilter implementation class, the suspicion of the network thread part can be eliminated. The problem can only lie in Invoker, Handler and HTTPClientFilter. This exception needs to be blocked by catch and stuffed into response. At the same time, in order for the exception to be returned as response body instead of being "thrown", response.isSuccessed () needs to return true, which requires that the Http status code of response must be 2xx. By adding a custom HttpClientFilter to the demo and throwing an InvocationException with a status code of 200 in the afterReceiveResponse () method, we reproduce this problem, whose log characteristics are consistent with the online logs of the A service.

Root cause determination

A response with an exception in it, but the Http status code is 2xx, this scenario should not happen. After confirming to the A service developer that they did not directly operate the response in the custom Handler and HttpClientFilter, we could not draw a conclusion on the problem through the log, so we had to wait for the A service developer to reproduce the problem scenario locally.

Fortunately, this problem can be reproduced locally. The root cause is that after the B service interface jar package that A service depends on is replaced, the old version of the business interface response type has one more attribute than the new version, and the type of this attribute cannot be found, as shown below:

Class ResponseType {private InnerFieldType someField; / / the InnerFieldType here will report ClassNotFound}

So when DefaultHttpClientFilter's extractResult () method tries to deserialize the json string in Http body into a reply object in the business code, it throws an exception, which, after being wrapped as InvocationException, is returned by "return" instead of "throw", and no logs are printed in the process. The key code is on lines 85-89 of DefaultHttpClientFilter:

Try {return produceProcessor.decodeResponse (responseEx.getBodyBuffer (), responseMeta.getJavaType ());} catch (Exception e) {return ExceptionFactory.createConsumerException (e); / / exception is returned}

The exception returned by "return" is stuffed into the response as a normal reply object, and the status code of the response is the status code of the Http reply, so there is an error encountered online.

Summary

The lack of logs exposed by the ServiceComb framework during this positioning process will be fixed in subsequent versions. However, for developers, what is more important is that the service needs to be fully verified before deployment, and it is not advisable to temporarily replace relying on the jar package.

So how to locate this kind of problem in the process of local debugging? According to the scenario described in this article (RPC invocation mode, synchronous running mode), when a micro-service call is triggered in the business code, the processing flow of ServiceComb is roughly as follows:

Invoker-> InvokerUtils-> Handler-> HTTPClientFilter-> Network Thread

Invoker is a dynamic proxy in RPC invocation mode. When the business code is called through the provider interface, the parameters are first passed to the invoke () method. Because consumer works in synchronous mode, Invoker calls the innerSyncInvoke () method of InvokerUtils through the syncInvoke () method. Here, the next () method of Invocation is called, triggering the execution of the Handler chain. At the end of the handler chain is TransportClientHandler, which invokes the corresponding transport method to send the request. In the Rest over Vertx transport mode, what we need to focus on is the invoke () method of RestClientInvocation, which iterates through the beforeSendRequest () method that executes HttpClientFilter, and then dispatches the request to the network thread. The business thread is now in a state of waiting to return (CountDownLatch is used in the SyncResponseExecutor.waitResponse () method to wait).

When the request reply is returned, the RestClientInvocation.processResponseBody () method returns the Http response body to the business thread (by triggering the CountDownLatch of the SyncResponseExecutor). The reply is first processed by traversing the afterReceiveResponse () method of HttpClientFilter in RestClientInvocation, then through the callback processing of the Handler chain, and finally returned to the syncInvoke () method of InvokerUtils. Where Http response body is deserialized as a business interface return object in the extractResult () method of DefaultHttpClientFilter. This method determines how to treat the result according to the HTTP status code of response. If it is the status code of 2xx, the result in response will be returned to the business logic as a normal reply, otherwise the result will be wrapped in InvocationException and thrown to the business logic.

/ / protected void processResponseBody (Buffer responseBuf) {invocation.getResponseExecutor () .execute (()-> {/ / synchronous mode) in RestClientInvocation, the response return process is try {HttpServletResponseEx responseEx = new VertxClientResponseToHttpServletResponse (clientResponse, responseBuf) executed in the business thread from here on. For (HttpClientFilter filter: httpClientFilters) {/ / HttpClientFilter processes the return message body. A normal filter will return null Response response = filter.afterReceiveResponse (invocation, responseEx); if (response! = null) {/ / DefaultHttpClientFilter will deserialize the message body into a reply object and load response to return asyncResp.complete (response); / / trigger handler chain return through callback } catch (Throwable e) {asyncResp.fail (invocation.getInvocationType (), e); / / wrapper exception, trigger handler chain}} via callback);}

When analyzing such problems locally, you first need to know the process of sending the request, the entry of the RPC dynamic proxy, the start and end point of the Handler chain, and the call point of the HttpClientFilter. These are the key nodes in the process, and the scope of the problem can be roughly determined based on this information. As for further positioning, we need to analyze it according to specific problems.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.