Shulou(Shulou.com)05/31 Report--

This article mainly shows you "Presto custom function @ SqlNullable how to use", the content is easy to understand, clear, hope to help you solve doubts, the following let the editor lead you to study and learn "Presto custom function @ SqlNullable" this article.

When we see the title, we will think that the problem is caused by the @ SqlNullable annotation. Let's look at a piece of code first. It was this interesting code that made me struggle for more than 2 hours and caused the problem of Presto.

@ Description ("user_id") @ ScalarFunction ("user_id") @ SqlType (StandardTypes.VARCHAR) public static Slice userId (@ SqlType (StandardTypes.VARCHAR) Slice value) {String _ value = value.toStringUtf8 (); if (StringUtils.containsWhitespace (_ value)) {_ value = StringUtils.replace (_ value, "", "+");} return Slices.utf8Slice (makeErrorMsgBase64 (_ value);}

This code is very simple, that is, we decode the passed base64 string into the actual string, just from the code point of view is not a problem. The problem occurs when we actually run this function, and here is an example of how to use it:

Select user_id (str) from temp.users limit 100

There seems to be nothing unusual about the SQL executed, but it is such a simple function that causes the problem of Presto, that is, the java.lang.NullPointerException: undefined error. The details of this error are as follows

Java.lang.NullPointerException: undefined at io.prestosql.type.VarcharOperators.equal (VarcharOperators.java:53) at io.prestosql.$gen.CursorProcessor_20200927_063218_2398.filter (Unknown Source) at io.prestosql.$gen.CursorProcessor_20200927_063218_2398.process (Unknown Source) at io.prestosql.operator.ScanFilterAndProjectOperator$RecordCursorToPages.process (ScanFilterAndProjectOperator.java:323) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java:372) At io.prestosql.operator.WorkProcessorUtils.getNextState (WorkProcessorUtils.java:221) at io.prestosql.operator.WorkProcessorUtils$YieldingProcess.process (WorkProcessorUtils.java:181) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java:372) at io.prestosql.operator.WorkProcessorUtils.getNextState (WorkProcessorUtils.java:221) at io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2 (WorkProcessorUtils.java:200) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java At io.prestosql.operator.WorkProcessorUtils.lambda$flatten$6 (WorkProcessorUtils.java:277) at io.prestosql.operator.WorkProcessorUtils$3.process (WorkProcessorUtils.java:319) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java:372) at io.prestosql.operator.WorkProcessorUtils$3.process (WorkProcessorUtils.java:306) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java:372) at io.prestosql.operator.WorkProcessorUtils .getNextState (WorkProcessorUtils.java:221) at io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2 (WorkProcessorUtils.java:200) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java:372) at io.prestosql.operator.WorkProcessorUtils.getNextState (WorkProcessorUtils.java:221) at io.prestosql.operator.WorkProcessorUtils.lambda$finishWhen$3 (WorkProcessorUtils.java:215) at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process (WorkProcessorUtils.java:372) at io .prestosql.operator.WorkProcessorSourceOperatorAdapter.getOutput (WorkProcessorSourceOperatorAdapter.java:149) at io.prestosql.operator.Driver.processInternal (Driver.java:379) at io.prestosql.operator.Driver.lambda$processFor$8 (Driver.java:283) at io.prestosql.operator.Driver.tryWithLock (Driver.java:675) at io.prestosql.operator.Driver.processFor (Driver.java:276) at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor (SqlTaskExecution.java:1076) At io.prestosql.execution.executor.PrioritizedSplitRunner.process (PrioritizedSplitRunner.java:shichengoooo@163.com) at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run (TaskExecutor.java:484) at io.prestosql.$gen.Presto_341____20200925_110330_2.run (Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker .run (ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run (Thread.java:834)

From the above error, we calculate that the null pointer exception should be caused by the data, so what is the problem? The problem lies in the query str field. After the actual query, we found that there is a''data in this field, and there was a null pointer problem during the conversion, so we changed the source code of the UDF to

@ Description ("user_id") @ ScalarFunction ("user_id") @ SqlType (StandardTypes.VARCHAR) @ SqlNullablepublic static Slice userId (@ SqlNullable @ SqlType (StandardTypes.VARCHAR) Slice value) {String _ value = value.toStringUtf8 (); if (StringUtils.containsWhitespace (_ value)) {_ value = StringUtils.replace (_ value, "", "+");} return Slices.utf8Slice (makeErrorMsgBase64 (_ value));}

We added @ SqlNullable annotations to the methods and parameters to mark that this function can receive empty data, which seemed to be no problem. We reissued the function, executed SQL again and found the same problem, so we modified the code to the following

@ Description ("user_id") @ ScalarFunction ("user_id") @ SqlType (StandardTypes.VARCHAR) public static Slice userId (@ SqlNullable @ SqlType (StandardTypes.VARCHAR) Slice value) {String _ value = value.toStringUtf8 (); if (StringUtils.containsWhitespace (_ value)) {_ value = StringUtils.replace (_ value, "", "+");} return Slices.utf8Slice (makeErrorMsgBase64 (_ value));}

Delete the @ SqlNullable annotation on the method, and run it again to find that this error will not occur again, but the null pointer error is constantly reported in the Presto service, but it is no longer reported to the query client. I thought this problem had been solved, but something more interesting happened. After we successfully tested with version 343, the online version was 341.After upgrading, we found that the problem returned again. If you add the @ SqlNullable annotation to the method again, it will fix this problem on version 341. Now the problem has been reported to the official, and it is recommended that you use version 343!

The above is all the contents of the article "how to use the Presto Custom function @ SqlNullable". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.