Shulou(Shulou.com)06/01 Report--

This article is about how to use the Golang template template to customize the analysis results in Kuiper, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Brief introduction

After data analysis and processing through Kuiper, users can send data analysis results to different systems using a variety of sink. For the same analysis results, different sink may not require the same format. For example, in a certain Internet of things scenario, when it is found that the temperature of a device is too high, you need to send a request to a rest service in the cloud, and at the same time, you need to send a control command to the device through the MQTT protocol, the two may need different data formats, so you need to "secondary process" the results from the analysis before you can send the data to different targets. The following describes how to use the data template (data template) in sink to achieve the "secondary processing" of the analysis results.

Introduction to Golang template

Golang templates apply a piece of logic to the data, and then format the data according to the logic specified by the user. Golang templates are commonly used in web page development, such as converting and controlling a data structure in Golang, converting it to HTML tags and outputting it to the browser. Golang's template (template) is used in Kuiper to achieve "secondary processing" of the analysis results, please refer to the following official introduction from Golang.

Templates are executed by applying them to a data structure. An Annotations in a template refers to an element in a data structure (typically a field in a structure, or a key in a map). Annotations are used to control execution and obtain values for display. The execution of the template iterates over the data structure and sets cursors, represented by the symbol "." it is called "dot" and points to the current position in the data structure during execution.

The input text of the template can be any text encoded by UTF-8. The "Actions"-- the data evaluation or control structure-- is defined by "{{" and "}}"; all text outside the action is left intact to the output, except for raw strings, and the action cannot cross lines (except comments).

Action (Actions)

Golang templates provide built-in actions that allow users to write various control statements to extract content. such as,

Output different content according to the judgment condition

{{if pipeline}} T1 {{else}} T0 {{end}}

Loop through the data and process it

{{range pipeline}} T1 {{else}} T0 {{end}}

Readers can see that actions are defined by {{}}. In the process of using Kuiper data templates, because the output is generally in JSON format, while JSON format is defined by {}, readers will find it difficult to understand when they are not familiar with the use of Kuiper data templates. Such as in the following example

{{if pipeline} {"field1": true} {{else}} {"field1": false} {{end}}

The meaning of the above expression is as follows (please note the delimiter of the action and the delimiter of JSON):

If the condition pipeline is met, the JSON string {"field1": true} is output

Otherwise, output JSON string {"field1": false}

Kuiper sink data format

The template of Golang can act on various data structures, such as map, slice, channel, etc., while the data type obtained by the data template in Kuiper's sink is fixed, which is a data type that contains Golang map slices, as shown below.

[] Map [string] interface {} slice (slice) data is sent in strips

The data flowing into sink is a map [string] interface {} sliced data structure, but when users send data to the target sink, they may need a single piece of data, not all the data. As described in this article on the integration of Kuiper and AWS IoT Hub, the sample data generated by the rules is shown below.

[{"device_id": "1", "t_av": 36.25,4, "t_max": 80, "t_min": 10}, {"device_id": "2", "t_av": 27, "t_count": 4, "t_max": 45, "t_min": 12}]

When sending to sink, you want each piece of data to be sent separately. First, you need to set the sendSingle of sink to true, and then use the data template: {{json.}}. The complete configuration is as follows, and users can copy it to the end of a sink configuration.

... "sendSingle": true, "dataTemplate": "{{json.}"

When sendSingle is set to true, Kuiper traverses the [] map [string] interface {} data type passed to sink, and the user-specified data template is applied to each piece of data during traversal.

Json is a function provided by Kuiper (users can refer to the Kuiper extension template function to learn more about Kuiper extensions). You can convert the passed parameters into JSON string output, and convert the contents of map into JSON strings for each piece of data traversed.

Golang also provides some built-in functions, and users can refer to more Golang built-in functions for more function information.

Data content conversion

Or for the above example, we need to do some conversion to the returned t_av (average temperature). The basic requirement of the conversion is to add different description words according to the different average temperature for the processing in the target sink. The rules are as follows

When the temperature is less than 30, the description field is "Current temperature is$t_av, it's normal."

When the temperature is greater than 30, the description field is "Current temperature is$t_av, it's high."

Assuming that the target sink still needs JSON data, the content of the data template is as follows

... "dataTemplate": "{\" device_id\ ": {{device_id}},\" description\ ":\" {{if lt .t _ av 30.0}} Current temperature is {{. T_av}}, it's normal.\ "{{else if ge .t _ av 30.0}} Current temperature is {{. T_av}}, it's high.\" {{end}} "" sendSingle ": true

In the above data template, the built-in action of {{if pipeline}} T1 {{else if pipeline}} T0 {{end}} looks complicated. Adjust it slightly, remove the escape and add indentation and typesetting as follows (Note: when generating Kuiper rules, the following optimized post-typesetting rules cannot be passed).

{"device_id": {{. Device_id}}, "description": "{{if lt .t _ av 30.0}} Current temperature is {{. T_av}}, it's normal." {{else if ge .t _ av 30.0}} Current temperature is {{. T_av}}, it's high. "{{end}

Use the binary comparison function built into Golang

Lt: less than

Ge: greater than or equal to

It is worth noting that in the lt and ge functions, the type of the second parameter value should be the same as the actual data type of the data in map, otherwise an error will occur. As in the example above, if the temperature is greater than 30, because the type of the actual average in map is float, the value of the second parameter needs to be passed in 30.0 instead of 30.

In addition, the template is still applied to each record in the slice, so you still need to set the sendSingle property to true. Finally, the content of the data template for the above data is as follows

{"device_id": 1, "description": "Current temperature is 36.25, it's high."} {"device_id": 2, "description": "Current temperature is 27, it's normal."} data traversal

By setting the sendSingle property of sink to true, you can traverse the slice data passed to sink. Here, we will introduce some more complex examples, such as how to traverse yourself through the traversal function provided in the data template, for example, the results of sink contain data of nested array types.

Suppose the content of the data flowing into the sink is as follows

{"device_id": "1", "values": [{"temperature": 10.5}, {"temperature": 20.3}, {"temperature": 30.3}]}

The demand is

When you find that a temperature value in the "values" array is less than or equal to 25, add a property named description and set its value to fine.

When you find a temperature value greater than 25 in the "values" array, add a property named description and set its value to high.

"sendSingle": true, "dataTemplate": "{{$len: = len .values}} {{$loopsize: = add $len-1}} {\" device_id\ ":\" {{. Device_id}}\ ",\" description\ ": [{{range $index $ele: = .values}} {{if le .temperature 25.0}}\ "fine\" {{else if gt .temperature 25.0}}\ "high\" {{end}} {{if eq $temperature $index}}] {{else}}, {{end} {end} "

The data template is complex, which is explained as follows

{{$len: = len .values}} {{$loopsize: = add $len-1}}, this section executes two expressions. The first len function takes the length of values in the data, and the second add subtracts its value by 1 and assigns it to the variable loopsize: since direct subtraction of 1 is not currently supported in Golang's expression, add is a function extended by Kuiper to achieve this function.

{\ "device_id\":\ "{{. Device_id}}\",\ "description\": [after this template is applied to the sample data, the JSON string {"device_id": "1", "description": [

{{range $index, $ele: = .values}} {{if le. Temperature 25.0}}\ "fine\" {{else if gt. Temperature 25.0}}\ "high\" {{end}} {{if eq $temperature $index}} {{else}, {{end}} {{end}}, this template looks complicated, but if you adjust it, remove the escape and add indentation, the typesetting is as follows It may seem clearer (note that the following optimized post-typesetting rules cannot be passed in when generating Kuiper rules).

{{range $index, $ele: = .values}} {{if le. Temperature 25.0}} "fine" {{else if gt. Temperature 25.0}} "high" {{end}} {{if eq $loopsize $index}] {{else}, {{end}} {end}}

The first condition determines whether the generation is fine or high;, and the second condition determines whether to generate a delimited array or the end of the array].

In addition, the template is still applied to each record in the slice, so you still need to set the sendSingle property to true. Finally, the content of the data template for the above data is as follows

{"device_id": "1", "description": ["fine", "fine", "high"]}

Through the data template function provided by Kuiper, the secondary processing of the analysis results can be realized to meet the needs of different sink objectives. However, readers can also see that due to the limitations of the Golang template itself, it will be clumsy to achieve more complex data conversion. I hope that the functions of the Golang template can be made more powerful and flexible in the future, so that it can support the processing of more complex requirements. At present, it is suggested that users can achieve some relatively simple data conversion through the data template; if users need to deal with the data more complex, and expand their own sink, they can directly deal with it in the implementation of sink.

In addition, the Kuiper team is planning to support the template function in the custom extension sink in the future, so that some complex logic can be implemented inside the function, and the user only needs a simple template function call when calling.

The above is how to use the Golang template template in Kuiper to customize the analysis results. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.