Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to run the TensorFlow Lite model in Kuiper. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

EMQ X Kuiper is an edge lightweight Internet of things data analysis / streaming software that can run on a variety of resource-constrained Internet of things devices.

TensorFlow Lite is a set of tools that help developers run TensorFlow models on mobile, embedded, and Internet of things devices, making machine learning predictions on devices with low latency and small binary capacity.

By integrating Kuiper and TensorFlow Lite, users can analyze the data in the flow through an AI that contains a pre-built TensorFlow model. In this tutorial, we will guide you through building a kuiper plug-in that marks stream images (binary data) generated by edge devices through a pre-trained image recognition TensorFlow model.

Conditional preparation

To run the TensorFlow Lite interpreter, we need a trained model. This tutorial will not cover how to train and cover this model, you can learn more by looking at tflite converter. We can either train a new model or choose one online. In this tutorial, we will use mattn/go tflite's label image model. This repo creates a golang binding for tflite C API. We will also use it to implement our plug-ins.

Development plug-in

To integrate Kuiper and TensorFlow Lite, we will develop a custom Kuiper function plug-in for use by Kuiper rules. For example, we will create a LabelImage function whose input is binary type data that represents the image and the output is a string that represents the label of the image. For example, if there is a peacock in the input image, LabelImage (col) will output the peacock.

To develop a function plug-in, we need:

Create a plug-in go file. For example, in the kuiper source code, create a plugins/functions/labelImage/labelImage.go file.

Create an implementation api. Struct of the function interface.

Export struct.

The key to implementation is the Exec function. The pseudo code is as follows:

Func (f * labelImage) Exec (args [] interface {}, ctx api.FunctionContext) (interface {}, bool) {/ /. Initialize and verify / / decode the input image img, _, err: = image.Decode (bytes.NewReader (arg [0])) if err! = nil {return err False} var outerErr error f.once.Do (func () {/ / load the tag, tflite model and initialize the tflite interpreter}) / / run the interpreter on the input image / / return the most likely tag return result, true}

You also need to pay attention to the export of plug-ins. This function is stateless, so we will export only one instance of struct. All rules that use this function share an instance to avoid the overhead of creating instances and loading models. The model and label paths are specified when instantiated.

Var LabelImage = labelImage {modelPath: "labelImage/mobilenet_quant_v1_224.tflite", labelPath: "labelImage/labels.txt",}

Review this tutorial for detailed steps for creating a Kuiper plug-in. See labelImage.go for the complete source code.

Build and install the plug-in

To use the plug-in, we need to build it in the environment where Kuiper is running, and then install it in Kuiper.

Through a pre-built zip installation

If we use a debian-based Kuiper docker image with 1.1.1 or 1.1.1-slim tags, we can install the pre-built labelImage plug-in. For example, to install the Kuiper 1.1.2 plug-in in docker image emqx/kuiper:1.1.2-slim, the pre-built zip file is located in https://www.emqx.io/downloads/kuiper-plugins/v1.1.2/debian/functions/labelImage_amd64.zip. Run the rest command as shown below to install.

POST http://{{kuiperHost:kuiperRestPort}}/plugins/functionsContent-Type: application/json {"name": "labelImage", "file": "https://www.emqx.io/downloads/kuiper-plugins/v1.1.2/debian/functions/labelImage_amd64.zip"} manual build"

If you do not run Kuiper with the official Kuiper docker image, the pre-built labelImage plug-in will not apply due to the limitations of the golang plug-in. You need to build the plug-in manually. There are 3 steps to manually create the plug-in zip file:

Build the TensorFlowLite C API.

Build the labelImage plug-in.

Package the plug-in with the installation script.

Build TensorFlowLite C API

There is a very simple explanation from tensorflow repo about building C API. We will expand it step by step in this section. Note that the plug-in is only tested against TensorFlow v2.2.0-rc3, so we will build on this version. Taking ubuntu as an example, here are the build steps:

Install Python 3.

Copy the requirements.txt to the location you specified. Install the required python library: pip3 install-r requirements.txt. The requirements is from the corresponding TensorFlow version of tensorflow/tensorflow/tools/pip_package/setup.py.

Install TensorFlow's build tool Bazel.

Clone tesorflow repo and switch to the desired branch with the git checkout v2.2.0-rc3-b mybranch command.

Generate the target .so file, and the output will be in. / bazel-bin. Copy two so files to the tensorflow/lib folder.

$cd $tensorflowSrc$ bazel build-- config monolithic-c opt / / tensorflow/lite:libtensorflowlite.so$ bazel build-- config monolithic-c opt / / tensorflow/lite/c:libtensorflowlite_c.so$ mkdir lib$ cp bazel-bin/tensorflow/lite/libtensorflowlite.so lib$ cp bazel-bin/tensorflow/lite/c/libtensorflowlite_c.so lib

Install the so file.

Update the ldconfig file sudo vi / etc / ld.so.conf.d / tflite.conf.

Add the path {{tensorflowPath}} / lib to tflite.conf, then save and exit.

Run ldconfig: sudo ldconfig.

Check the installation result: ldconfig-p | grep libtensorflow. Make sure that two so files are listed.

Build the labelImage plug-in

Ensure that the Kuiper github repo is cloned. The plug-in source file is located in plugins/functions/labelImage/labelImage.go. Before building the plug-in, export the tensorflow repo and the path to the build library.

$cd {{kuiperRepoPath}} $export CGO_CFLAGS=-I/root/tensorflow$ export CGO_LDFLAGS=-L/root/tensorflow/lib$ go build-trimpath-- buildmode=plugin-o plugins/functions/LabelImage.so plugins/functions/labelImage/*.go

With these commands, the plug-in will be built into plugins/functions/LabelImage.so. For development purposes, you can restart Kuiper to automatically load the plug-in and test it. After the test is complete, we should package it as a zip file that can be used by the Kuiper plug-in to install API so that it can be used in other computers, such as production environments.

Package plug-in

Package all files and directories in the plugins/functions/labelImage directory into a zip file along with the built LabelImage.so. The file structure of the zip file should be similar to:

Etc

Labels.txt

Mobilenet_quant_v1_224.tflite

Lib

Libtensorflowlite.so

Libtensorflowlite_c.so

Install.sh

LabelImage.so

Tflite.conf

Install the packaged plug-in to the target system, as shown in the pre-built zip installation.

Run the plug-in

Once the plug-in is installed, we can use it in the rules. We will create a rule to receive byte data from the image from the mqtt topic and mark the image with the tflite model.

Define flow

Define the flow through Kuiper rest API. We create a stream called tfdemo with a binary format and a theme of tfdemo.

POST http://{{host}}/streamsContent-Type: application/json {"sql": "CREATE STREAM tfdemo () WITH (DATASOURCE=\" tfdemo\ ", FORMAT=\" BINARY\ ")} define rules

Define rules through Kuiper rest API. We will create a rule called ruleTf. We just read the image from the tfdemo stream and run the custom function labelImage on it. The returned result will be the label of the image recognized by AI.

POST http://{{host}}/rulesContent-Type: application/json {"id": "ruleTf", "sql": "SELECT labelImage (self) FROM tfdemo", "actions": [{"log": {}}]} input data

Here, we create a go program that sends image data to the tfdemo topic for processing by the rules.

Package mainimport ("fmt" mqtt "github.com/eclipse/paho.mqtt.golang"io/ioutil"time") func main () {const TOPIC = "tfdemo" images: = [] string {"peacock.png", "frog.jpg" / / other images you need} opts: = mqtt.NewClientOptions () .AddBroker ("tcp://yourownhost:1883") client: = mqtt.NewClient (opts) if token: = client.Connect () Token.Wait () & & token.Error ()! = nil {panic (token.Error ())} for _, image: = range images {fmt.Println ("Publishing" + image) Payload, err: = ioutil.ReadFile (image) if err! = nil {fmt.Println (err) continue} if token: = client.Publish (TOPIC, 0, false, payload) Token.Wait () & & token.Error ()! = nil {fmt.Println (token.Error ())} else {fmt.Println ("Published" + image);} time.Sleep (1 * time.Second)} client.Disconnect (0)}

Run pub.go, which will start entering the image into the tfdemo theme.

Examination result

Because our rule definition has only one goal: log, the result will be written to the log file. We fill the stream with peacock.png and frog.png images. If we check the log file, we will find:

Time= "2021-02-05 16:23:29" level=info msg= "sink result for rule ruleTf: [{\" labelImage\ ":\" peacock\ "}]" file= "sinks/log_sink.go:16" rule=ruleTftime= "2021-02-05 16:23:30" level=info msg= "sink result for rule ruleTf: [{\" labelImage\ ":\" bullfrog\ "}]" file= "sinks/log_sink.go:16" rule=ruleTf

The image marks are correct.

After reading the above, do you have any further understanding of how to run the TensorFlow Lite model in Kuiper? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.