Shulou(Shulou.com)06/01 Report--

This article mainly explains "EMQ X+TDengine how to build MQTT Internet of things visualization platform", the content of the article is simple and clear, easy to learn and understand, now please follow the editor's ideas slowly in depth, together to study and learn "EMQ X+TDengine how to build MQTT Internet of things visualization platform" bar!

Introduction to EMQ X

EMQ X is an open source MQTT messaging server based on highly concurrent Erlang/OTP platform, which supports millions of connections and distributed cluster architecture, and publishes and subscribes to it. EMQ X has a lot of built-in out-of-the-box features, and both its open source EMQ X Broker and Enterprise EMQ X Enterprise support storing device messages to TDengine through a rules engine.

What is TDengine?

TDengine is a big data platform specially designed and optimized for Internet of things, vehicle Internet, Industrial Internet, IT operation and maintenance. In addition to the core timing database function which is more than 10 times faster, it also provides caching, data subscription, streaming computing and other functions to minimize the complexity of R & D and operation and maintenance, and the core code, including cluster functions, is all open source.

TDengine is available in community, enterprise and cloud service editions. For more information on installation / usage tutorials, see the TDengine usage documentation.

Introduction to Grafana

Grafana is a cross-platform, open source measurement, analysis and visualization tool that can query and process data from various data sources for visual display. It can quickly and flexibly create client-side charts, panel plug-ins have many different ways to visualize metrics and logs, and there are rich dashboard plug-ins in the official library, such as heat maps, line charts, charts, etc. It supports data sources such as Graphite,TDengine, InfluxDB,OpenTSDB,Prometheus,Elasticsearch,CloudWatch and KairosDB, and supports independent / mixed query display of data items. You can create custom alarm rules and notify them to other messaging services or components.

Business scenario

This paper simulates the environment data collection scenario of the Internet of things. Assuming that there are some environmental data acquisition points, all the data of the collection points are transmitted to the acquisition platform (MQTT Publish) through MQTT protocol. The theme design is as follows:

Sensor/data

The format of the data sent by the sensor is JSON, and the data include temperature, humidity, noise volume, PM10, PM2.5, sulfur dioxide, nitrogen dioxide, carbon monoxide, sensor ID, area, acquisition time and so on.

{"temperature": 30, "humidity": 20, "volume": 44.5, "PM10": 23, "pm25": 61, "SO2": 14, "NO2": 4, "CO": 5, "id": "10-c6-1f-1a-1f-47", "area": 1, "ts": 1596157444170}

Now you need real-time storage to view the data at any subsequent time, with the following requirements:

Each device reports data every 5 seconds, and the database needs to store each piece of data for subsequent retrospective analysis.

Through the visualization system to view the index data in any region and any time interval, such as average, maximum and minimum.

Environmental preparation

All the components used in this article have Docker images. Except for EMQ X, which needs to modify a few configurations to facilitate operation, download and installation, both TDengine and Grafana are built with Docker.

Install package resources and use tutorials by referring to their respective websites:

EMQ X:EMQ official website https://www.emqx.io/cn/

TDengine: https://www.taosdata.com/cn/, the official website of Taos data

Grafana:Grafana official website https://grafana.com/

Install EMQ X

If you are a novice EMQ X user, it is recommended to get started quickly through EMQ X documents.

Visit the EMQ X download page to download the installation package suitable for your operating system. At the deadline of this article, the latest open source version of EMQ X is v4.1.1. The steps to download the zip package are as follows:

# # decompress the downloaded installation package unzip emqx-macosx-v4.1.1.zipcd emqx## to start EMQ X in console mode to facilitate debugging. / bin/emqx console

After successful startup, the browser visits http://127.0.0.1:18083 to access the EMQ X Management console Dashboard, and uses the admin public default username password to complete the initial login.

EMQ X Enterprise Edition 4.1.2 provides native TDengine write plug-ins for better performance and easier to use. Please move to the rules engine-write data to TDengine for viewing.

Install TDengine

To facilitate testing using Docker for installation (you need to map the network port), you can also install it using the installation package:

# # pull and launch container docker run-d-name tdengine-p 6030-6041 tdengine/tdengine:latest## start check container running status docker ps-an install Grafana

Use the following command to install and start Grafana through Docker:

Docker run-d-- name=grafana-p 3000UR 3000 grafana/grafana

After starting successfully, the browser visits http://127.0.0.1:3000 to access the Grafana visualization panel, and uses the default user name password of admin admin to log in for the first time. After login, follow the prompts to change the password and log in with the new password to enter the main interface:

Configure EMQ X to store data to TDengineTDengine to create databases and data tables

Enter the TDengine Docker container:

Docker exec-it tdengine bash

Create a test database:

Taoscreate database test

Create the sensor_data table. For more information about TDengine data structures and SQL commands, see TAOS SQL:

Use test;CREATE TABLE sensor_data (ts timestamp, temperature float, humidity float, volume float, PM10 float, pm25 float, SO2 float, NO2 float, CO float, sensor_id NCHAR, area TINYINT, coll_time timestamp); configure EMQ X rules engine

Open EMQ X Dashboared, go to the rules engine-> rules page, and click the create button to enter the creation page.

Rule SQL

The rule SQL is used for EMQ X messages and event filtering, and the following SQL indicates that payload data is filtered from the sensor/data topic:

SELECT payloadFROM "sensor/data"

Use the SQL test function to input test data to test the screening results. The test results are obtained and the output is as follows, indicating that the SQL is written correctly:

{"payload": "{\" temperature\ ": 30,\" humidity\ ": 20,\" volume\ ": 44.5,\" PM10\ ": 23,\" pm2.5\ ": 61,\" SO2\ ": 14,\" NO2\ ": 4,\" CO\ ": 5,\" id\ ":\" 10-c6-1f-1a-1f-47\ ",\" area\ ": 1,\" ts\ ": 1596157444170}"} "}

Response action

To support the development of different types of platforms, TDengine provides API that conforms to REST design standards. The simplest connection method is provided through RESTful Connector, that is, using the HTTP request to carry the authentication information and the SQL operation TDengine to be performed.

Using the send to Web service in the open source version of EMQ X, you can write data to TDengine through RESTful Connector. The upcoming version 4.1.1 of EMQ X Enterprise Edition will provide native higher performance write Connector.

Sending to the Web service requires two pieces of data, one is the associated resource, and the other is the message content template.

Associated resources: HTTP server configuration information, here is the RESTful Connector of TDengine

Message content template: here is the INSERT SQL carrying data. Note that we should specify the database name in SQL and enclose the character type in single quotation marks. The message content template is as follows:

INSERT INTO test.sensor_data VALUES (now, ${payload.temperature}, ${payload.humidity}, ${payload.volume}, ${payload.PM10}, ${payload.pm25}, ${payload.SO2}, ${payload.NO2}, ${payload.CO},'${payload.id}', ${payload.area}, ${payload.ts})

Creation process

Click the add button under the response action, select send data to the Web service in the pop-up box, and click New Resource to create a new WebHook resource.

Select Webhook for resource type, enter http://127.0.0.1:6041/rest/sql for request URL, select POST for request method, and add Authorization request header as authentication information.

The value of Authorization is Basic + TDengine's {username}: {password} string encoded by Base64, for example, the actual value entered after root:taosdata encoding is: Basic cm9vdDp0YW9zZGF0YQ==

Select the newly created resource on the response action creation page and fill in the message template content.

Generate simulation data

The following script simulates a scenario in which 10000 devices report a piece of simulation data every 5 seconds over the past 24 hours and send it to EMQ X.

Total number of items: 24 * 3600 / 5 * 100 = 1.72 million

Message TPS: 20

The reader installs Node.js. After modifying the configuration parameters as needed, you can start with the following command:

Npm install mqtt mockjs-save-registry= https://registry.npm.taobao.orgnode mock.js

Attachment: data generated by simulation and sent to EMQ X code. Please adjust the relevant parameters according to the performance of the cluster.

/ / mock.jsconst mqtt = require ('mqtt') const Mock = require (' mockjs') const EMQX_SERVER = 'mqtt://localhost:1883'const CLIENT_NUM = 100const STEP = 5000 / / simulated collection interval msconst AWAIT = 5000 / / dormancy time after each transmission Prevent message rate from being too fast msconst CLIENT_POOL = [] startMock () function sleep (timer = 100) {return new Promise (resolve = > {setTimeout (resolve, timer)})} async function startMock () {const now = Date.now () for (let I = 0 I

< CLIENT_NUM; i++) { const client = await createClient(`mock_client_${i}`) CLIENT_POOL.push(client) } // last 24h every 5s const last = 24 * 3600 * 1000 for (let ts = now - last; ts { const client = mqtt.connect(EMQX_SERVER, { clientId, }) client.on('connect', () =>

{console.log (`client ${clientId} connected`) resolve (client)}) client.on ('reconnect', () = > {console.log (' reconnect')}) client.on ('error', (e) = > {console.error (e) reject (e)} / * Generate mock data*/function generateMockData () {return {"temperature": parseFloat (Mock.Random.float (22)) 100) .tofixed (2)), "humidity": parseFloat (Mock.Random.float (12,86) .tofixed (2)), "volume": parseFloat (Mock.Random.float (20200) .tofixed (2)), "PM10": parseFloat (Mock.Random.float (0300) .tofixed (2)), "pm25": parseFloat (Mock.Random.float (0300) .fixed (2)), "SO2": parseFloat (Mock.Random.float (0) 50) .tofixed (2)), "NO2": parseFloat (Mock.Random.float (0,50) .tofixed (2)), "CO": parseFloat (Mock.Random.float (0,50) .tofixed (2)), "area": Mock.Random.integer (0,20), "ts": 1596157444170,}} visual configuration

After the component installation is completed and the simulation data is successfully written, follow the operation instructions of the Grafana visual interface to complete the visual configuration of the data required by the business.

Add data source (Add data source)

Grafana's TDengine data source requires manual installation of plug-ins, and the specific installation paradigm is based on the TDengine documentation.

Add the data source, that is, the data source information displayed. Select the TDengine type data source, and enter the connection parameters to configure. By default, the key configuration information is as follows:

Add a dashboard (New Dashboard)

Get the Grafana dashboard export file from the EMQ X Sample warehouse and import it to view the chart example.

After adding the data source, add the data dashboard information that needs to be displayed. The dashboard is a collection of multiple visual panels. After clicking New Dashboard, select + Query to add the data panel through query.

Four steps are required to create the panel: Queries (query), Visualization (visualization), General (chart configuration), Alert (alarm), and creation time

Average panel

Use Grafana's visual query building tool to query the average of all devices.

The following SQL queries the average of key metrics in the data according to the specified time period ($form $to) and the specified time interval ($interval):

Select avg (temperature), avg (humidity), avg (volume), avg (PM10), avg (pm25), avg (SO2), avg (NO2), avg (CO) from test.sensor_data where coll_time > = $from and coll_time

< $to interval($interval) Visualization 默认不做更改， General 里面修改面板名称为 历史平均值，如果需要对业务进行监控告警，可以在 Alert 里编排告警规则，此处仅做可视化展示，不使用此功能。 完成创建后，点击左上角返回按钮，该 Dashboard 里成功添加一个数据面板。点击顶部导航栏 保存 图标，输入 Dashboard 名称完成 Dashboard 的创建。 最大值、最小值面板 继续点击 Dashboard 的 Add panel 按钮，添加最大值、最小值图表。操作步骤同添加平均值，仅对查询中 SELECT 统计方法字段做出调整，调整为 AVG 函数为 MAX 与 MIN： select max(temperature), max(humidity), max(volume), max(PM10), max(pm25), max(SO2), max(NO2), max(CO), min(temperature), min(humidity), min(volume), min(PM10), min(pm25), min(SO2), min(NO2), min(CO) from test.sensor_data where coll_time >

= $from and coll_time < $to interval ($interval)

Dashboard effect

Save the dashboard, drag and adjust the size and position of each data panel, and finally get a better visual effect of the data dashboard. The upper right corner of the dashboard can select a time interval and automatically refresh time, when the equipment continues to send data collection data, the dashboard data value will change, achieving a better visualization effect.

FAQ

Q: why is there no icon data in Grafana?

Please drag the time range to check and make sure there is data within the selected period of time.

Q: what is the difference between EMQ X Open Source Edition and EMQ X Enterprise Edition in writing TDengine?

The open source version uses Webhook + TDengine RESTful Connector, with some performance loss on both sides, and the maximum write speed is about 700bits per second.

Enterprise Edition uses EMQ X native plug-ins, which can achieve 20000 writes per second.

Q: the rules are executed, but the data cannot be written?

Please check whether the authentication information is configured correctly and whether the request header, connection address, port and other information match the TDengin version.

Summary

So far, we have completed the construction of the system of data transmission, storage and display of the Internet of things with the help of EMQ X + TDengine. Readers can learn about the rich expansion capabilities of EMQ X and the application of the complete big data platform characteristics of TDengine in the data acquisition of the Internet of things. After deeply learning and mastering other functions of Grafana, users can customize a more perfect data visualization and even monitoring alarm system.

Thank you for reading, the above is the content of "how to build MQTT Internet of things visualization platform for EMQ X+TDengine". After the study of this article, I believe you have a deeper understanding of how EMQ X+TDengine builds the visualization platform of MQTT Internet of things, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.