Shulou(Shulou.com)05/31 Report--

This article shows you how to build a distributed atmospheric monitoring system with cloud capabilities based on Serverless. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Aforementioned

The following describes the relevant process after the data is uploaded to the cloud.

Due to the continuous construction of the project platform, the current open source information is the second version, so the content of this article is based on the capability of this version. This version of the architecture further optimizes data authentication and database protection.

It is divided into two parts:

Architecture composition: from a technical point of view, disassembling design logic

Module implementation: from a functional point of view, explain the implementation logic

[note 1] the effect of the first page of the second edition

[note 2] the effect of the first page of the third edition under construction

[note 3] in the third phase of the construction version, log service, object storage and Mini Program functions have been added. Provide flexible log storage and high-speed retrieval capabilities, low-cost database backup + recovery + migration synchronization capabilities, mobile interaction and other capabilities.

Architecture composition Phase II Architecture Diagram

Note: light blue indicates the focus of the architectural change relative to the first phase.

Deployment dimension

Terminal: equipment deployed at a monitoring point. The relevant design has been described earlier in this series.

Cloud: a platform deployed on the cloud. The related design will be described in this article.

Functional dimension

Logical split based on the basic components of the platform design. And according to the design requirements of the framework layer, matching the selection of specific products.

Specific functions and selection instructions:

Monitoring node: it is composed of soft and hard parts. The firmware is developed based on the Tencent Internet of things terminal operating system TencentOS tiny, and the hardware devices compatible with Tencent Cloud are connected with Tencent Cloud IoT explorer at the protocol level.

Local gateway: this part is mainly based on communication protocol conversion, does not parse specific data, and only needs to be configured according to the previous guidelines.

Internet of things access layer: based on Tencent Cloud IoT explorer products, it provides functions related to encryption, access and forwarding. After obtaining the data, the forwarding action based on http protocol is carried out and passed to the downstream API gateway.

Middleware layer: based on API gateway, it provides connection and load balancing capabilities, and based on authentication and flow control capabilities, it connects cloud functions and provides reliable data input and output services.

Logic layer: based on cloud function, it realizes the ability to receive data into storage (data solidification), timed statistical summary (data analysis), and output application data (data output, support display).

Storage layer: based on the cloud database TencentDB for MySQL, it stores monitoring data, summary data and platform configuration.

Display layer: based on Tencent Cloud Image, it provides the capabilities of map display, data display, line chart display and so on. And the configuration provides linkage processing, which improves the response speed and reduces the repeated transmission of resource files.

Module realization

The whole system, with the data as the core, has been designed, so this part has two components:

Data structure: explain the core data structure and association mode of each link.

Module description: explain the implementation of each functional module and the corresponding source code.

Data structure IoT explorer (Internet of things Development platform) push data structure

The original data that is pushed to the API gateway and extracted by the cloud function in the environment variable event.

The main part of it is:

Non-'body' part, which is information related to the api gateway

'body' = IoT explorer raw data

'body' = the raw data reported by the terminal node, encoded by base64

{"body": "{\" payload\ ":\" eyJtZXRob2QiOiJyZXBvcnQiLCJjbGllbnRUb2tlbiI6IjIwMjAtMDgtMThUMTE6MTc6NDkuNDAzWiIsInBhcmFtcyI6eyJQTTFfQ0YxIjo1LCJQTTJkNV9DRjEiOjYsIlBNMTBfQ0YxIjo3LCJQTTEiOjUsIlBNMmQ1Ijo2LCJQTTEwIjo3LCJwYXJ0aWNsZXNfMGQzIjo3OTIsInBhcnRpY2xlc18wZDUiOjI1NCwicGFydGljbGVzXzEiOjE1LCJwYXJ0aWNsZXNfMmQ1IjozLCJwYXJ0aWNsZXNfNSI6MywicGFydGljbGVzXzEwIjowLCJ2ZXJzaW9uIjoxNTEsIkVycm9yIjowfX0=\ ",\" seq\ ": 18639,\" timestamp\ ": 1597749469,\" topic\ ":\" $thing/up/property/?/?\ ",\" devicename\ ":\"?\ ",\" productid\ ":\"?\ "}", "headerParameters": {} "headers": {"accept-encoding": "gzip", "content-length": "511", "content-type": "application/json", "endpoint-timeout": "15", "host": "service-???-???.gz.apigw.tencentcs.com", "user-agent": "Go-http-client/1.1" "x-anonymous-consumer": "true", "x-api-requestid": "? 03a993936ae191f44651bread?", "x-b3-traceid": "? 03a993936ae191f44651pipeline?", "x-qualifier": "$LATEST"}, "httpMethod": "POST", "path": "/?", "pathParameters": {}, "queryString": {} "queryStringParameters": {}, "requestContext": {"httpMethod": "POST", "identity": {}, "path": "/?", "serviceId": "service-?", "sourceIp": "??", "stage": "release"}}

'after body' decrypts the base64 code:

{"clientToken": "2020-08-18T11:17:49.403Z", "method": "report", "params": {"Error": 0, "PM1": 5, "PM10": 7, "PM10_CF1": 7, "PM1_CF1": 5, "PM2d5": 6, "PM2d5_CF1": 6, "particles_0d3": 792 "particles_0d5": 254, "particles_1": 15, "particles_10": 0, "particles_2d5": 3, "particles_5": 3, "version": 151}} database form structure

For the detailed format of the database, please refer to github-sql

The purpose of each table is:

Config: system configuration tabl

Space: location table. Record the node deployment location.

Client: node table. Record the node information and the latest key information, associate the space table.

Base_data: log table. Record the information reported each time, and associate space and client tables.

Aggregate_data: summary information table. Record the data and information collected by hours, days and other periods.

Temp_base_data: temporary log table. If the reported device name is not in the node table, it will be stored in this table, which is mainly used for debugging.

The data structure module of API interface describes the data storage.

Module function: receive IoT explorer push data, match and convert it into database

Use products: API gateway, serverless cloud function

Associated source code: github-upload

Deployment method: please refer to the related article online formaldehyde Monitoring "4.2.2 serverless Cloud function / 4.2.3 Cloud API Gateway" section

Terminal online update

Module function: trigger in 3 minutes to update terminal information for quick query and display

Use product: cloud function

Associated source code: github-client_active_new

Deployment method: please refer to the relevant article online formaldehyde monitoring "4.2.2 serverless cloud function / 4.2.3 cloud API gateway" section, but change to "timing trigger" mode

Data statistics

Module function: trigger by hour and day, statistics and monitoring data, for quick query and display

Use product: cloud function

Associated source code: github-aggregate

Deployment method: please refer to the relevant article online formaldehyde monitoring "4.2.2 serverless cloud function / 4.2.3 cloud API gateway" section, but change to "timing trigger" mode

Data Interface-Terminal Summary Information Table

Module function: provide data query capability through API gateway, respond to cloud image calls, and provide terminal summary information table

Use products: API gateway, cloud function

Associated source code: github-client_info

Deployment method: please refer to the related article online formaldehyde Monitoring "4.2.2 serverless Cloud function / 4.2.3 Cloud API Gateway" section

Data interface-terminal single point information table

Module function: provide data query capability through API gateway, respond to cloud image calls, and provide terminal single point information table

Use products: API gateway, cloud function

Associated source code: github-client_data

Deployment method: please refer to the related article online formaldehyde Monitoring "4.2.2 serverless Cloud function / 4.2.3 Cloud API Gateway" section

Data presentation

Module function: provide Web end display ability, including map drawing points, list display, broken line display, etc.

Product: Tencent Cloud Picture

Associated source code: none, currently configured manually

Deployment method: please refer to the relevant article online formaldehyde monitoring "4.2.4 Tencent Cloud Picture" section, but when selecting data sources, use "API" instead. The "home page" map uses the "data interface-terminal summary information table" interface, and the "single point data page" uses the "data interface-terminal single point information table" interface. For the specific API URL called, please correspond to the service link provided by the API gateway.

Cost analysis of the second phase

The number of tables above shows the calculation results reported by each node every 15 seconds.

The price of hardware is different due to the difference between manufacturer and purchase quantity. Here, based on the cloud service cost, if the scale of 10000 nodes is reported per node every minute, the annual cost of single-node cloud service:

Phase II architecture RMB 3.335 (database / year + cloud map / year) / 10000 + (API gateway + cloud function) / 4 = (468 / 48) / 10000 + (9 / 2. 1) / 4

Three-phase architecture 1.09 yuan (message queue / year + database / year * 2 + cloud image / year) / 10000 + (cloud function + log service + object storage) / 4 = (32280468048) / 10000+ (2.1' 2.2402) / 4

The second phase has no object storage and log service module, but it has been used for several months in the third phase of construction, so the actual data can be included in the statistics.

Follow-up plan

The third phase of construction will be completed, and the contents of the plan include:

Interactive optimization (done): enable the space table to show the location.

Interactive construction (doing): increase Mini Program capacity, provide mobile query capability, and alarm push capability.

Disaster recovery Optimization 1 (done): increase object storage, provide low-cost data backup, recovery, asynchronous migration synchronization capabilities.

Disaster recovery Optimization 2 (todo): add message queue CKafka to deal with the risk of data accumulation and loss in the case of network exception and database exception.

Disaster recovery Optimization 3 (doing): add log service for data reconciliation to deal with possible data loss or confusion in multi-link data processing.

Preview of the third phase of the architecture diagram

Note: light blue indicates the focus of architectural changes relative to Phase II.

The third edition partially completes the WEB display of the effect.

The above content is how to build a distributed atmospheric monitoring system with the cloud capability based on Serverless. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.