Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to create an Apache IoTDB data model". Many people will encounter such a dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

What is time series data?

The Internet of things was born in 1999, under the continuous innovation of its concept and technology, more and more ubiquitous devices and facilities are connected through the network, and continue to send live data to the cloud.

Take the national meteorological observation station as an example, there are nearly 60,000 meteorological observation stations in the country, and each meteorological observation station has 70 kinds of meteorological physical quantities to be collected. Each train in a city subway has 3200 indicators to be measured, and the number of trains in the city is up to 300. In the monitoring of server operation and maintenance, a server needs to monitor more than ten indicators such as IOPS, CPU, network and so on. Two concepts are shown in these examples: equipment and metrics. The so-called measurement index (also known as working condition, measuring point) refers to the data items that users are concerned about, which can reflect a certain state of the target, such as CPU utilization, temperature, humidity and so on. A device is an entity with a series of metrics, such as a server, a process, a train, a weather station, and so on. A metric of a device forms a unique identification of time series data.

With the passage of time, this time series data will produce a series of (timestamp, value) binary data points, forming a time series data set. Therefore, we define a time series as an infinite set of time series identifiers (devices and metrics), a series of timestamps and data value pairs. A time series database will manage millions or even tens of millions of such time series.

IoTDB data model and its manual creation method

The metadata management of IoTDB adopts the structure of directory tree, which is used between different levels. Split. The root node defaults to root, but there are three main concepts. Storage groups, devices, measuring points.

Manually create a storage group:

Set storage group to root.FU01

Manually create a time series:

Create timeseries root.FU01.deviceType1.AZQ01.Temperature with datatype=FLOAT, encoding=GORILLA, compression=SNAPPY

The device does not need to be created, and the penultimate layer is used as a device by default when creating a time series. Taking the above time series as an example, the device ID will be set to root.FU01.deviceType1.AZQ01. For multiple measuring points of a device with a timestamp, it is best to write at the same time to avoid out-of-order data as far as possible.

When enough time series are created, the metadata looks like the following tree:

Six data types are currently supported

BOOLEAN 、 INT32 、 INT64 、 FLOAT 、 DOUBLE 、 TEXT

There are four main coding methods.

TS_2DIFF (default encoding of time column): applicable to INT32, INT64RLE: applicable to INT32, INT64, FLOAT, DOUBLE (lossy compression for FLOAT and DOUBLE, 2 decimal places are retained by default, float_precision can be modified in configuration file) GORILLA: applicable to FLOAT, DOUBLEPLAIN: full match

Compression method:

UNCOMPRESSED, SNAPPY (default)

Recommended modeling method

Storage groups: about 10-50 are recommended. Each storage group is an independent storage engine. Increasing storage groups can increase write parallelism.

Equipment: less than 100000 recommended

Total number of sequences: less than 10 million is recommended

This modeling method is fine under normal load. If the system prompts you that the system load is too high, you can set enable_parameter_adapter to false. You need to configure parameters manually to prevent memory from exploding. The simple rules are as follows:

Memtable_size_threshold=tsfile_size_threshold

= IoTDB allocated memory / 2 / number of storage groups / 4 (disordered data)

= IoTDB allocated memory / 2 / number of storage groups / 2 (no disordered data)

MAX_HEAP_SIZE is set in IoTDB allocation memory in conf/iotdb-env.sh.

The configuration file is in conf/iotdb-engine.properties.

Most of the recommended load according to this adjustment is no problem, no matter how high the load is, you can contact us. This manual adjustment parameter will be removed in version 0.11, liberating productivity!

One method is to determine whether there is disorder: as long as each device writes an incremental timestamp, there is no out-of-order data, otherwise it is possible to produce out-of-order data.

For example: the device root.turbine.d1 has three measuring points S1, S2, S3

# No out-of-order data insert into root.turbine.d1 (timestamp,s1,s2,s3) values; insert into root.turbine.d1 (timestamp,s1,s2,s3) values (2); # time stamps write 2, then 1, there may be out-of-order data insert into root.turbine.d1 (timestamp,s1,s2,s3) values (2); insert into root.turbine.d1 (timestamp,s1,s2,s3) values (1) # the timestamp writes 1 first, then 1. Although it is a different test point, it still belongs to a device, and there may be out-of-order data insert into root.turbine.d1 (timestamp,s1,s2) values (1m 1m 2); insert into root.turbine.d1 (timestamp,s3) values (1m 3)

Automatically create a metadata schema

In addition to manually creating metadata, it also supports automatic creation of metadata, which is done during the data writing process. Mainly for scenarios where you don't know the total number of sequences in advance and consume message queues for writing in real time, you don't need to create sequences for every piece of data in the code.

When we write data to a time series, we first check whether its storage group exists, and automatically create it if it does not exist. We define root as layer 0, and the storage group defaults to the first layer, that is, the layer under root. You can modify the level default_storage_group_level created by default in the configuration file.

The automatically created data type is automatically inferred from the type of value written to it. If the data passed in is in string format, that is, it is written with the insert statement of JDBC, or written by the insertRecord (s) interface with the value type of String in Session, it will be judged according to the format of the value. There are mainly four kinds of string format, as well as the default type:

No. Integer: such as 123 = > FLOAT belt. Floating-point numbers for example: 12.34 = > FLOAT Boolean: true,false = > BOOLEAN other types: abc,124sa = > TEXT

The default types of strings in the first three formats can all be configured in the configuration file, (version 0.10.0, current master branch, near the boolean_string_infer_type parameter)

Simple trial

The script defaults to the foreground and needs to be started manually in the background.

Nohup. / sbin/start-server.sh > / dev/null 2 > & 1 &

Next, you can start the Cli command line:

. / sbin/start-client.sh-h 127.0.0.1-p 6667-u root-pw rootor./sbin/start-client.sh (root is used by default to connect to this machine)

In version 0. 10. 0, it will be renamed start-cli.sh.

That's all for "how to create an Apache IoTDB data model". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.