Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to use Apache IoTDB's TsFile". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use Apache IoTDB's TsFile.

There are many file formats in big data's ecology, such as Parquet,ORC,Avro, etc., which are designed for nested data. These file formats generally have predefined schema, data is written in rows, organized by attributes, and stored in columns. However, these file formats generally can not meet the management requirements of time series data. For example, in some scenarios of time series data, usually each series is written independently and the timestamp is not aligned, and the query results need to be sorted by timestamp. TsFile (Time series File) is the file format we designed for time series data scenarios. Today, we mainly introduce the usage, mainly for version 0.10.

Working with scen

Because the file format is lightweight, it is suitable to be used as a data compression package at the edge end, which can be inside the device, industrial computer or factory level. The data generated on the device can be persisted to a file for storage at any time. The equipment mentioned here may be a fan, on which there will be multiple measuring points, such as wind speed sensor, temperature sensor and so on. The data collected by each sensor is a time series. Lenovo's IoT platform has been using TsFile to store time series data since 2017.

Therefore, the target scenario of TsFile is to manage the timing data of one or more devices.

Equipment-measuring point model

Device (DeviceId): a concept similar to a table.

MeasurementId: a device can have multiple points, similar to the concept of columns in a table.

Time series path (Path): Path (device Id, test point Id) can be defined by devices and test points.

Measuring point description information (MeasurementSchema): each time series corresponds to a description information, including data type, coding mode, compression mode.

Each time series has two columns: time column and value column.

Recently like to draw a picture, to draw a picture, basically like this, different equipment can have different measuring points.

Registration metadata

With TsFile, the first step is to register the metadata.

Registration time series: Path+MeasurementSchema

Each time series can be registered in this way.

Registration time series needs to provide a Path and a MeasurementSchema

String path = "test.tsfile"

Prior to 0.10, all devices shared a point table, and so did the schema of the same name Measurement (this is the source of the same restriction on the type of test point of the same name under a storage group in the IoTDB). After 0.10, each time series is truly independent and does not interfere with each other.

Register devices by template: device template + device

It is troublesome to register one by one, so it provides the function of a device template. Each template defines a set of MeasurementSchema, such as 10 test points, and when a device is associated with this template, 10 sequences are automatically registered.

First generate the device template, and then register the template.

Map template = new HashMap ()

Next, register the device and associate it to the template by template name:

TsFileWriter.registerDevice ("device_1", "template_1")

In this way, I registered 2 devices, each with 2 test points.

Register a template to write data in real time

This is an advanced simplified version. When we register only one device template, we can write data directly without registering the device. If it is found that the data written by this device is not registered during the writing process, it will directly find the MeasurementSchema with the same name in the template to register. This also inherits the fine tradition of pre-0.9 (in pre-0.9, TsFile can only register a template, and then you can write data).

Write data

There is a limit for data writing in TsFile, and each column needs to be written incrementally over time, otherwise correctness is not guaranteed.

Write a row of data per device: TSRecord

A TSRecord is the value of a device, a timestamp, and multiple measuring points. A row of data similar to a table.

Write a batch of data by device: Tablet

Haha, I see Tablet again. Yes, this structure is a structure that runs through TsFile and IoTDB Session. Represents the values of multiple measuring points of a device with multiple timestamps, similar to a subtable. This child table cannot have a null value.

Similarly, this write interface is fast and can reach a write speed of tens of millions of points per second.

Read data

The interface of the query receives a batch of paths and an expression (which can be filtered by time and value), which actually corresponds to two clauses, select and where.

When querying, the default table structure of TsFile is wide table, time, d1.m1, d1.m2, d2.m1, d2.m2. By default, this structure aligns the given query Path by Time and performs conditional filtering.

At this point, I believe you have a deeper understanding of "how to use Apache IoTDB's TsFile". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.