Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to use the Apache IoTDB file synchronization tool". In the daily operation, I believe many people have doubts about how to use the Apache IoTDB file synchronization tool. The editor consulted all kinds of data and sorted out a simple and easy-to-use method of operation. I hope it will be helpful to answer the doubts about "how to use the Apache IoTDB file synchronization tool". Next, please follow the editor to study!

Writing process

In order to better understand the time-consuming generation of data files, you need to give a brief introduction to the writing process.

IoTDB uses the LSM structure. The data is first written to the memory buffer memtable. If the memtable reaches a certain size, the disk will fall, and multiple memtable will correspond to a data file.

For example, a storage group has a time series with a write frequency of 1 second, and each data point is 16 bytes. The size of a memtable is 160 bytes, and the size of a TsFile file is 200 bytes. The storage group just started with an empty TsFile to write to.

(1) when memtable stores 10 data points for the first time, submit an asynchronous flushing task (appended to the current TsFile being written), and check the TsFile size. At this time, it is still 0 bytes, leaving the file unclosed.

(2) the asynchronous disk flushing task begins, and after brushing, the TsFile becomes 200byte.

(3) when memtable stores 10 data points for the second time, submit an asynchronous flushing task and check the TsFile size at the same time, and find that it reaches 200 bytes, so mark the close file.

(4) the task of asynchronous flushing starts, and after brushing, turn off the current TsFile.

In this example, write 2 memtable (20 pieces of data) to close the file once, write frequency is 1 second, the file generation time is about 20 seconds.

Synchronous process

The sender periodically checks whether there is a newly created Schema and a newly generated complete data file locally, and if it exists, it sends it to the receiver. The delay of synchronization is basically max (synchronization check interval, file generation time-consuming).

In the iotdb-sync-client.properties configuration file, there is a parameter sync_period_in_second, which controls how often the sender checks. If you configure 60 seconds, the maximum synchronization delay is 60 seconds, and if you configure 10 seconds, the delay is 20 seconds (it takes time to generate a file). As you can see, the synchronization delay has something to do with the configuration and the write frequency.

The sync_period_in_second in the configuration file can be matched relatively small, no big problem. The main problem is the generation interval of TsFile, which is controlled by memtable_size_threshold and tsfile_size_threshold, and the larger the memtable, the faster the historical data query. Therefore, there needs to be a balance between the minimum latency that can be achieved in synchronization and query performance.

For a relatively simple view of the file generation rate in your system, you can go to the data directory data/data/ storage group {/ partition} to see the last modification interval of the .resource file.

Test synchronization

Test synchronization on a machine, the sample script is linux environment, version 0.10.1. First download the binary package, unzip it twice and rename it, one sender and one receiver.

Start the receiver

Cd receiver## configure is_sync_enable=true## in conf/iotdb-engine.properties to start the receiver IoTDBnohup. / sbin/start-server.sh > / dev/null 2 > & 1 & # # start the receiver CLI and connect the local port 6667 with the root user by default. / sbin/start-cli.sh

Start the sender to prepare the data

Cd sender## configures sync_period_in_second=10## in rpc_port=6668conf/iotdb-sync-client.properties in conf/iotdb-engine.properties to start iotdbnohup. / sbin/start-server.sh > / dev/null 2 > & 1 & # # start the sending process nohup. / tools/start-sync-client.sh > / dev/null 2 > & 1 & # # start the sender CLI. Changed to port 6668. / sbin/start-cli.sh-h 127.0.0.1-p 6668-u root-pw root## enter insert into root.turbine1.d1 (timestamp,s1,s3) values Flush (this is the essence, force the memtable to be brushed and the files sealed)

The receiver verifies the data

# # enter select * from root in the receiver CLI

If you want to configure it yourself, you need to set enable_parameter_adapter to false, otherwise the system will automatically adjust the size of memtable and tsfile.

At this point, the study on "how to use the Apache IoTDB file synchronization tool" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.