Shulou(Shulou.com)06/03 Report--

Unwittingly, everyone has been with DataPipeline for 3 years. During this period, thanks to the positive feedback and communication from customers, we summarized some common problems in our daily work and summarized them based on them.

In order to avoid being abrupt, we will start with more basic and general-purpose questions, and then release some slightly more complex questions and answers one after another. I hope you will continue to pay attention to our updates in the coming days.

Q1: read mode supported by DataPipeline

At the beginning of its establishment, A:DataPipeline has only one mode, which only supports real-time stream synchronization, which in our view is a trend in the future.

But later found that many customers actually have the need for batch synchronization. For example, banks may have some monthly and daily settlements every night, and securities companies also have similar settlement services. For some historical reasons, or for performance and database configuration considerations, some databases themselves may not be able to open change log. So in fact, real-time stream data can not be obtained from the source side in all cases.

Considering the above problems, we believe that in the process of supporting data fusion, a product must be able to support both batch and streaming processing modes, and provide different processing strategies in the product for performance and stability considerations. this is a relatively reasonable infrastructure.

For more information, please see: DataPipeline CTO Chen Su: consistent semantic guarantee for building a batch-stream integrated data fusion platform

Q2: what is the connection mode of the destination side

A: for relational databases, the write method is JDBC, and future versions will increase throughput by loading files. Other types of destinations vary according to the specific type. For example, the FTP destination uses FTP Client,Kafka and the destination uses Kafka Producer.

Q3: can data be encrypted by collecting and writing

A: if you want to encrypt the data content, you can use advanced cleaning.

Q4:DataPipeline installation and deployment mode

A:DataPipeline products are deployed in Docker containers and support Docker clusters; virtual environment (VMW) deployment is supported, but it is not recommended. DataPipeline is developing to support non-Docker deployments.

Does Q5:DataPipeline support graphical monitoring?

A:DataPipeline supports graphical monitoring of read and write rates, data volume, task progress, error queues, operation records, table structures, and so on.

Q6: how long is the database log retention policy appropriate?

A: for example, for MySQL Binlog retention policy, it is recommended to retain log policy > = 3 days.

Q7: how to ensure consistency in subsequent incremental import data

A:DataPipeline supports at least once synchronization mechanism by default to ensure that data will not be lost during synchronization. This is suitable for scenarios where there is a primary key at the source and a primary key at the destination, such as synchronization from a relational database to a relational database.

If there is no primary key deduplication capability like Hive, DataPipeline supports turning on the task-level end-to-end consistency option to ensure data consistency through a multi-phase commit protocol.

Q8: how to use the monitoring alarm on the project?

The data tasks of A:DataPipeline include monitoring Kanban and alarm. The alarm will be sent to the specified mailbox. Depending on the error type, you can choose to restart or notify technical support. DataPipeline will have engineers to assist customers in troubleshooting errors.

Q9: is it convenient to expand capacity

A:DataPipeline supports dynamic capacity expansion. When cluster resources are tight, there is no need to suspend existing tasks. After adding new nodes, the cluster capacity can be expanded.

Q10: if a piece of data changes many times and frequently, how can DataPipeline ensure the parallelism and order of the data?

The A:DataPipeline source will split the task into multiple non-interfering subtasks according to certain principles and execute them in parallel. For example, in the JDBC source reading scenario, if the task consists of multiple tables, each table is sequentially read by an independent thread, and the thread parallelism can be set in the task properties.

To ensure sequential write and read, each single sub-task creates an independent topic and sets a partition by default, so that when the target end consumes, only one consumer of the same topic is consumed, thus ensuring the order of consumption. If non-sequential consumption is acceptable, you can also create multiple partitions for a topic so that the destination can make better use of Kafka's parallelism to improve throughput.

This article focuses on 10 kinds of questions and answers. If you encounter similar questions at work, or if you have doubts about some questions, you are welcome to communicate with us.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.