Shulou(Shulou.com)06/01 Report--

This article introduces you how to synchronize Kettle data. The content is very detailed. Interested friends can refer to it for reference. I hope it can help you.

The previous business requirement is to achieve incremental synchronization of data. The specific description is: insert new data and update changed data. I can use insert update in output.

Now the business proposes a new requirement: in addition to the above description, when the source table data is deleted, the target table data is also deleted.

Although I highly recommend this, there are three reasons:

(1)Delete operation should not exist in the database. If you really want to delete a piece of data, theoretically, you should set the flag bit in the database and execute update, so that it will not be displayed to the front-end business, but the data will still exist in the database for easy traceability and rollback.

(2)ETL realizes the operation of deleting data. Once there is a problem, it is difficult to find and explain the reason, and it is easy to quarrel with each other and prevaricate.

(3)I need to redevelop ETL. Although it is difficult for me to accept it personally, but the business demand is heaven, I also have to bite the bullet to complete it. What may happen after that can only be "step by step" and resigned to fate. Document the key processes as follows:

1. There are 4 core objects in the transformation, including 2 inputs, 1 merge record and 1 data synchronization

[Table input is found in input: merge records in link; data synchronization is in output] Set source data and target data respectively, where the number of records is limited to 0, that is, all synchronization.

2. Merge record settings, note that the old data source is the data to be synchronized, that is, the target data; the new data is the source data, that is, the synchronized data. The flag field can be named arbitrarily (to comply with the naming convention). The matching key field is preferably the primary key (the unique value is barely a row), and the data field is the column to be synchronized (note that the key field must be included, otherwise the column of the target data is null, please note!!). As shown below:

3. After the merge is completed, there are four values for the flag field, namely:

"Identical" : key field exists in both old and new data sources with the same field value

"changed" : key field exists in both old and new data sources, but the field values are different

'new': key field not found in old data source

'deleted': no key fields found in new data source

The configuration of data synchronization needs to pay attention to the following points:

(1)No matter whether it is a query keyword or an update field, the flag field must be removed (note, remove the flag field!); Other fields are set according to business requirements;

(2)It was not possible to find operation field [null] in the input stream! "Wrong.

4. According to the need for regular operation, or manual operation, if the timing, then re-establish a job, set to start the relevant configuration can be.

In addition, it should be emphasized that if the source data and target data fields are different, the source data can use as to change the column name, and must pay attention to the name and column order consistent.

How to synchronize Kettle data is shared here. I hope the above content can be of some help to everyone and learn more knowledge. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.