Shulou(Shulou.com)06/01 Report--

Flume and Sqoop are Hadoop data integration and collection systems, and their positioning is different. Here is an introduction based on personal experience and understanding:

Flume, developed by cloudera, has two major products: the architecture of Flume-og and Flume-ng,Flume-og is too complex, there will be data loss in the search, so give up. Now we use Flume-ng, mainly log collection, this log can be TCP system log data, can be file data (that is, usually we are in the Intel server, through the organization of the interface, or through the firewall collected logs), stored on HDFS, can be integrated with kafka, this is the function of Flume.

The Flume architecture is distributed, and the nodes and number of Flume can be expanded as needed. Its expansion has two meanings: one is horizontal, which is expanded according to the number and type of the original data source; the second is vertical, which can add more aggregation layers to do more process data processing, rather than data loading and then conversion.

The performance of Flume is high, reliable and highly available, and its reliability is mainly reflected in two aspects: on the one hand, there is a piece of data that is more important, in order to ensure the reliability of data transmission, two agent can point to this data, and two agent can be demonstrated to switch, if one of them fails, the other can also be transmitted. On the other hand, it can do cache penetration area inside agent, and the received data can be saved to disk and put into the database. Even if there is something wrong with agent, the data still exists.

Flume is for log collection, but more data is coming from structured databases, so we need Sqoop. Sqoop is a bridge between relational database and HDFS, which can realize the transmission of data between relational database and HDFS. So when do we pass the data to HDFS? The main purpose is to load new transactions and new accounts. When writing, in addition to hdfs, you can also write hive, and even directly build tables. And can be set in the source database to guide the entire database, or to import a table, or to import specific columns, which are common in the data warehouse ETL.

Sqoop allows incremental import, and there are two kinds of increments. One is to add directly (for example, new orders and transactions can be added); the other is to change the status. For example, a customer was previously a whitelist customer and the repayment is very good, but if a month is overdue, join the blacklist and return to the whitelist later, and the status is constantly changing, then you can no longer add the same as the transaction. What you need to do at this time is the zipper. Need a modified date, so is there any change in this state? if so, what about the previously loaded one? You can configure them through sqoop and have them updated in Hadoop when loaded. We know that HDFS files cannot be updated, so file merging is carried out at this time to erase the data of the text by merging.

When will the data be exported? The exported data lies in the analyzed data in Hadoop. We may need to download a data Mart and export the data based on this Mart, so sqoop can also export the data. The mechanism of sqoop export is: the default is that mysql,mysql is inefficient, so choose the second way-direct mode, using some export tools provided by the database itself. But the efficiency of these export tools is not high enough, higher is the professional customized connectors, currently customized connectors are MySQL, Postgres, Netezza, Teradata, Oracle.

The above is based on some of their own study and work experience summed up on the relevant knowledge of Flume and Sqoop, some specific knowledge is not involved here, if you want to know, you can learn by yourself. I usually follow the official Wechat accounts of "big data cn" and "big data Times Learning Center". Some information and knowledge points shared in them are of great help to me. I recommend you to have a look, looking forward to common progress!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.