Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the method of MapReduce operating Mysql". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the method of MapReduce operating Mysql".

The introduction of hadoop technology has been provoked and criticized by relational database researchers, arguing that MapReduce does not have the ability to store and process structured data in relational databases. To this end, the hadoop community and researchers have made a lot of efforts to support MapReduce access to relational databases in hadoop0.19, such as mysql,MySQL, PostgreSQL, Oracle and other database systems.

1. Read data from Mysql

Hadoop accesses the relational database mainly through the following interfaces: DBInputFormat class, package location: org.apache.hadoop.mapred.lib.db. DBInputFormat interacts with the database through the JDBC interface provided by the database vendor in Hadoop applications, and can use standard SQL to read records in the database. To learn DBInputFormat, you must first know two conditions.

Before using DBInputFormat, you must copy the JDBC driver you want to use to the $HADOOP_HOME/lib/ directory of each node in the distributed system.

When MapReduce accesses a relational database, a large number of frequently query and read data from MapReduce programs, which greatly increases the access load of the database. Therefore, the DBInputFormat interface is only suitable for reading a small amount of data, but not suitable for dealing with data warehouse. The methods to deal with the data warehouse are as follows: using the Dump tool of the database to output a large amount of data to be analyzed as text, and uploading it to Hdfs for processing.

The DBInputFormat class contains the following three built-in classes

Protected class DBRecordReader implementsRecordReader: used to read tuple records from a database table.

2.public static class NullDBWritable implements DBWritable,Writable: mainly used to implement the DBWritable interface. DBWritable interface to implement two functions, the first is write, the second is readFileds, these two functions are not difficult to understand, one is to write, the other is to read all fields. The prototype is as follows:

Public void write (PreparedStatement statement) throwsSQLException;public void readFields (ResultSet resultSet) throwsSQLException

Protected static class DBInputSplit implements InputSplit: mainly used to describe the scope of the input tuple collection, including start and end attributes. Start is used to represent the index number of the first record, and end represents the index number of the last record.

Here is a detailed description of how to use DBInputFormat to read database records. The specific steps are as follows:

DBConfiguration.configureDB (JobConf job, StringdriverClass, String dbUrl, String userName, String passwd) functions to configure JDBC drivers, data sources, and user names and passwords for database access. The JDBC driver of the MySQL database is "com.mysql.jdbc.Driver", and the data source is "jdbc:mysql://localhost/testDB", where testDB is the accessed database. UseName is usually "root", and passwd is the password of your database.

DBInputFormat.setInput (JobConf job, Class

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.