Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to use Hadoop and Couchbase". In daily operation, I believe many people have doubts about how to use Hadoop and Couchbase. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use Hadoop and Couchbase together". Next, please follow the editor to study!

Hadoop and data processing

Hadoop combines many important features, which makes Hadoop useful for shredding large amounts of data into smaller, practical blocks.

The main component of Hadoop is the HDFS file system, which supports the distribution of information across the entire cluster. Information stored in this distributed format can be processed separately on each cluster node through a system called MapReduce. The MapReduce process converts the information stored in the HDFS file system into smaller, processed, and more manageable blocks.

Because Hadoop can run on multiple nodes, it can be used to process large amounts of input data and simplify it into more practical blocks of information. This process can be handled using a simple MapReduce system.

MapReduce transforms incoming information (not necessarily in a structured format), transforming it into a structure that is easier to use, query, and process.

For example, a typical use is to process log information from hundreds of different applications so that specific problems, counts, or other events can be identified. By using the MapReduce format, you can start measuring and looking for trends, converting the usual large amount of information into smaller blocks of data. For example, when viewing the logs of a Web server, you might want to see errors that occur in a specific range on a particular page. You can write a MapReduce function to identify a specific error on a particular page and generate that information in the output. Using this method, you can refine multiple lines of information from the log file to get a much smaller collection of records that contain only error messages.

Understand MapReduce

MapReduce works in two phases. The map process takes incoming information and maps it to some standardized format. For some information types, this mapping can be direct and explicit. For example, if you want to process input data such as the Web log, you can extract only one column of data from the text of the Web log. For other data, the mapping may be more complex. When working with text information, such as research papers, you may need to extract phrases or more complex blocks of data.

The reduce phase is used to collect and summarize data. Reduction can actually occur in many different ways, but the typical process is to process a basic count, sum, or other statistics based on individual data from the mapping phase.

Imagine a simple example, such as the number of words used as an example MapReduce in Hadoop, where the mapping phase decomposes the original text to identify each word and generate an output data block for each word. The reduce function takes the blocks of information for these maps and refines them so that they are incremented on each unique word you see. Given a text file containing 100 words, the mapping process produces 100 blocks, but the reduction phase can summarize this, providing the number of unique words (such as 56) and the number of occurrences of each word.

With the Web log, the map takes the input data, creates a record for each error in the log file, and then generates a data block for each error that contains the date, time, and page that caused the problem.

Within Hadoop, the MapReduce phase occurs on each node that stores each block of source information. This enables Hadoop to handle large sets of information by allowing multiple nodes to process data at the same time. For example, for 100 nodes, you can process 100 log files at the same time, simplifying a lot of GB (or TB) information much faster than through a single node.

Hadoop information

A major limitation of core Hadoop products is the inability to store and query information in the database. Data is added to the HDFS system, but you cannot ask Hadoop to return a list of all data that matches a particular dataset. The main reason is that Hadoop does not store, structure, or understand the structure of the data stored in HDFS. This is why MapReduce systems need to analyze and process information into a more structured format.

However, we can combine the processing power of Hadoop with more traditional databases, allowing us to query data generated by Hadoop through its own MapReduce system. There are many possible solutions, including some traditional SQL databases, but we can maintain the MapReduce style by using Couchbase Server (which is very effective for large datasets).

The basic structure of data sharing between systems is shown in figure 1.

Figure 1. The basic structure of data sharing between systems

Install Hadoop

If you have not already installed Hadoop, the easiest way is to use a Cloudera installation. To maintain compatibility between Hadoop, Sqoop, and Couchbase, the solution for * * is to use a CDH3 installation (see Resources). To do this, you need to use Ubuntu versions 10.10 to 11.10. Later versions of Ubuntu introduce incompatibility issues because they no longer support a package required for Cloudera Hadoop installation.

Before installing, make sure that a Java ™virtual machine is installed and that the correct home directory is configured for JDK in the JAVA_HOME variable. Note that you must have a complete Java development kit, not just the Java runtime environment (JRE), because Sqoop compiles the code to export and import data between Couchbase Server and Hadoop.

To use CDH3 installation on Ubuntu and similar systems, you need to perform the following steps:

1. Download the CDH3 configuration package. This adds the configuration of the CDH3 source file to the apt repository.

2. Update your repository cache: $apt-get update.

3 、 . Install the main Hadoop package: $apt-get install hadoop-0.20.

4. Install the Hadoop component (see listing 1)

Listing 1. Install Hadoop components

$for comp in namenode datanode secondarynamenode jobtracker tasktrackerdoapt-get install hadoop-0.20-$compdone

5. Edit the configuration file to ensure that you have set up the core components.

6. Edit / etc/hadoop/conf/core-site.xml to look like listing 2.

Listing 2. Edited / etc/hadoop/conf/core-site.xml file

Fs.default.namehdfs://localhost:9000

This configures the default hdfs location where the data is stored.

Edit / etc/hadoop/conf/hdfs-site.xml (see listing 3).

Listing 3. Edited / etc/hadoop/conf/hdfs-site.xml file

Dfs.replication1

This supports replication of stored data.

Edit / etc/hadoop/conf/mapred-site.xml (see listing 4).

Listing 4. Edited / etc/hadoop/conf/mapred-site.xml file

Mapred.job.trackerlocalhost:9001

This implements MapReduce's job tracker.

7. *, edit the Hadoop environment so that it correctly points to your JDK installation directory in / usr/lib/hadoop/conf/hadoop-env.sh. There will be a commented out JAVA_HOME variable line. You should uncomment it and set it to your JDK location. For example: export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk.

Now, start Hadoop on your system. The easiest way is to use the start-all.sh script: $/ usr/lib/hadoop/bin/start-all.sh.

Assuming that all the settings are configured correctly, you should now have a running Hadoop system.

Overview of Couchbase Server

Couchbase Server is a clustered, document-based database system that uses a cache layer to provide very fast data access, storing most of the data in RAM. The system uses multiple nodes and a cache layer that is automatically distributed over the entire cluster. This enables flexibility where you can expand and tighten your cluster to take advantage of more RAM or disk Imando O to help improve performance.

All data in Couchbase Server will eventually be persisted on disk, but initially write and update operations will be performed through the cache layer, which is the source of high performance and an advantage that we can take advantage of when we process Hadoop data to obtain real-time information and query content.

The basic form of Couchbase Server is a basic document and key / value-based storage. The information provided by the cluster can be retrieved only if you know the document ID. In Couchbase Server 2.0, you can save the document in JSON format and then use the view system to create a view on the stored JSON document. A view is an MapReduce combination executed on a document stored in a database. The output from the view is an index that matches the structure you defined through the MapReduce function. The existence of the index provides you with the ability to query the underlying document data.

We can use this feature to get processed information from Hadoop, store it in Couchbase Server, and then use it as a basis for querying the data. Couchbase Server can easily use a MapReduce system to process documents and create indexes. This provides a certain level of compatibility and consistency between the methods used to process data.

Install Couchbase Server

It is easy to install Couchbase Server. Download the Couchbase Server version 2.0 for your platform from the Couchbase website (see Resources) and install the package using dpkg or RPM (depending on your platform).

After installation, Couchbase Server starts automatically. To configure it, open a Web browser and point it to your machine's localhost:8091 (or access it remotely using the machine's IP address).

Follow the on-screen configuration instructions. You can use most of the default settings provided during installation, but the most important settings are the location of the data file that writes the data in the database, and the amount of RAM that you assign to Couchbase Server.

Enables Couchbase Server to communicate with Hadoop connectors

Couchbase Server uses the Sqoop connector to communicate with your Hadoop cluster. Sqoop provides a connection to transfer data in bulk between Hadoop and Couchbase Server.

Technically, Sqoop is an application designed to transform information between a structured database and Hadoop. The name Sqoop actually comes from SQL and Hadoop.

Install Sqoop

If you use CDH3 installation, you can use the newspaper manager to install Sqoop:$ sudo apt-get install sqoop.

This will install Sqoop in / usr/lib/sqoop.

Note: a * * bug in Sqoop indicates that it sometimes attempts to transfer uowu data sets. The patch is included in Sqoop version 1.4.2. If you encounter problems, try using V1.4.2 or later.

Install Couchbase Hadoop Connector

Couchbase Hadoop Connector is a collection of Java jar files that support connections between Sqoop and Couchbase. Download the Hadoop connector from the Couchbase website (see Resources). The file is packaged as a zip file. Extract it, and then run the install.sh script in it to provide the location of the Sqoop system. For example: $sudo bash install.sh / usr/lib/sqoop

This will install all necessary libraries and configuration files. Now we can start exchanging information between the two systems.

Import data from Couchbase Server to Hadoop

Although this scenario is not the one we will deal with directly here, it is important to note that we can import data from Couchbase Server into Hadoop. This can be useful if you load a lot of data in Couchbase Server and want to use Hadoop to process and simplify it. To do this, you can load the entire dataset from Couchbase Server into a Hadoop file in HDFS using the following command: $sqoop import-- connect http://192.168.0.71:8091/pools-- table cbdata.

The URL provided here is the location of the Couchbase Server bucket pool (bucket pool). The table specified here is actually the name of the directory in HDFS where the data will be stored.

The data itself is stored as a key / value dump of information from the Couchbase Server. In Couchbase Server 2.0, this means that the data is written using a unique document ID, containing the JSON value of the record.

Write JSON data to Hadoop MapReduce

To exchange information between Hadoop and Couchbase Server, you need to express it in a common language, in this case JSON (see listing 5).

Listing 5. Output JSON in Hadoop MapReduce

Package org.mcslp;import java.io.IOException;import java.util.*;import org.apache.hadoop.fs.Path;import org.apache.hadoop.conf.*;import org.apache.hadoop.io.*;import org.apache.hadoop.mapred.*;import org.apache.hadoop.util.*;import com.google.gson.*;public class WordCount {public static class Map extends MapReduceBase implements Mapper {private final static IntWritable one = new IntWritable (1); private Text word = new Text () Public void map (LongWritable key, Text value, OutputCollector output,Reporter reporter) throws IOException {String line = value.toString (); StringTokenizer tokenizer = new StringTokenizer (line); while (tokenizer.hasMoreTokens ()) {word.set (tokenizer.nextToken ()); output.collect (word, one);}} public static class Reduce extends MapReduceBase implements Reducer {class wordRecord {private String word;private int count;wordRecord () {} public void reduce (Text key,Iterator values,OutputCollector output,Reporter reporter) throws IOException {int sum = 0 While (values.hasNext ()) {sum + = values.next (). Get ();} wordRecord word = new wordRecord (); word.word = key.toString (); word.count = sum;Gson json = new Gson (); System.out.println (json.toJson (word)); output.collect (key, new Text (json.toJson (word));} public static void main (String [] args) throws Exception {JobConf conf = new JobConf (WordCount.class); conf.setJobName ("wordcount") Conf.setOutputKeyClass (Text.class); conf.setOutputValueClass (IntWritable.class); conf.setMapperClass (Map.class); conf.setReducerClass (Reduce.class); conf.setInputFormat (TextInputFormat.class); conf.setOutputFormat (TextOutputFormat.class); FileInputFormat.setInputPaths (conf, new Path (args [0])); FileOutputFormat.setOutputPath (conf, new Path (args [1])); JobClient.runJob (conf);}}

This code is a modified version of the word count example provided by the Hadoop distribution.

This version uses the Google Gson library to write JSON information from the refinement phase of the process. For convenience, we used a new class (wordRecord), which is converted from Gson to a JSON record, which is the format required by Couchbase Server to process and parse content on a document-by-document basis.

Note that we do not define a Combiner class for Hadoop. This will prevent Hadoop from trying to refine this information, which will fail in the current code because our downsizing phase only accepts that word and single digit and outputs a JSON value. For the auxiliary reduce / combine phase, we need to parse the JSON input or define a new Combiner class to output the JSON version of the information. This simplifies the definition slightly.

To use this code in Hadoop, you first need to copy the Google Gson library to the Hadoop directory (/ usr/lib/hadoop/lib). Then restart Hadoop to ensure that Hadoop has correctly recognized the library.

Next, compile your code into a directory: $javac-classpath ${HADOOP_HOME} / hadoop-$ {HADOOP_VERSION}-core.jar:./google-gson-2.2.1/gson-2.2.1.jar-d wordcount_classes WordCount.java.

Now create a jar file for your library: $jar-cvf wordcount.jar-C wordcount_classes/.

After you complete this process, you can copy some text files to a directory, and then use this jar file to process the text files into many separate words, creating a JSON record to contain each word and count. For example, to process this data on some Project Gutenberg text: $hadoop jar wordcount.jar org.mcslp.WordCount / user/mc/gutenberg / user/mc/gutenberg-output.

This will generate a list of words in our directory that have been counted by the MapReduce function in Hadoop.

Export data from Hadoop to Couchbase Server

To retrieve data from Hadoop and import it into Couchbase Server, you need to export the data using Sqoop: $sqoop export--connect http://10.2.1.55:8091/pools-- table ignored-- export-dir gutenberg-output.

The-- table parameter is ignored in this example, but-- export-dir is the name of the directory where the information to be exported is located.

Writing MapReduce in Couchbase Server

In Hadoop, the MapReduce function is written in Java. In Couchbase Server, the MapReduce function is written in Javascript. As an interpreted language, this means that you do not need to compile the view, which allows you to edit and refine the MapReduce structure.

To create a view in Couchbase Server, open the administrative console (on http://localhost:8091) and click the View button. Views are collected in a design document. You can create multiple views in a single design document, or you can create multiple design documents. To improve the overall performance of the server, the system also supports an editable development view and a production view that cannot be edited. The production view cannot be edited because doing so invalidates the view index and results in the need to rebuild the index.

Click the Create Development View button and name your design document and view.

Within Couchbase Server, there are two identical functions: map and reduce. The map function is used to map input data (JSON documents) to a table. Then use the reduce function to summarize and refine the table. The reduce function is optional and not required for indexing, so for the purposes of this article, we will ignore the reduce function.

For the map function, the format of the function is shown in listing 6.

Listing 6. Format of the map function

Map (doc) {}

The parameter doc is each stored JSON document. The storage format of Couchbase Server is an JSON document, and the view is written in the Javascript language, so we can access a field called count in JSON using the following statement: doc.count.

To send information from the map function, you can call the emit () function. The emit () function takes two parameters, * keys (for selecting and querying information), and the second parameter is the corresponding value. Therefore, we can create a map function to use to output words and counts, as shown in the code in listing 7.

Listing 7. Map function that outputs words and counts

Function (doc) {if (doc.word) {emit (doc.word,doc.count);}}

This outputs a line of data for each output document, which contains the document ID (actually our word), the word used as the key, and the number of times that word appears in the source text. You can see the original JSON output in listing 8.

Listing 8. Raw JSON output

{"total_rows": "acceptance", "key": "acceptance", "value": 2}, {"id": "accompagner", "key": "accompagner", "value": 1}, {"id": "achieve", "key": "achieve", "value": 1}, {"id": "adulteration", "key": "adulteration", "value": 1}, {"id": "arsenic", "key": "arsenic" "value": 2}, {"id": "attainder", "key": "attainder", "value": 1}, {"id": "beerpull", "key": "beerpull", "value": 2}, {"id": "beware", "key": "beware", "value": 5}, {"id": breeze "," key ":" breeze "," value ": 2}, {" id ":" brighteyed "," key ":" brighteyed "," value ": 1}}

In the output, id is the document ID,key is the key you specified in the emit statement, and value is the value specified in the emit statement.

Get real-time data

Now that we have processed the information in Hadoop, import it into Couchbase Server, and then create a view for the data in Couchbase Server, and we can start querying the processed and stored information. Views can be accessed using a REST-style API, or when using a Couchbase Server SDK, access it through the corresponding view query function.

Queries can be executed through three main options:

Separate keys. For example, display information that matches a particular key, such as' unkind'.

List of keys. You can provide an array of key values, which returns all records whose key values match a supplied value. For example, ['unkind','kind'] will return a record that matches one of the words.

Key range. You can specify a start and end key.

For example, to find a specified number of words, you can query using the key parameter:

Http://192.168.0.71:8092/words/_design/dev_words/_view/byword?connection_timeout=60000&limit=10&skip=0&key=%22breeze%22

Couchbase Server will naturally output the result of a MapReduce sorted by a specified key in UTF-8 sorting. This means that you can get a range of values by specifying start and end values. For example, to get all the words between 'breeze'' and 'kind'', use:

Http://192.168.0.71:8092/words/_design/dev_words/_view/byword?connection_timeout=60000&limit=10&skip=0&startkey=%22breeze%22&endkey=%22kind%22

This query is simple but powerful, especially when you realize that you can use it in conjunction with a flexible view system to generate data in the format you want.

Concluding remarks

Hadoop itself provides a powerful processing platform, but it does not provide a way to actually extract useful information from processed data. By connecting Hadoop to another system, you can use that system to query and extract information. Because Hadoop uses MapReduce for related processing, you can use the knowledge of MapReduce to provide a query platform through the MapReduce system in Couchbase Server. Using this method, you can process the data in Hadoop, export the data from Hadoop to Couchbase Server in JSON document format, and then query the processed information in Couchbase Server using MapReduce.

At this point, the study on "how to use Hadoop and Couchbase together" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.