Shulou(Shulou.com)06/02 Report--

This article will explain in detail the three methods to achieve the joint operation of R and Hadoop. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

In order to meet the needs of using R language to deal with pb-scale data, we need to use it in conjunction with Hadoop. This paper describes the different technologies to realize the joint operation of the two. Method 1: use Streaming APIs

Hadoop supports some Streaming API to pass in functions from the R language and run them in MapReduce mode. These Streaming API can pass into the Hadoop any R script that can access and operate the standard Imax O interface in map-reduce mode. So, you don't need to open any extra clients or anything like that. The following is an example:

Method 2: use the Rhipe package

The Rhipe package allows users to use MapReduce in R. Before using this method, the corresponding preparatory work should be done. R needs to be installed on every data node in the Hadoop cluster, and Protocol Buffers is also installed on each node. (for more information, please refer to http://wiki.apache.org/hadoop/ProtocolBuffers) Magi Rhipe, which also needs to be used on each node.

Here is an example of using Rhipe to apply the MapReduce framework in R:

Method 3: use RHadoop

RHadoop is an open source library under Recolution Analytics. Similar to Rhipe, its function is to execute R function in MapReduce mode. Subsequent lists are some of the packages in the library. The plyrmr package can perform some common data collation operations on the big data set in Hadoop. The rmr package provides functions that allow R and Hadoop to work together. The rdfs package provides functions to connect R to the distributed file system (HDFS). The functions in the rhbase package can connect R and HBase.

In the following example, we will demonstrate how to use some functions in the rmr package to make R work in conjunction with Hadoop.

Method summary

In general, the above three methods can easily achieve the joint operation of R and Hadoop, so that R has the ability to deal with big data on the distributed file system (HDFS). But at the same time, these three methods also have their own advantages and disadvantages.

Key conclusions:

1. Streaming APIs is the easiest to use, and its installation and setup are very convenient. Both Rhipe and RHadoop need to make some settings for R, and they also need some package support on the Hadoop cluster. However, in terms of function execution, Streaming APIs needs to map and reduce functions in turn, while Rhipe and RHadoop allow developers to define and call MapReduce functions in R functions.

2. Unlike Rhipe and RHadoop, you don't need a client to use Streamings APIs.

3. In addition, we can also use the Apache Mahout,Apache Hive,Segue framework and other commercial R from Revolution Analytics to achieve large-scale machine learning.

On the realization of R and Hadoop joint operation of the three methods are shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.