Shulou(Shulou.com)05/31 Report--

This article mainly shows you "how to use the command line interface in Hadoop to run Mini cluster", the content is easy to understand, well-organized, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to use command line interface in Hadoop to run Mini cluster" this article.

Objective:

Using a command line interface to run a Mini cluster, users can use a simple command to start or stop a single-node Hadoop cluster without setting any environment variables and managing configuration files. This CLI Mini cluster will start Yarn/MapReduce and HDFS.

This is a case where you want to quickly experience a real Hadoop cluster or test a function without a java program.

Hadoop Tarbar

You should use the release version of Hadoop tarball as much as possible, of course, you can also create a tarball from the source code:

$mvn clean install-DskipTests$mvn package-pdist-Dtar-DskipTests-Dmaven.javadoc.skip

Note: protoc 2.5.0 is required to be installed before installing tarball

Tarball can be installed in the hadoop-dist/target/ directory.

Run the Mini cluster:

To extract tarball from the root directory, you can start the CLI Mini cluster using the following command:

$bin/hadoop jar. / share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar minicluster-rmport RM_PORT-jhsport JHS_PORT

In the above command line example, RM_PORT and JHS_PORT should be replaced by the port number of the user's choice. If not specified, the port will be randomly assigned to the user.

Through other property configurations, there are some control command parameters that can be used by the user when the control service is started. Valid command line arguments:

$- D Options to pass into configuration object$-datanodes how many datanode nodes are started (default is 1) $- format format DFS (default is false) $- help print help information. $- jhsport (default 0--we choose) $- namenode NameNode URL (default is DFS cluster or a temporary directory) $- nnport NameNode port (default 0--we choose) $- nodemanagers how many NameNodes are started (default 1) $- nodfs does not allow you to start a mini DFS cluster $- nomr Don't start a mini MR cluster$-rmport ResourceManager port (default 0--we choose) $- writeConfig to save the configuration to the specified file (path). $- writeDetails writes basic information to the specified directory (path) in json format.

You can use the-help parameter command to display the parameter usage of a valid list.

These are all the contents of the article "how to run a Mini cluster using a command line interface in Hadoop". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.