Shulou(Shulou.com)06/03 Report--

The previous article explained in detail how to remotely connect Hadoop clusters under Eclipse for Hadoop program development. Here is an example of a Hadoop In Action book, possibly due to a problem with Hadoop version updates, which caused some of the sample programs in the tree to fail to execute properly.

The whole code is to merge several small files under the local directory into a larger file and write it to HDFS. Without further ado, the code is as follows:

Supplementary note: later found that the source code on the book is no problem, but the source code on the book to be typed into jar package, placed on the cluster machine to run, if the Eclipse debugging run below, there will be problems. The reasons for the problem are as follows

//Read the local file system. If you want to run it correctly, you must type it into a jar package. Run FileSystem hdfs = FileSystem.get(conf);FileSystem local = FileSystem.getLocal(conf);//HDFS can be read remotely through URI. Therefore, you should use this method to debug Eclipse. This form is also possible. FileSystem hdfs = FileSystem.get(URI.create(serverPath), conf);FileSystem local = FileSystem.getLocal(conf);

package com.hadoop.examples;import java.io.IOException;import java.net.URI;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FSDataInputStream;import org.apache.hadoop.fs.FSDataOutputStream;import org.apache.hadoop.fs.FileStatus;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;/** * @Package * @ClassName: PutMerge * @Description: Read files in local directory, write to HDFS, during writing, * Combine these three files into one file * @author lxy * @date March 25, 2015 at 9:59:38 * @version V1.0 */public class PutMerge { public static void main(String[] args) throws IOException { //Enter the directory, there are three txt files under the directory, and the file content will be given at the end of the article. String localPathStr = "E:\\test"; //Output directory, HDFS path, the file content after merging will be given at the end of the article String serverPath = "hdfs://192.168.3.57:8020/user/lxy/mergeresult/merge.txt"; //Enter directory, which is a local directory Path inputDir = new Path(localPathStr); //output directory, is an HDFS path Path hdfsFile = new Path(serverPath); Configuration conf = new Configuration(); /** * The original code for Hadoop in Action is as follows * FileSystem hdfs = FileSystem.get(conf); * But then, when debugging with Eclipse, executing the following statement will report an exception because it reads locally. * file system * FSDataOutputStream out = hdfs.create(hdfsFile); */ //According to the serverPath above, you get an org.apache.hadoop.hdfs.DistributedFileSystem object FileSystem hdfs = FileSystem.get(URI.create(serverPath), conf); FileSystem local = FileSystem.getLocal(conf); try { //Get a list of files and folders in the input directory FileStatus[] inputFiles = local.listStatus(inputDir); //create a file on hdfs FSDataOutputStream out = hdfs.create(hdfsFile); for (int i = 0; i

< inputFiles.length; i++) { System.out.println(inputFiles[i].getPath().getName()); //打开本地输入流 FSDataInputStream in = local.open(inputFiles[i].getPath()); byte buffer[] = new byte[256]; int bytesRead = 0; while ((bytesRead = in.read(buffer)) >

0) { //write data to files on hdfs out.write(buffer, 0, bytesRead); } //Release resources in.close(); } //Release resources out.close(); } catch (IOException e) { e.printStackTrace(); } }}

There are three txt files in my test directory

1.txt

1 hello Hadoop2 hello Hadoop3 hello Hadoop4 hello Hadoop5 hello Hadoop6 hello Hadoop7 hello Hadoop

2.txt

8 hello Hadoop9 hello Hadoop10 hello Hadoop11 hello Hadoop12 hello Hadoop13 hello Hadoop14 hello Hadoop

3.txt

15 hello Hadoop16 hello Hadoop17 hello Hadoop18 hello Hadoop19 hello Hadoop20 hello Hadoop21 hello Hadoop

The consolidated file looks like this:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.