How to use JAVA API to operate HDFS
This article mainly introduces how to use JAVA API to operate HDFS, the article is very detailed, has a certain reference value, interested friends must read it!
Step 1: hadoop environment configuration in windows
The windows operating system needs to configure the hadoop environment.
Mac is essentially a unix system and does not require configuration
= = reference document "Windows&Mac native development environment configuration" =
Link: https://pan.baidu.com/s/1tFJSlRxn18YELUUAUkXXQA
Extraction code: g9ka
Step 2: create the java maven project and import the package:
3.1.4
Org.apache.hadoop
Hadoop-client
${hadoop.version}
Org.apache.hadoop
Hadoop-common
${hadoop.version}
Org.apache.hadoop
Hadoop-hdfs
${hadoop.version}
Org.apache.hadoop
Hadoop-mapreduce-client-core
${hadoop.version}
Junit
Junit
4.11
Test
Org.testng
Testng
RELEASE
Log4j
Log4j
1.2.17
Org.apache.maven.plugins
Maven-compiler-plugin
3.0
1.8
1.8
UTF-8
Org.apache.maven.plugins
Maven-shade-plugin
2.4.3
Package
Shade
True
Step 3: develop the javaAPI operation of hdfs
/ / simplified version
@ Test
Public void mkDirOnHDFS () throws IOException {
/ / configuration item
Configuration configuration = new Configuration ()
/ / set the hdfs cluster to connect to
Configuration.set ("fs.defaultFS", "hdfs://node01:8020")
/ / get the file system
FileSystem fileSystem = FileSystem.get (configuration)
/ / call the method to create a directory. If the directory already exists, the creation fails and false is returned.
Boolean mkdirs = fileSystem.mkdirs (new Path ("/ kaikeba/dir1"))
/ / release resources
FileSystem.close ()
}
/ / specify the user to which the directory belongs
@ Test
Public void mkDirOnHDFS2 () throws IOException, URISyntaxException, InterruptedException {
/ / configuration item
Configuration configuration = new Configuration ()
/ / get the file system
FileSystem fileSystem = FileSystem.get (new URI ("hdfs://node01:8020"), configuration, "test")
/ / call the method to create a directory
Boolean mkdirs = fileSystem.mkdirs (new Path ("/ kaikeba/dir2"))
/ / release resources
FileSystem.close ()
}
/ / specify directory permissions when creating a directory
@ Test
Public void mkDirOnHDFS3 () throws IOException {
Configuration configuration = new Configuration ()
Configuration.set ("fs.defaultFS", "hdfs://node01:8020")
FileSystem fileSystem = FileSystem.get (configuration)
FsPermission fsPermission = new FsPermission (FsAction.ALL, FsAction.READ, FsAction.READ)
Boolean mkdirs = fileSystem.mkdirs (new Path ("hdfs://node01:8020/kaikeba/dir3"), fsPermission)
If (mkdirs) {
System.out.println ("Directory created successfully")
}
FileSystem.close ()
}
Note: we must configure the corresponding hadoop cluster and related host in accordance with the requirements of the previous environment, and only if the hadoop starts successfully can the above program run normally.
The above is all the contents of the article "how to use JAVA API to operate HDFS". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!
