Shulou(Shulou.com)05/31 Report--

This article mainly introduces what the Hadoop file system has, the article is very detailed, has a certain reference value, interested friends must read it!

Hadoop has an abstract file system concept, and HDFS is just one of the implementations. The Java abstract class org.apache.hadoop.fs.FileSystem shows a file system of Hadoop and has several concrete implementations, as shown in Table 3-1.

File system

URI

Scheme

Java real

Now (all in

Org.apache.hadoop)

Description

Local

File

Fs.LocalFileSystem

For client-side checksums

The use of locally connected disks

The file system. Is it aimed at?

A book with a checksum

Use of local file system

RawLocalFileSystem .

See Chapter 4 for details.

HDFS

Hdfs

Hdfs.Distributed-

FileSystem

Distribution of Hadoop

File system. HDFS

Is designed to be used in combination

Map-Reduce achieves high performance

Efficient work

HFTP

Hftp

Hdfs.HftpFileSystem

One is mentioned on HTTP.

For read-only access to HDFS

Ask about the file system (although

Its name is HFTP, but

It has nothing to do with FTP. Pass

Often used in conjunction with distcp

(see Chapter 3), in transit

OK, different versions of HDFS

Replicate data between clusters

HSFTP

Hsftp

Hdfs.Hsftp-

FileSystem

Provide access to the

HDFS read-only access

File system (ibid., with

Independent of FTP)

HAR

Har

Fs.HarFileSystem

One is built on other texts.

The document is archived on the piece system.

The file system of the. Hadoop

Archiving is usually in HDFS

Used when archiving files of

To reduce the use of name node memory

KFS (Cloud-Store)

Kfs

Fs.kfs.Kosmos-

FleSystem

Cloudstore (formerly known as

Kosmos file system)

Is it similar to HDFS or

Google's GFS file.

System, compiled with C++

Write. Detailed

For more information, please see http://kosmosfs.

Sourceforge.net/

FTP

Ftp

Fs.ftp.FTP-

FileSystem

Supported by the FTP server

File system

S3 (Local)

S3n

Fs.s3native.Native-

S3FileSystem.

By Amazon S3

Documents held

The system. See

Http://wiki.apache.org

/ hadoop/AmazonS3

S3 (based on

Block)

S3

Fs.s3.S3FileSystem

By Amazon S3

Held file system

Save files in block format

(similar to HDFS)

To solve the 5 GB of S3

File size limit

Hadoop provides many file system interfaces, and it generally uses the URI scheme to select the appropriate file system instance interaction. For example, the file system shell we studied in the previous section can operate on all Hadoop file systems. List the files in the root directory of the local file system and enter the following command:

1.% hadoop fs-ls file:///

Although it is possible (and sometimes convenient) to run MapReduce programs that can access any file system, you still need to choose a distributed file system with optimal local data, such as HDFS or KFS, when dealing with large amounts of data.

These are all the contents of the article "what are the Hadoop file systems?" Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.