Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

How to import and export data in hadoop distributed file system

2025-01-18 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to import and export data from hadoop distributed file system". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Request

Weblog_20140319.txt files are copied from the local file system to a specified folder in HDFS

Operation step 1. Create a folder in HDFS to store the target files: hadoop fs-mkdir / data/weblogs.2. Copy the target file from the local system to the folder you just created: hadoop fs-copyFromLocal weblog_20140319.txt / data/weblogs3. List the information of the weblog_20140319.txt file on HDFS: hadoop fs-ls / data/weblogs/weblog_20140319.txt4. Copy the weblog_20140319.txt on HDFS to the current folder on the local system: hadoop fs-copyToLocal / data/weblogs/weblog_20140319.txt. / weblog_20140319.txt

Some of the resulting data processed on Hadoop may be directly used by external systems, or it may need further processing by other systems, or the MapReduce framework does not meet the application scenario at all, and any similar situation needs to export data from HDFS. The easiest way to download data is Hadoop Shell.

When copying files from HDFS to the local system, note: 1. Have enough free space 2. Network connection speed (file sizes in HDFS are very common from several TB to dozens of TB. In 1Gbit network environment, it takes 23 hours to export 10TB data from HDFS to the local file system, provided there is enough free space.)

That's all for "how to import and export data from hadoop distributed file system". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report