Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to import and export data from hadoop". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

A classic data architecture

Data is collected from distributed systems, imported into HDFS (Hadoop distributed File system), processed, filtered, transformed, aggregated, and exported to one or more external systems through MapReduce or other MapReduce-based encapsulated languages (Hive,Pig,Cascading).

Chestnut:

Do a basic data analysis of the click-through rate of a website, collect page access logs from multiple servers, then push them to HDFS, start a MapReduce job, input these data as MapReduce, and then the data will be parsed, summarized, and associated with Ip addresses, resulting in URL, page visits and geographic location data for each cookie. The relevant results generated can be imported into a relational database. Ad hoc query (users according to their own needs, flexible selection of query conditions, the system can generate corresponding statistical reports according to the choice of user query) can be built on these data at this time. Analysts can quickly generate a variety of report data, such as the current number of unique users, the pages most visited by users, split users by region, and other data summaries.

Introduction

HDFS provides a number of Shell commands based on HDFS FileSystem API to access the file system. The shell script that comes with Hadoop performs all operations from the command line, and the delivery script is called hadoop. It is usually installed in the $HADOOP_BIN directory, where $HADOOP_BIN is the full installation directory of the Hadoopbin file, and it is necessary to configure $HADDOP_BIN in the ¥PATH environment variable so that all commands can be executed in the form of hadoop fs-command.

View command

If you need to get all the commands for the file system, you can run the hadoop command to pass the option fs without arguments. The names of the resulting commands are very similar to those of the Unix shell command, and you can use the help option to get a specific description of a command. For example: hadoop fs-help ls

Mastering orders is the key to efficiency

Here, we use Hadoop shell commands to import data into HDFS and export data from HDFS. These commands are more used to load data, download processed data, manage file systems, and preview related data. Mastering these commands is a prerequisite for efficient use of HDFS.

That's all for "how to import and export data from hadoop". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.