Shulou(Shulou.com)06/02 Report--

Background of Hive production:

Before we explain Hive, we need to familiarize ourselves with some concepts of Hadoop.

Hadoop can be divided into the following parts of the HDFS hadoop file system, used for data storage MapReduce for data processing Yarn for resource management that the MapReduce program in Hadoop generally handles input is some standardized logs, assuming that we have the following log files. Name subject score Zhang San Chinese 90 Li Si Chinese 80 Wang Wu Chinese 88 Zhang San Mathematics 99 Li Si Mathematics 98 Wang Wu Mathematics 90 We need to deal with these data, such as those with the highest scores and statistical average scores. Then we need to write MapReduce programs like an eight-part essay: 1, write Mapper 2, write Reducer 3, write main 4, define job 5 in main, set job input, output and parameters 6, execute job so that we need to be very familiar with MapReduce programming, and this way is more time-consuming and laborious. At the same time, in ordinary companies, we generally leave this kind of data with a fixed format to a special DB to deal with, but DB does not understand the programming of MapReduce, so it is not realistic for them to write MapReduce programs to deal with data, is there a tool or a tool that allows them to use a way similar to sql to clean data. Of course the answer is yes, and that is our Hive. What is Hive? Hive is a basic data warehouse tool for dealing with structured data on the basis of hadoop. It says here that it is a tool.

Its main function is to facilitate us to deal with data, but the storage of data is still on the HDFS.

Hive is based on Hadoop and can provide query statements similar to SQL language to simplify big data's processing and clarity.

It is convenient for DB to process data.

Hive was first developed by Facebook, then developed by the Apache Software Foundation, and will be a top-level project under Apache.

Hive is an open source project that is used in many different companies.

The characteristics of Hive 1, it is not like a relational database can only deal with a small amount of data, hive because of the architecture on top of Hadoop, itself gives it the ability to deal with big data. 2. It provides a query language similar to SQL, called HQL or HiveQL. 3. Because it is a secondary extension on MapReduce, hive has a good extensibility. If there is a processing that hive can not provide, we can package it into a function of hive by writing mapreduce programs. The basic principle of hive HIve is just a tool, he converts HQL into MapReduce programs to run on Yarn, to deal with the data stored on HDFS, so that we can deal with simple data processing, just write HQL, no need to write MapReduce procedures. Hive architecture

In the process of using Hive, users provide HQL statements to hive by means of CLI, JDBC/ODBC, WebUI and so on. Hive converts optimized HQL statements into MapReduce programs and runs them on yarn through compilation, optimization and execution.

For the line with the highest query score at the beginning, we only need to write the following HQL:

Select * from table order by sorce desc limit 0Pol 1

At this point, the whole introduction of hive has been explained by everyone. In the next article, we will explain the installation of hive.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.