Shulou(Shulou.com)05/31 Report--

This article mainly introduces the reasons why relational database is not suitable for Hadoop, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

The problem should start from the computer hard disk, at this stage, the speed of hard disk addressing time is far less than the speed of transmission rate. Addressing is the process of moving the magnetic head to a specific hard disk location for read and write operations. It is the main cause of hard disk operation delay, and the transfer rate depends on the bandwidth of the hard disk. Figuratively speaking, the efficiency of workers loading goods is far less than that of transporting goods on high speed.

If the access mode of the data contains a large number of hard disk addressing, it will inevitably take longer to read a large number of data sets, and the streaming data read mode depends on the transfer rate. On the other hand, if the database system updates only a small number of records, then the traditional B-tree has an advantage (a data structure used in relational databases that is limited by the proportion of addressing). However, when there are a large number of data updates in the database system, the efficiency of B-tree obviously lags behind MapReduce, because it is necessary to use "sort / merge" to rebuild the database.

In many cases, MapReduce can be seen as a complement to a relational database management system.

Traditional relational database MapReduce data size GBPB data access interactive and batch update multiple read write write multiple read structure static mode dynamic mode integrity horizontal expansion nonlinear linear

Another difference between MapReduce and relational databases is the degree of structuralization of the datasets they operate on. Structured data (structured data) is materialized data with a given format, such as an XML document or a database table that meets a specific predefined format. This is what RDBMS includes. On the other hand, semi-structured data (semi-structured data) is relatively loose, although it may have a format, but it is often ignored, so it can only be used as a general guide to data structures. For example, a spreadsheet is structurally a grid of cells, but any form of data can be stored in each cell. Unstructured data (unstructured data) has no special internal structure, such as plain text or image data. MapReduce is very effective for unstructured or semi-structured data because it interprets the data only when it is processed. In other words, the keys and values entered by MapReduce are not inherent attributes of the data, but are chosen by the person who analyzes the data.

Relational databases tend to be normalized to keep the data intact and free of redundancy. The specification poses a problem for MapReduce because it makes record reads non-local, and one of the core assumptions of MapReduce is that it can perform high-speed streaming read and write operations. For example, the web server log is a typical non-standardized data record, such as recording the full name of the client host every time, which may cause the full name of the same client to appear multiple times, which is one of the reasons why MapReduce is very suitable for analyzing various log files.

MapReduce is a linear scalable programming model. Programmers need to write two functions, map and reduce functions, each of which defines the mapping of one key-value set to another key-value set. These functions do not have to pay attention to the size of the dataset and the clusters it uses, and can be applied intact to small or large datasets. If you enter twice as much data, the running time will also be twice as long, but if the cluster size is twice as large, the job will still run as fast as the original. SQL queries generally do not have this feature.

Thank you for reading this article carefully. I hope the article "why relational databases are not suitable for Hadoop" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.