Shulou(Shulou.com)06/02 Report--

This article mainly talks about "what is the difference between Hadoop and MPPDB". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what is the difference between Hadoop and MPPDB"?

1. What is MPP?

MPP (Massively Parallel Processing), that is, massively parallel processing, in the database non-shared cluster, each node has an independent disk storage system and memory system. Business data is divided into each node according to the database model and application characteristics. Each data node is connected to each other through a private network or commercial general network to cooperate with each other to provide database services as a whole. Non-shared database cluster has the advantages of complete scalability, high availability, high performance, excellent performance-to-price ratio, resource sharing and so on.

To put it simply, MPP is to distribute tasks to multiple servers and nodes in parallel. After the calculation is completed on each node, the results of each part are summarized together to get the final result (similar to Hadoop).

2. MPP (massively parallel processing) architecture

(MPP architecture)

3. Characteristics of MPP architecture

● tasks are executed in parallel

● data distributed storage (localization)

● distributed computing

● private resources

● scale-out

● Shared Nothing architecture.

4. MPP server architecture

It is connected by multiple SMP servers through the Internet of certain nodes, works together and accomplishes the same task. From the user's point of view, it is a server system. Its basic feature is that several SMP servers (each SMP server is called node) are connected through the node Internet, and each node only accesses its own local resources (memory, storage, etc.), so it is a completely Share Nothing structure, so its scalability is the best, and its expansion is unlimited in theory.

5 、 MPPDB

MPPDB is a distributed parallel structured database cluster with Shared Nothing architecture, which has the characteristics of high performance, high availability and high scalability. It can provide a cost-effective general computing platform for very large-scale data management, and is widely used to support various data warehouse systems, BI systems and decision support systems.

6. MPPDB architecture

MPP adopts a completely parallel distributed flat architecture of MPP + Shared Nothing, in which each node (node) is independent, self-sufficient and peer-to-peer, and there is no single bottleneck in the whole system, so it has very strong scalability.

7. MPPDB characteristics

MPP has the following technical characteristics:

1) low hardware cost: PC Server, which fully uses x86 architecture, does not require expensive Unix servers and disk arrays

2) Cluster architecture and deployment: completely parallel MPP + Shared Nothing distributed architecture, using Non-Master deployment, node peer-to-peer flat structure

3) distributed compression storage of massive data: it can deal with structured data above PB level, and adopts hash distribution and random storage strategy to store data; at the same time, advanced compression algorithm is adopted to reduce the space needed for data storage, which can reduce the space used by 1x to 20 times, and improve IWeiO performance accordingly.

4) data loading efficiency: policy-based data loading mode, the overall loading speed of the cluster can reach 2TB/h.

5) High scalability and reliability: support the expansion and reduction of cluster nodes, and support full and incremental backup / restore

6) High availability and easy maintenance: data provides redundant protection through copies, automatic fault detection and management, and automatic synchronization of metadata and business data. Provide graphical tools to simplify database management by administrators

7) High concurrency: read and write are not mutually exclusive, query while loading data is supported, and the concurrency capacity of a single node is greater than 300 users.

8) column-column hybrid storage: a row-column hybrid storage scheme is provided, which improves the query response time of special query scenarios in column storage database.

9) Standardization: support SQL92 standard, support C API, ODBC, JDBC, ADO.NET and other interface specifications.

8. Common MPPDB

● GREENPLUM (EMC)

● Asterdata (Teradata)

● Nettezza (IBM)

● Vertica (HP)

● GBase 8a MPP cluster (NTU General)

9. Comparison and applicable scenarios between MPPDB, Hadoop and traditional database technology.

Both MPPDB and Hadoop distribute the operations to the nodes and merge the results (distributed computing), but they have their own advantages and disadvantages and scope of application because of the different theories and technical routes adopted. The comparison between the two technologies and traditional database technologies is as follows:

To sum up, the specific and applicable scenarios of Hadoop and MPP are:

● Hadoop has advantages in dealing with unstructured and semi-structured data, especially suitable for mass data batch processing and other application requirements.

● MPP is suitable to replace big data processing under the existing relational data institutions, and has high efficiency.

MPP is suitable for multi-dimensional data self-help analysis, data Mart, etc.; Hadoop is suitable for massive data storage query, batch data ETL, non-institutionalized data analysis (log analysis, text analysis) and so on.

From the above comparison, we can foresee the future trend of big data's storage and processing: MPPDB+Hadoop mashup, using MPP to deal with PB-level, high-quality structured data, while providing rich SQL and transaction support capabilities for applications; using Hadoop to achieve semi-structured, unstructured data processing. This can meet the needs of efficient processing of structured, semi-structured and unstructured data at the same time.

At this point, I believe you have a deeper understanding of "what is the difference between Hadoop and MPPDB". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.