Shulou(Shulou.com)06/03 Report--

On May 16, 2018, the aggregator (warehouse version) was officially released with a number of cool techs items.

In the application verification before the release, the warehouse version has already won the praise of users with its strength. When evaluating the warehouse version, Bank of Beijing users said: in the practice of data analysis, the problem of long system response time caused by high concurrent access and large amount of data computing has not been well solved. The emergence of the aggregator (warehouse version) has completely solved this problem! Using the aggregator to front the high-frequency hot data and building the data computing middle layer can be said to be the best solution, which is better than the million-dollar database products in many scenarios!

The actual test also proves that the performance of the aggregator (warehouse version) is really excellent! Take the performance index as an example, the test target is high-frequency hot spot data, 30 million rows a day. The aggregator and GreenPlum execute the same conditional query, the aggregator only completes in 2 seconds, and GreenPlum executes in 5 seconds. The GP test environment consists of five node clusters, each of which is a physical machine with 2 to 6 cores of CPU,96G memory, while the server where the aggregator is located is only a virtual machine with 1 to 2 cores of CPU,16G memory. To be able to greatly surpass GP shows the power of cool techs!

What kind of black skills win the praise of users with such excellent performance? Next, let's say one by one:

One of cool techs: group table

Group table, also known as combined table, is the basic way of data storage of aggregator (warehouse version). The group table supports partial modification of data, supports update and recovery, and can safely and easily synchronize hot data from full data sources. The group table supports the index, and the data itself is stored in an orderly manner. The commonly used conditional filtering calculation does not rely on the index to ensure high performance. The index is automatically updated after the data is modified.

In the above-mentioned tests, the method of serial hard disk storage is used. The group table can also use row storage and full memory to store data, and support the operation of in-memory database.

The second part of cool techs: can be stored in parallel

The column storage mechanism used in the group table is different from the conventional storage mechanism. Conventional column storage (such as parquet format) can only be divided into blocks and then stored within the block, which is limited when doing parallel computing. The parallel compressed storage mechanism of the group table adopts the technology of multiplication and segmentation, allows parallel computing of any segment, and can make use of the computing power of multiple CPU cores to maximize the IO of the hard disk.

Cool techs III: cluster group table

In addition to the above characteristics, the group table also supports the distribution of data on multiple machines to form a cluster group table. Cluster group tables can use multi-machine parallelism to scale computing power and storage capacity horizontally. In usage, the cluster group table and the ordinary group table are basically the same. In other words, the cluster group table is a transparent cluster, and users do not need to care about the details of the multi-node group table, but can be used as a group table.

IV of cool techs: unification of the main schedule

Some datasheets have the same dimensional (one-to-one) relationship or master-child relationship. For example: customer form, VIP customer table; user basic information, family information, education experience, work experience; order, order details.

Master schedule unification refers to putting the same dimension table or master child table into a group table and storing only one primary key. The JOIN calculation of these tables can also be omitted, the storage space can be reduced and the performance can be effectively improved.

The fifth part of cool techs: serial key technology

JOIN calculations of foreign key relationships are also common, such as sales record tables associated with item tables through "item numbers". The serial number key technology is to change the commodity number in the sales record into an integer, which is the order number of the commodity corresponding to the commodity number in the commodity table.

The serial key technology enables the foreign key connection (JOIN) calculation to use the serial number directly, without the need to calculate and compare the hash value, which reduces the computing time and improves the performance. At the same time, using the foreign key connection technology of serial key technology, it is easy to execute multiple connections (JOIN) in parallel.

Cool techs VI: JDBC Intelligent Gateway

The aggregator provides JDBC driver and simple SQL interface, and has programmable gateway mechanism. By writing the new generation programming language SPL code built into the aggregator, the calculation rules of high-frequency hot data can be realized freely.

The calculation rules that can be considered include: analyze the date parameters in the SQL filter conditions passed in the front end, and directly access the data if it hits the date that has been cached by the aggregator (warehouse version). If it is not hit, the SQL will be forwarded to the background traditional database for execution. The aggregator can also record access and analyze the temporal and spatial distribution of hot spot data.

It can be predicted that this product will bring new value and opportunities to your software project. Now, the Runqian official website provides a full-featured trial version, download it now!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.