Shulou(Shulou.com)06/01 Report--

This article shows you the benefits of using Apache Doris, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Doris is the incubation project of Apache. As a relational analysis database of MPP architecture, it has excellent performance. In addition, Doris is highly compatible with MySQL protocol, supports on-line table structure changes and does not rely on any external storage; strong scalability, a single cluster can scale up to more than 200sets; high availability of multi-copy data and metadata, while supporting Kafka streaming import. It is hoped to attract more external contributors to make effective improvements in the future.

As an incubation project of Apache, Doris is used by many large enterprises, such as Sohu, a little information, Meituan, Xiaomi, JD.com and so on. Doris products were first developed in Baidu about a decade ago, and the current internal partial data warehouse business basically runs on it. As a relational analysis database based on MPP architecture, it was designed to focus on convenience and problem solving, supporting PB level. Because the traditional data warehouse is mainly in the direction of business intelligence, it does not do a lot of optimization for high concurrency and high availability, but focuses on functions such as multidimensional analysis and report query, while Doris can ensure 7 × 24-hour service. Through rolling upgrades without interrupting online services, Doris opened source on Github in 2017 and entered the Apache incubator in 2018. When it comes to the characteristics of Doris, excellent performance is an aspect that can not be ignored. To understand the excellent performance of a database, we need to look at its scenario. It is unscientific to talk about database performance in general without the scenario. Each single backend of Doris can achieve the query performance of 8000 + times per second in the case of high concurrency, and the cluster of 100 nodes can basically achieve the performance of tens of thousands to hundreds of thousands of QPS. This performance can basically meet the query requirements similar to Baidu statistics and support hundreds of thousands of websites and APP throughout the network. In addition, Doris is highly compatible with MySQL protocol and supports on-line table structure changes; only two processes, FE and BE, are set up to facilitate deployment, in which FE is responsible for metadata storage, query parsing and planning, and BE is responsible for data storage and query operator execution.

Based on this, we all know that systems like Kylin are extremely inconvenient to change online, just adding deleted columns may require remaking tables, affecting online services, and relying on many Hadoop components, making deployment and use very complex. However, Doris is highly integrated and does not rely too much on external storage. It can be installed quickly and the master node can be highly available. Even switching can be imperceptible and easy to operate. In addition, Doris has strong scalability and elegant architecture, and its single cluster can be expanded horizontally to more than 200. it supports Kafka streaming into the cluster, and shows high availability, multi-copy and other advantages through ODBC, JDBC link front-end visualization tools or development applications. Doris, as open source software for large enterprises, is accumulated in practice and has a good effect on solving practical problems.

In JD.com, they mainly use Doris to solve the report query of JD.com advertising platform, which needs to meet the data indicators such as daily tens of millions of level query, daily 10 billion increment, millisecond query delay and daily 10 billion report query / multidimensional analysis / log analysis. Doris performed prominently during the 618 promotion period. In addition, in the process of Meituan data warehouse migration, the data modeling was originally based on Apache Kylin, because the pre-construction of Kylin Cube will cause more data duplication, and it also needs to recalculate a large number of historical data every day to maintain the data timeliness; but now it is mainly based on Doris data modeling, daily synchronization of dimension table and fact table, ad-hoc query query.

Doris's next major R & D plan includes the separation of storage and computing. We find that there is a lot of historical data on the storage node, and we hope that it can be stored in the cloud object storage in the future, so that Doris can be improved to cloud native database. After all, when flexible computing resources are deployed on the cloud, once computing and storage are separated, the cost will be reduced accordingly; new storage formats will be introduced to improve storage efficiency, including indexing, new real-time update storage engine, etc., in order to improve efficiency; in addition, efforts will be made on query optimization, such as developing a new query optimization framework to support CBO.

What are the benefits of using Apache Doris? have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.