Shulou(Shulou.com)11/24 Report--

Recently, the "Chinese programmer Festival" sponsored by the China Software Industry Association was held in Beijing, Shenzhen and Ningbo at the same time, in which the Database Technology Summit Forum was held in Beijing. Even technology appeared in this forum and shared a keynote speech entitled "Big Model, Real-time demand driving Hucang platform to Open".

With the complex international situation and accelerated competition in science and technology, database, as one of the root technologies of scientific and technological development, plays an increasingly important role in information management, decision support, data mining and so on. In order to further promote the development and application of domestic database technology, this forum brought together industry experts and scholars to exchange and discuss database technology topics such as multimodal, distributed, domestic migration, lake and warehouse integration, open source community, etc., to enhance programmers'in-depth understanding of cutting-edge new technologies and applications, and expand the depth of knowledge.

At present, with the continuous expansion of the requirements of AI and real-time scenarios, AI + real-time has become an inevitable technical focus of the enterprise data platform. Yang Zhe, director of even-numbered marketing, shared how to make enterprises embrace the dual capabilities of AI + real-time through an open data platform.

The way to break the situation given by even numbers is to use a piece of data in an open format in data storage (such as Parquet, ORC, Hudi, etc.). Each computing engine uses an open data format, and the data is written to the data platform in an open file format, which can then be directly read and used by multiple engines.

With the openness of storage, in terms of computing engines, we can optimize and reduce the number of computing engines as much as possible, and choose each advantage computing engine for structured data, unstructured data and streaming data.

 adopts the common Flink for the calculation of stream data.

 can use Spark for unstructured data and machine learning.

For structured data,  needs to be compatible with open data formats, real-time query, offline analysis, high concurrency and high availability analysis engines, such as even OushuDB.

At this point, the architecture of open format, one data and multiple engines has initially taken shape, and such a "one-number-multi-engine" architecture has formed a solution that can break the current enterprise data dilemma-real-time lake warehouse (Realtime Lakehouse). " "one number and multiple engines" is the best practice of continuous iteration in the landing of lake and warehouse integrated projects in many industries. Enterprises must be based on the principle of "simplifying complexity into simplicity" and "enhancing their strengths and avoiding weaknesses" when selecting multiple engines.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.