Shulou(Shulou.com)11/24 Report--

The time has come for ChatGPT to trigger the tenth month of AI upsurge in China. "AI+X" is undoubtedly a popular proposition in technology-based enterprises this year. The so-called layman looks at the hustle and bustle and the expert looks at the doorway. Whether this gust of wind will completely change the technical structure of the original products, and how to use new technology and new trends to cut into the track and even to take the lead is the focus of the pan-technology field.

As the pioneer of intelligent sales SaaS in China, Discovery Technology has built a total knowledge graph of more than 180 million enterprises to solve a series of sales problems such as clue mining, business opportunity access, customer management and order analysis. In the face of such a huge amount of data, how does the big data architecture of Exploration evolve, and what new development and exploration are the technical teams in SaaS enterprises doing?

On September 16, the "Exploration Technology Lecture Hall" opened at the Exploration Technology headquarters. Chen Kailan, CTO of Exploration Technology, made an opening speech. Deng Chujian, a senior development expert of Exploration Technology, and Guan Tao, a senior development expert of Cloud Technology, brought their own technical views and case experiences, including the interpretation of the core technology of big data architecture and the thinking of future data algorithm engine.

Guan Tao: imagine the data architecture of "One Size Fits All"

There is no doubt that Guan Tao is an expert in big data's field.

As early as 2006, Guan Tao joined Microsoft Research Asia and participated in the construction of Microsoft's first generation distributed KV system SearchRepository and Bing search distributed storage background Kirin Store. After that, Guan Tao joined Microsoft's US Cloud Computing and Enterprise Division, presided over and participated in the development of Azure Datalake, Cosmos / Scope and other very large-scale distributed storage and computing platforms. Until he returned to China in 2016, he joined Ali, where he was responsible for big data platform Feitian MaxCompute and Dataworks, and later served as head of computing platform field of Alibaba and Ant Group Technical Committee and group leader of big data of Aliyun Architecture Group. Until 2021, he founded Cloud Technology as chief technology officer.

Guan Tao, CTO of Cloud Technology

Guan Tao said that even after years of development, domestic "original" data platform technology is still on the low side, but the data field continues to maintain rapid growth. In 2023, domestic data platform technology is still mostly assembled data architecture. Engine data semantics are not unified, heterogeneous storage and multiple sets of metadata have brought extremely high development costs, maintenance costs and a large number of computing and storage redundancy.

Most crucially, this type of assembled data architecture lacks the flexibility to meet the business.

Guan Tao believes that there should be an integrated data analysis engine to achieve a unified interface, unified processing logic, and provide high coverage and flexible adjustment of a variety of optimization indicators. In addition, AI expands the data types that can be processed, providing new processing power for structured data, and then there will be a new big data architecture that supports the new AI computing paradigm.

Then what is the reasonable solution?

The trail students asked Guan Tao for advice

Guan Tao's answer is that the open lake-warehouse integrated architecture is the best choice for Data+AI fusion scenarios, while the data platform with multiple independence, integrated analysis engine and BA+AI can liberate data productivity to the greatest extent.

Of course, Guan Tao believes that "incremental computing" may be the key to the implementation of one size fits all, which is also the goal of the next generation architecture of cloud technology Lakehouse.

Deng Chujian: there is no best structure, only the most suitable one

Hu Cang Yi Yi is also one of the technologies applied by the big data architecture team of Exploration Technology. Deng Chujian, a senior development expert from big data, brought his sharing.

As a senior technical expert on big data platform architecture, Deng Chujian has been focusing on the research and development of PB-level large-scale data real-time computing architecture, and has rich technical practice and achievements in enterprise big data and knowledge graph technology.

Deng Chujian, senior development expert of big data

Deng Chujian said that the data sources and characteristics of the exploration are unique in big data's field, with the characteristics of large amount of data, multi-source, multi-structured and semi-structured data, weak original relevance, large-scale random updates, and so on. How to extract the commercially valuable part from this massive low-value density data, present it to customers in a timely manner, and realize the vision of establishing business links for customers is the main challenge facing the big data architecture.

When the industry is setting up standardized data packets, the probe chooses to build its own big data architecture. The goal of the exploration is very clear, to do a "full knowledge graph", this is a way to "open up wasteland", and it is also what toB sales in China really need at present-- business opportunities are often in these data and information, and they are indispensable, and opportunities may be missed even if they are reached later.

Deng Chujian revealed that the big data architecture of Exploration Technology has evolved to the third generation of TDR (Tungee Data Refresh), which takes deposit and calculation separation, real-time update, and data governance as the core, which greatly meets the new requirements of data iteration in various industries and versions, and provides customers with immeasurable business value. In the next generation, the probe big data architecture will iterate in the direction of the exclusive data engine, namely Tungee Data Engine.

The trail students asked Deng Chujian for advice.

In addition, the exploration big data architecture has experienced the traditional database technology to the large database, and then to the data lake, and is currently applying technologies such as integrated lake and warehouse, integrated flow and approval, intelligent scheduling, and large language model engine integration.

The structure of each generation of exploring big data has solved the core problems of the period and is reasonable. Deng Chujian believes that excellent architecture is not achieved overnight, but continues to evolve. There is no best structure, only the most suitable structure, and the development process of architecture is also the development process of organizational structure. The technological framework should be in line with the organizational communication structure in order to give full play to the best results, adapt to the times, keep pace with the times, constantly absorb new technologies and experiences, and continuous iterative evolution is the foundation of development.

In this exploration technology lecture hall, the technical team of Exploration Technology and Cloud Technology had an in-depth exchange on the current situation and trend of big data architecture, looking forward to diversified technological thinking and viewpoints to promote further technological innovation. in the near future to help the industry technology to continue iterative upgrading, lead the industry development.

As the pioneer of domestic intelligent sales SaaS, Exploration Technology insists on innovation-oriented, technology-first, based on the real sales scene needs of various industries and enterprises, and is committed to using big data and AI technology to provide enterprises with full-process intelligent sales SaaS services, using science and technology to help enterprises reduce costs, increase efficiency and gain business opportunities. In the past seven years, Exploration Technology has constructed a full knowledge graph of more than 180 million enterprises, opened up a new era of intelligent sales SaaS, has won honors such as "specialization and innovation", "high-tech high-growth enterprise" and "global unicorn", and served more than 20000 customers, including Alibaba, Microsoft, Amazon, Dell and other industry giants.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.