Shulou(Shulou.com)06/02 Report--

As a leader in international data "stream" processing technology, Confluent provides real-time data processing solutions and has a large number of enterprise customers in the market to help enterprises easily access all kinds of data. As the first domestic "iPaaS+AI" one-stop big data converged service provider to support Kafka solutions, DataPipeline has rich practical experience and solution capabilities in retail, financial, Internet and manufacturing industries.

For Shanghai DataPipeline & Confluent Kafka Meetup, we invited Wang Guozhang, architect and technical leader of confluent streaming data processing system, Lv Peng, architect of DataPipeline, Pan Guoqing, head of real-time computing platform of Ctrip big data platform, Qin Jiangjie, senior technical expert of Alibaba real-time computing platform, and Ouyang Wulin, architect of NetEase Yunda data platform, to share the latest research results of Kafka and industry application cases.

Activity flow

Guest speaker

Wang Guozhang

Confluent stream data processing system architect and technical leader

Author of Apache Kafka PMC,Kafka Streams. He received his doctorate from the Department of computer Science at Cornell University, majoring in database management and distributed data systems. He is currently working in Confluent as the architect and technical leader of streaming data processing systems. Previously worked in the LinkedIn data Architecture Group, mainly responsible for the development and maintenance of real-time data processing platforms, including Apache Kafka and Apache Samza systems.

The theme of the speech: "Apache Kafka, from 0. 7 to 2. 0: the pit we stepped on in those years"

It has been seven years since it was donated to the Apache Foundation in 2011. What challenges has Kafka experienced from the earliest "distributed messaging system" to the current "streaming data platform" that integrates distribution, storage, and computing? What kind of evolution has it gone through?

1. From the development trend of hardware, show the evolution process of Kafka architecture

two。 Share the general principles of distributed system engineering practice from the experience of Kafka development and maintenance

3. Open source data system development experience, how to maintain and develop an open source community.

Pan Guoqing

The person in charge of Ctrip big data platform real-time computing platform

The person in charge of Ctrip big data platform real-time computing platform joined Ctrip in 2016, mainly engaged in the construction and evolution of Ctrip real-time computing platform, as well as the construction of Ctrip real-time feature platform, and has rich practical experience in the field of real-time computing.

Lecture topic: "Architecture and practice of Ctrip Real-time Computing platform"

1. The Evolution of Ctrip Real-time Computing platform

two。 Real-time architecture design and the pit that has been stepped on

3. The practice of Real-time Computing in Ctrip

4. Planning for the future.

LV Peng

DataPipeline architect

Lu Peng, currently an DataPipeline architect, is responsible for the architecture and optimization of data transfer and ETL. Has worked in Talend and sales Yi Big data Department, has more than 5 years of big data platform and data warehouse architecture experience, has a wealth of practical experience in the field of ETL.

Lecture topic: "Real-time data flow practice of DataPipeline on big data platform"

1. The main problems and challenges facing Enterprise data in the era of big data

2. The advantages and disadvantages of kafka connect data flow in the era of big data.

3. Practical cases of kafka connect on big data platform:

1) greenplum synchronization practice and optimization strategy of data warehouse under kafka connect

2) practice of hive synchronization under kafka connect

4. What work has DataPipeline done related to big data, the pitfalls encountered and the corresponding solutions

5. Summarize the future work of DataPipeline in the field of big data.

Zhang Yonghua

VIPSHOP, messaging platform architect

Graduated from Wuhan University and currently works as the architect of VIPSHOP's messaging platform. He has been engaged in the design and development of message system for more than 4 years, and has rich practical experience in message middleware products such as Kafka, RabbitMQ, RocketMQ and so on.

Topic: "thinking and implementation of establishing Enterprise message system platform based on Kafka1.0"

1. With the increase of business system access, the management of cluster and metadata resources (Topic/Group) is particularly important. There is an urgent need to establish a unified platform to manage system resources and authentication and authorization access services.

two。 Improve the high reliability in concurrent consumption scenarios, and how to elegantly solve the problem of consumption retry after abnormal consumption

3. How to realize the delivery function of delay message and how to improve the delay accuracy

4. Native kafka supports intergroup broadcasting, which causes difficulties for the dynamic growth of business group applications. How to achieve intra-group broadcasting to improve the status quo?

5. This paper introduces how to migrate services quickly and effectively in case of cluster failure.

Qin Jiangjie

Alibaba, Senior Technical expert of Real-time Computing platform

Qin Jiangjie, Alibaba, senior technical expert of real-time computing platform. Master graduated from Carnegie Mellon University, worked at LinkedIn and participated in Apache Kafka development, Apache Kafka PMC member.

Lecture topic: "detailed explanation of Flink's strict one-time semantic implementation based on Kafka transactions"

In a stream processing system, strict one-time semantics from end to end can only be implemented when the output Sink is supported. Support for things has been added in Kafka 0.11, so that other systems can take advantage of this feature to achieve strict once semantics. This sharing will detail how Flink uses this Kafka feature to implement end-to-end strict primary semantics.

Ouyang Wulin

NetEase big data platform architect

Once worked in China Mobile, 51 Credit Card, mainly engaged in cloud computing platform development related work. Currently working in NetEase, mainly engaged in big data platform development.

Topic of the speech: "the practice of Kafka on NetEase Cloud"

Challenges and opportunities facing the Cloud on 1.Kafka

two。 Implementation of NetEase Cloud Kafka Architecture

3. The solution to the key problems of NetEase Kafka

4. Planning for the future.

Time & place

Time: October 21, 2018, 13:30 signing in

Address: 11 / F, SOHO12 Building, 968 Jinzhong Road, Changning District, Shanghai

Registration for activities

Add DataPipeline Jun Wechat: datapipelien2018 to bring you into the live event group.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.