Shulou(Shulou.com)06/03 Report--

At present, Chinese enterprises are still in the initial stage of communication, exchange and utilization, but the enterprise application data integration market is huge. According to Forrester data, the global data application integration market in 2017 is $32 billion pure software, and if labor is included, it will reach $394 billion.

In the field of data application integration, there are not only traditional IT leaders such as Oracle, SAP, Microsoft and Informatica, but also many innovative enterprises. DataPipeline is a company that helps enterprises connect internal and external data islands and realize data exchange and fusion by providing batch-flow integrated data fusion, data cleaning, data synchronization and other services. Recently, DataPipeline CTO Chen Su talked about the development of DataPipeline, data application integration industry, company management and personal experience. The following is an interview record:

I. Break data islands and redefine data application integration

Please describe the main strategy and market layout of DataPipeline in detail.

Chen Su: DataPipeline's target customers are concentrated in finance, retail, manufacturing, real estate and Internet industries. Its service customers mainly have the following characteristics: large and medium-sized enterprises, high data value density and emphasis on timeliness of data.

The differentiation strategy includes:

Support large and medium-sized enterprises with big data application requirements; applications can be deployed on the cloud; real-time requirements are high, which is different from the previous batch; it can support changes in business, data and architecture; in terms of user experience, more emphasis is placed on automation and intelligence. DataPipeline is on the track of data application integration. How do you understand this industry?

Chen Su: At present, most Chinese enterprises are still in the initial stage in terms of ×× communication, exchange and utilization. The key reason is that they have not done a good job in basic work such as data integration, data cleaning and data synchronization.

I think there will be three changes in the future of data application integration:

First, it will be more complex than it was in the past. There may have been structured data in some databases, but now there are structured, semi-structured, unstructured data, on-cloud, off-cloud, hybrid cloud approaches, object storage in databases and data warehouses, etc.

Second, it is more time-sensitive. In the past, the data flow was relatively slow, and the overall operation speed of the business would also be slower. However, with the improvement of the real-time decision-making requirements of enterprises, we need to analyze the data in time, so the timeliness requirements will also be improved.

Third, high scalability and flexibility. With the rapid development of society, the demand of business departments for data is also changing at all times. This means that users 'IT architecture, software, and overall growth strategy need to adapt to this change.

Increased complexity, faster timeliness, and increased architectural change are the top three challenges for data usage, but they also create new opportunities.

What do you think is the market size and potential for data application integration?

Chen Su: Forrester data shows that the pure software scale of the global data application integration market in 2017 is US $32 billion, and if labor is included, it will reach US $394 billion.

Gartner data also shows that iPaaS, the application data integration segment, topped $1 billion for the first time in 2017, growing 72 percent.

What are the competitors of the current data application integration track? Where does the competition focus?

Chen Su: There are many enterprises participating in the market competition. In the field of data application integration, there are not only traditional IT tycoons such as Oracle, SAP, Microsoft and Informatica, but also some innovative enterprises. However, compared with the new generation of cloud, big data real-time data application integration, there are fewer new players. In China, data application integration enterprises are actually relatively missing.

At present, in the track of data application integration, some enterprises focus on data integration and some on application integration. There are also some enterprises such as Ali, which have a wider coverage than innovative enterprises, with more or less differentiation, and fewer enterprises in basic technology innovation.

On the other hand, there are some tools on the market that have been in use for up to 10 years, which are based on traditional software architectures, while new tools are cloud-based, mainly deployed on the cloud, and tools that support large amounts of data and real-time applications with distributed architectures are still relatively few.

What do you think are the main advantages of DataPipeline? What strategies does DataPipeline use? What kind of effect has been achieved so far?

Chen Su: Technically, DataPipeline focuses on streaming data processing, high-performance synchronization, and quickly solves data fusion problems.

In terms of products, DataPipeline is a company that provides batch-flow integrated data fusion services for enterprises. By providing data batch flow integration processing, task scheduling, data quality management, visual O & M and monitoring, API data access, metadata management and other functions, it helps customers realize complex heterogeneous data source and destination data fusion and other comprehensive services more agile and efficiently, providing powerful technology drivers for customers 'flexible data consumption needs.

DataPipeline has successfully served many industry-leading enterprise customers such as Starbucks, Xicha, Dingdang Kuaiyao, etc., and has established strategic cooperation relationships with dozens of upstream and downstream partners in the industry.

TGO Kunpeng Club: What is the next step in DataPipeline planning?

Chen Su: We will continue to adhere to the established strategy and serve customers with technology-driven, while continuing to invest resources in customer success to bring greater value to customers.

2. Sweat more in peacetime and bleed less on the battlefield

What kind of team culture, or what kind of team atmosphere and rules do you have in your team right now?

Chen Su: DataPipeline has clear enterprise core values, which can be summarized as customer success and personal growth. There are six specific items--never forget the original heart, dig deep into the source, customer first, words must be fruitful, technology-driven, selfless sharing, and the culture of the technical team is in line with these six core values.

As a ToB company, we emphasize customer first. Operations, testing, and development should place positioning and solving customer problems at the highest priority of their work. In order to reduce the disruption of daily R & D work, we have established a rotation system to ensure that there is a dedicated team to respond to customer support needs every week. Customer environment is complex, sometimes students on duty will encounter difficult problems to solve. At any time, as long as the students on duty throw the problem into the On Call group, the technical backbone of the company will immediately conduct problem consultation, give timely response strategies and suggestions, and even immediately remotely access the customer site to assist in locating the problem. Of course, staying up late and working overtime is always not good, so we have a perfect system of rest to ensure the physical and mental health of employees.

At least once a week, we have an internal team sharing, which can range from technology trends, design insights and technical points at work to fitness tips and travel experiences. Some of the better technical topics are distilled and shared by team members at open source community meetups.

We emphasize technology-driven. Only problems that can be solved through programs should not be solved by "human flesh." Therefore, a large part of the work of colleagues in the testing and operation and maintenance team is also writing code to improve efficiency through automated testing and automated operation and maintenance. If there are problems in development and testing, I encourage them to find the root cause as much as possible and solve the problem in an elegant way. As the saying goes, sweat more in peacetime and bleed less in battle.

What aspects of your membership do you value more in the recruitment process?

Chen Su: Interview time mainly depends on technical depth and understanding.

Generally speaking, job seekers with good school background have a higher probability of passing the interview, but we don't just look at school background. During the interview process, the authenticity of the candidate's R & D experience, the technical depth reflected, whether there is a habit of paying attention to technical forums and reading open source projects, these are the points we focus on.

During the probation period, we will focus on the ability to analyze and solve problems and the ability to withstand pressure.

How do you motivate your team members? What are the main incentives?

Chen Su: At the beginning of 2018, we established a quarterly star selection system to reward employees who have made outstanding contributions or made significant progress each quarter. Over the past year or so, quarterly stars have been awarded to both early tech backbones and new colleagues. Through this form, we select outstanding talents, give more responsibilities and give corresponding rewards.

As the company grew, we introduced performance appraisal this year, which is a results-oriented assessment of employees 'actual output as the main basis for promotion and salary adjustment.

Technicians choose a company, in addition to income factors, technical growth is also an important consideration. DataPipeline encourages employees to actively participate in the development of open source projects and gives employees dedicated time to do open source related work. The company is also willing to invest resources to hold or participate in technical forums, so that employees can communicate with experts in the field, which is also one of the important reasons why everyone thinks the technical atmosphere of the company is very good.

Do you encourage team members to innovate? Mainly through what kind of method?

Chen Su: Innovation is the foundation for DataPipeline to survive in the highly competitive data integration market.

Our company's products are based on the open source framework Kafka Connect. In order to adapt to business needs, we have made a lot of modifications and feature enhancements based on this open source framework, including end-to-end data synchronization consistency, batch flow integration, source change detection and automatic adaptation, and optimized the framework's task scheduling mechanism.

Any team member who has a good idea or finds something worth improving is free to organize a discussion meeting and invite relevant colleagues to discuss the proposal together. When the proposal is approved, it will be scheduled according to priority and incorporated into the R & D plan. We pay special attention to the optimization points proposed by employees themselves, and we also give priority to employees who have contributed in this regard when selecting quarterly stars and performance evaluations.

Third, understand the core needs of customers and respect the development rules of the industry

Can you share your entrepreneurial experience? What impressed you most during the entrepreneurial process? Did you learn anything from it?

Chen Su: In 2010, after I graduated from Dr. ×××, my first job was as an algorithm engineer for precision marketing platform in China Mobile Research Institute, and later I was gradually promoted to project manager and technical director of user behavior laboratory.

At the beginning of 2015, I left China Mobile Research Institute and started my first venture with friends. We built an online education company, focusing on English training. The initial idea was to use machine learning technology to help users improve their learning efficiency. The company started from 36 krypton incubator, obtained angel round financing, and became the second phase graduation enterprise.

In the early days, in order to get traffic, we tried to do some drainage functions, such as real-time TOEFL test place inquiry and test place reservation. With these drainage apps, our user activity is growing rapidly, and the app has long been ranked at the top of the App Store and major domestic Android market segments, so we have successfully obtained Series A financing.

After round A, we started to do traffic conversion, developing a series of paid courses and supporting adaptive learning systems. In order to improve the live interactive experience, we developed a live broadcast system that does not rely on video streaming, which can broadcast courseware with extremely low bandwidth requirements and reduce the impact of network congestion. On Teacher's Day 2016, this system was officially launched. Over the next year or so, we began to validate our business model. Unfortunately, revenue has not improved significantly. At the end of 2017, the company was acquired by another online education company. Since then, I have joined DataPipeline, moving from ToC domain to ToB.

For the first time, what impressed me most was that the team made an App in 45 days and completed the angel round of financing in 90 days. This sense of accomplishment and happiness is unparalleled. I have learned that no matter how scarce resources are, a group of partners with a common vision can overcome difficulties and explode with endless fighting power. But then the failure of commercialization made me realize that the market is cruel and that technological innovation alone is not enough to keep a startup alive. You need a deep understanding of the core demands of customer groups and respect for the industry's own laws in order to achieve commercial success.

Because of the first experience, I believe that choice and effort are equally important, so I chose to start a second venture in DataPipeline. Although the company has achieved certain results in the past three years, the whole team still has a strong sense of survival crisis. Even better, the team is always open to discussing these issues: Is there a problem with customer selection, is the product not rich enough, or is there a lack of depth in some aspects? I am very pleased that there are no complaints and excuses among team members, some are just finding problems and solving problems together.

What is your biggest challenge right now? Is there a solution?

Chen Su: There is a common challenge to do ToB enterprise services in China: the contradiction between service productization and customer demand personalization.

We found that it was difficult to fully meet the needs of customers, especially large customers, solely by products. Some common new requirements can be solved by product iteration, but other system integration with customers and some specific business logic requirements should be solved by customized development by field implementation team.

To resolve this contradiction, we tried the following. First, open the product interface to the outside world, so that customers can integrate DataPipeline with their own systems, including the task behavior of DataPipeline can be controlled through the existing scheduling system; Second, we provide secondary development tools, and when upstream and downstream connector requirements are not provided with standardized components for the time being, they can be quickly developed by customers or our resident team; Finally, we standardized operations processes as much as possible and developed a set of troubleshooting tools that allowed customers to quickly locate whether problems were coming from DataPipeline or custom development.

The ultimate goal of these attempts is to achieve self-service operation of products and minimize the labor and time costs of operation and maintenance services.

What is the most fulfilling problem you have ever solved?

Chen Su: Compared with technical problems, I think how to coordinate R & D and customer service is a bigger problem.

In the early days of DataPipeline, there were very few people, and R & D, pre-sales and operation and maintenance were carried by several developers. Admittedly, over time, this pattern shows its effectiveness: developers are most familiar with business logic and code, they can answer customer questions directly, and when necessary, they can write code to solve bugs and adaptation problems.

As the number of customers grows, this coarse division of labor model increasingly exposes its problems. First, products are becoming more and more complex, and the control requirements for R & D progress and quality are becoming stricter. Developers are frequently interrupted by customer support, which seriously affects efficiency; second, some developers are not good at communicating with customers, which is easy to misunderstand; third, most field problems can be located and solved according to a standard process. From the perspective of cost, it is not economical for developers to do on-site investigation.

So we started recruiting pre-sales and operations teams to try to separate R & D personnel from customer service. But there is a new challenge: how to effectively transfer knowledge and skills to pre-sales and operations teams. DataPipeline product positioning determines that our pre-sales engineers usually need to communicate technical details with customers, while operation and maintenance engineers need to quickly locate the link where problems occur.

For example, customers feedback that data synchronization is slow, which may occur in upstream reads, Kafka IO, downstream write destinations, etc., or cluster task scheduling falls into an unstable state for some reason. Operation and maintenance engineers should be able to identify, solve problems at the operation and maintenance level, and coordinate with R & D personnel to locate and solve problems at the suspected code level. In order to enable the pre-sales and operation and maintenance teams to serve customers relatively independently, we have set the following rules:

(1) After all pre-sales and operation and maintenance engineers enter the company, they will be trained intensively from the aspects of product use and technical principles. Both pre-sales and operation and maintenance are required to be able to answer questions about product usage, familiar with product core technology points, such as high availability, data consistency, dynamic scaling, performance impact factors, use of advanced cleaning, etc. It is required to be able to conduct POC deployment at customer sites before sales, and to conduct product performance optimization and troubleshooting without R & D intervention;

(2) R & D personnel can only contact customers directly if they are determined to be bugs and performance defects. The rest of the questions will be answered by pre-sales personnel or operation and maintenance personnel. In case of unknown problems, pre-sales personnel and operation and maintenance personnel can help R & D requirements, record them and put them into the knowledge base.

In this way, we have basically liberated R & D personnel from daily customer service and further improved customer service satisfaction.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.