Shulou(Shulou.com)06/03 Report--

For many technical teams, they will more or less encounter some doubts and challenges in the process of building an intelligent data architecture. after many practices, some teams have successfully explored a way to build an intelligent data architecture. so how did they implement this technology? At a recent push technology salon in Chengdu Station, several architects opened the "brainstorm time" of data technology at the scene.

Research and Development Manager of Nokia Network Chengdu R & D Center

Liu Peng's R & D Management in the era of data growth

Enterprises need to be well prepared in R & D and management when dealing with the huge challenges brought by data growth.

In terms of research and development, improve the scalability of data storage; in management, in addition to increasing the management of people and devices, efforts should be made to build a team with data-driven leadership, so that the decisions in the team are based on data. at the same time, create a data-centric culture.

In big data's era, to become a data-driven leader (Data-Driver Leader), you need to have three moves.

The first trick: pay attention to the economic benefit data of the R & D team (Take an Economic View)

The R & D team is not only the cost center, they also have the ability to create economic benefits. as a data-driven leader, they should pay attention to whether the relevant technical practices can bring economic benefits to the company in their daily work.

Second trick: visualize team data (Transparency)

In an R & D team, department leaders need to visualize the data so that everyone on the team knows something about the core data, so that everyone has the ability to solve problems when they arise.

The third trick: based on data, timely and rapid feedback (Fast Feedback)

In the process of team operation, as a leader, we should not only make corresponding plans, but also constantly analyze data search problems, and provide feedback to members based on data and KPI.

Although data-driven leadership plays an important role in driving corporate decision-making, as the company expands, departmental walls and local optimization (Sub-optimization) become more and more obvious, and it is necessary to break the departmental wall so that departments and teams can work together around a common goal to achieve the goal of maximizing benefits.

A push data architect

Yuan Kai's Construction and practice of Machine Learning platform

The routine process of machine learning work: operators should first transform business problems into problems that can be solved by machine learning, then carry out data collection, cleaning and aggregation, and then open data exploration and feature engineering. after the above steps, we can get all the factors needed to predict things. At this time, operators can choose different algorithms, and train the algorithms to get the corresponding application model. Finally, operators also need to use real data for verification to ensure the feasibility of the model.

As a multi-domain cross-discipline, machine learning is an effective tool to solve many practical problems. Through machine learning, we construct unique cold, hot and temperature tags to analyze the basic attributes and behavioral characteristics of different groups, depict accurate portraits of users, and finally apply them to intelligent push and precision marketing.

In order to complete the construction of machine learning platform, we need to pay attention to three main points:

1. Only the end-to-end platform construction can really produce value, at the same time, the data and code precipitation of feature engineering need to share operation.

two。 Proceed from the effectiveness, focus on the pain points, do not blindly follow the industry, and do a good job in the integration of systems and training.

3. Cautiously introduce new technology stacks.

In order to avoid the common problems in the construction of machine learning platform, the push modeling platform will not only provide the corresponding IDE and the corresponding management system, but also provide standardized ID matching services and data extraction services to reduce the repetitive work of engineers. In addition, the packaging and deployment services and follow-up monitoring services provided by GE push can also help enterprises ensure the smooth operation of the platform.

Jumei big data Senior engineer

He Peng's big data 3.0 Stream Computing and Intelligent decision-making

During the period of big data 3.0, the first generation of Hadoop and the second generation of Spark memory computing, early flow computing and artificial intelligence flow computing coexisted.

Early stream computing has five difficulties: strong consistency, data disorder and delay. The emergence of Flink effectively solves these problems. At the same time, Flink also has six advantages, such as real-time incremental computing, SQL support and CEP support.

With the development of stream computing, it already has the powerful function of CEP, which is also the key factor to support the intelligence of stream computing. In real life, many complex scenarios can not be judged by explicit rules (traditional programming is rule programming, instruction programming and if else programming). For example, you can't use if else to write how to judge which picture is a leaf, and you can't use limited rules to write how to judge which human operation and which are robot brushing in the data stream, so you need to use machine learning model to detect and match. At the same time, ML and stream computing are needed together.

In general, there are two main scenarios of ML model: classification and regression. They can check for stateless f = fx in abstraction. Stateless model (stateful model typically represents rnn), but UDF corresponds to it in SQL semantics (stateless). It is necessary to put the model into the stream processing system, that is, to register the tensorflow PB model model as udf. After completing the above steps, the data flow will enter the ML detection phase.

When using stream computing related functions, it needs to be supported by a powerful platform to implement SQL development, authorization and other operations.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.