Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to understand Fedlearner. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Fedlearner

How is the headline open source Fedlearner different from the federal machine learning platforms of Huawei and WeChat that I have analyzed before? It is mainly reflected in the following aspects:

Production: Fedlearner has a large number of js and Html modules in its code, which is also the first time we can see what the federal machine learning platform looks like and what it needs to look like if it is made into a product.

Business diversification: previously, Huawei and WeChong put more emphasis on the landing of federal machine learning in the risk control business. The headlines began to emphasize the landing of federal learning in referrals, advertising and other businesses, and gave clear data to increase the effectiveness of advertising in a certain education sector by 209%.

Exportability: if previous federated machine learning platforms were introduced more theoretically, this time byte Fedlearner emphasizes outputability, such as quickly pulling up and managing clusters through the deployment model of K8S in order to maintain environmental consistency between both sides of federation modeling. This is a technical preparation for the export service of ToB.

The following introduces some of the work of Fedlearner in these three aspects.

Fedlearner production work

Take the recommendation advertising business as an example, the advertisers and Taiwan side of the federal machine learning platform should manage a set of model display services and model training services respectively.

Two sets of protocols are needed to ensure the federation modeling of the customer, one is the issue of data consistency. For example, in a vertical federated learning scenario, the user clicks an advertisement on the page, and the platform and the advertiser each capture part of the log. How to ensure the consistency of the logs captured by the two parts in real time and assemble them into training samples requires a set of real-time data sample splicing protocol.

Another protocol is the multi-party data security protocol. For example, the two business sides of AB, A has 400 million users and B has 300 million users. How to find the cross users of An and B in some way and not let An and B guess each other's data requires a set of multi-party data security protocol.

Based on the above two sets of protocols, in the process of joint modeling, GRPC communication is used and TensorFlow is used to exchange gradients between the two sides for joint modeling.

Business diversity

The biggest business scenario of federal machine learning is recommendation advertising, which I predicted in my article a year ago. Sure enough, the headlines put special emphasis on the application of recommended scenarios. He mentioned that the recommendation service is more suitable for the neural network algorithm and the risk control business is suitable for the tree algorithm. The author also agrees with this view, because the risk control needs to be highly explainable, and the tree algorithm naturally meets this demand. However, the recommendation business does not require high interpretability of the model, and the complexity of the neural network algorithm can fully ensure the accuracy of the recommendation sorting algorithm.

The head of the Fedlearner business gave a set of numbers to prove the landing effect of federal machine learning in the recommendation business.

This array is very persuasive. In fact, for new technology, the barrier we often face is not technical problems, but how to prove the value of the business. It takes the first person to eat crabs in order to promote the landing of new technology in the industry. Federal machine learning has a bright future in the recommendation advertising business.

Exportability

Fedlearner adopts a set of cloud native deployment solutions. The data is stored in HDFS and the system data is stored in MySQL. Manage and pull tasks through Kubernetes. The training task of each Fedlearner requires both participants to pull up the K8S task at the same time, through the unified management of Master nodes, and the establishment of Worker to achieve communication.

This solution fully takes into account the data warehouse compatibility of current users doing recommendation business, because the data warehouse system of most customers is still Hadoop ecology, and the data is stored in HDFS. At the same time, the use of K8S maximizes the consistency of the computing engine environment between the two sides of joint modeling.

The above is the editor for you to share how to understand Fedlearner, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.