Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you what is the big data processing methodology of Google engineers. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

There is no doubt that Google is recognized as the ancestor of big data. Nowadays, when many people mention big data, they still stay in the era of the "troika" initiated by Google: Google FS, MapReduce, and BigTable. In fact, the troika is no longer the top of the wave.

In recent years, the development of big data's technology, whether it is technical iteration or the prosperity of the ecosystem, is far beyond our imagination. From Spark to become a part of Hadoop ecology, to Flink came out of nowhere to challenge Spark to become a new star in big data's processing field, and now Google is determined to dominate the world with Apache Beam. The development of big data's technology can be described as ups and downs.

Rich tools and prosperous ecology also make it more difficult for developers to choose the right tools. You can imagine the amount of work and complexity required to integrate open source frameworks, tools, class libraries and platforms. The choice and use of technology is also a headache for big data developers.

When communicating with Google Brain engineers before, he mentioned that in big data's field, there are too few developers who can figure out the technology, and the technology VP of some small and medium-sized companies are often in a state of "catching up with the fashion of technology", not to mention ordinary developers. There are the following common misunderstandings in dealing with big data:

1. Underestimated the importance of data processing.

There is no high-quality data processing, artificial intelligence only artificial intelligence. For example, Google made such a mistake in semantic understanding that it was not until it was surpassed by a small German company that it realized the importance of high-quality data tagging and processing.

two。 The importance of data processing engineers in organizational architecture is underestimated.

Jesse Anderson, a leading figure in big data's field, has done a study that a reasonable organizational structure of an artificial intelligence team requires 4gamer and 5 data processing engineers. In fact, even if he is a front-end engineer, a lot of work is still data processing. Unfortunately, many teams don't realize this.

3. The complexity of larger data processing is underestimated.

Many people have not yet encountered a "large-scale" problem, so it is easy to think too simply. Google has many excellent candidates who can solve common programming problems very well, but as long as they ask how to design the system when the data scale becomes larger, the answer is often not satisfactory.

4. Overestimated the difficulty of data processing.

On the one hand, we need to realize that there are complex factors in large-scale data processing. But on the other hand, with the right tools and technical concepts, it is not difficult to get started with data processing. At Google, many fresh students can easily cope with hundreds of millions of data after six months on the job.

In order to help you master practical large-scale data processing technology more accurately and deeply than others, and even reach the level of Silicon Valley first-line system architect.

Briefly mention Google Brain (Google brain): the team's projects include an image enhancement system using neural networks, a learning framework for Google neural machine translation, and robots that learn new skills automatically through machine learning. Google Brain technology is used in the speech recognition system of Android operating system, the photo search of Google+ and the video recommendation system of YouTube.

The above is the Google engineer big data processing methodology shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.