Shulou(Shulou.com)06/03 Report--

Every time there are students ask Jiami Valley big data please add links to describe whether there is a simple quick way to learn, I think if there is really a so-called quick way that big data this discipline technology also has no threshold anyone can learn, that for you want to get a high salary what meaning, if you are small white or honestly read the following content.

The following is a mind map that I have compiled. The content is divided into several large blocks, including distributed computing and query, distributed scheduling and management, persistent storage, programming languages commonly used in big data, etc. There are many open source tools under each category. These are the things that apes love and hate as big data programs.

Big Data Needs Languages

Java

Scala

Python and Shell

distributed computing

What is distributed computing? Distributed computing is the study of how to divide a problem that requires a lot of computing power into many small parts, then distribute these parts to many servers for processing, and finally combine these calculations to get the final result.

distributed storage

Traditional network storage systems use centralized storage servers to store all data. The IO capacity of a single storage server is limited, which becomes the bottleneck of system performance. At the same time, the reliability and security of the server cannot meet the requirements, especially for large-scale storage applications.

Distributed storage system is to store data on multiple independent devices. The system structure is scalable, which uses multiple storage servers to share storage load and location server to locate storage information. It not only improves the reliability, availability and access efficiency of the system, but also is easy to expand.

Distributed scheduling and management

Now people seem to be very keen to talk about "decentralization," perhaps the trend brought about by blockchain. But centralization is important in big data, at least for now.

Distributed cluster management requires a component to allocate scheduling resources to each node, this thing is called yarn;

There needs to be a component to solve the "lock" problem in a distributed environment. This thing is called zookeeper;

There needs to be a component to record task dependencies and schedule tasks on a regular basis. This thing is called azkaban.

Of course, these "things" are not unique. In fact, there are many substitutes. I only gave a few common examples here.

As we all know, the technology of big data is changing with each passing day. As a program ape, if you want to maintain competitiveness, you must constantly learn. There is no shortcut to speed up. Only by firmly learning the knowledge step by step is the optimal solution.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.