Shulou(Shulou.com)06/03 Report--

Big data's thought on the overall Construction of platform

What is other people's big data platform like?

What is other people's big data platform like? If you have participated in some technology sharing forums or meetings, large and small, it should not be difficult to find that in all kinds of new PPT, such as "XXX big data platform practice invincible practical information sharing", when it comes to the technical components of big data platform, most of them. Will give a system architecture diagram that is more or less the same.

In this architecture diagram, all kinds of log and DB data acquisition components, storage and computing engine, monitoring and scheduling system, no matter what the actual application in practice, there is no lack of all components on the diagram, except for the addition or subtraction of individual components, each company's big data platform does not look much different.

So, if you want to ask what the infrastructure diagram of big data's platform looks like, you don't have to draw it yourself, but show it directly with HortonWorks's HDP distribution package, as shown in the following figure.

The cultivation of service consciousness and product thought

The function orientation of big data platform team

If we want to talk about the service of big data platform and evaluate the service level of big data platform, we must first discuss what is the functional orientation and service object scope of big data platform. Unfortunately, this is not a question with standard answers.

● in some companies, big data's team is only responsible for the development and operation of basic components, providing business parties with services in the form of SDK, component packages or clusters.

● in some companies, tools, platforms based on basic components, etc., are responsible for by a special tool team, layer by layer division of labor, cross-service between teams.

● in some companies, different business teams build independent business systems vertically on top of the basic platform components, and the platform basic component developers serve the upper business system developers.

● in some companies, big data's team takes all the links from bottom to top, from cluster operation and maintenance to the final output of specific terminal business data, and is responsible for the users who use the data at the terminal.

Workflow (job) scheduling system

Among all the components of big data platform, workflow scheduling system (Workflow Scheduler) is undoubtedly one of the most important core components, and it is an indispensable part of any big data development platform which has a little scale and is not simply done.

According to the specific context, address habits and functional reference scope, workflow scheduling system is often called job scheduling system (Job Scheduler), task scheduling system, workflow job scheduling system, or simply referred to as scheduling system in agreed scenarios. Below, we may also mix these terms as needed without causing ambiguity.

As a system with relatively complex business environment, workflow scheduling system involves a variety of contents, a variety of scenarios, and different solutions, so it is a system that requires both theory and practice.

Security and authority control

Doesn't the rights management of big data platform sound like the management of users and passwords? First find a database to store the mapping between the two, then find a place to record what everyone can do, and finally verify it when needed. Is there anything worth talking about if we don't discuss various encryption and decryption principles and algorithms?

In fact, if you have really come into contact with the work in this area, you will soon find that the rights management under the environment of big data platform is a nerve-racking hot potato, both at the technical level and at the product level. It is not only a technical issue, but also a business issue, and it may even be a philosophical issue of interpersonal communication and weighing the gains and losses of interests.

How to be a bad big data platform engineer

Happy families are all the same, and each unhappy family is unhappy in its own way.

Originally wanted to start from the perspective of how to become an excellent big data platform development engineer, but thought about it carefully, from this point of view, this topic is too simple! Although I am not possessed by a great god such as Jeff Dean, I am embarrassed to think that I am qualified to instruct others. But who doesn't know how to be reasonable?

For example, to speculate in stocks is to buy low and sell high. Play the Internet, is not to pull traffic to cash it? If you want to become an excellent big data platform development engineer, as long as you attach equal importance to both depth and breadth, delve into technology, understand products, build architecture, and understand Bug, it will be all right.

Since the reason is so simple, do we need more explanation? And we probably either crushed Buffett easily, or already took off smoothly in the tuyere!

Yes, excellent people are similar, which is too boring to say.

A total of 10 chapters of this book, from big data's basic platform construction ideas to the development of big data programmers, people without overall vision will not be able to understand the true meaning of this book! Limited to too much space, so, the editor will not introduce them to you one by one, and the rest depend on everyone to explore its beauty and wisdom. Therefore, partners who need this [big data platform Infrastructure Guide] can forward this article to follow the editor, and the private message editor "Learning" to get the way to get it!

In short, life is a journey to solve problems, anxiety is inevitable, but a moderate sense of crisis is not necessarily a bad thing. It is important not to let anxiety unduly affect your judgment and try to give yourself the ability to take the initiative to choose the future.

No matter whether you are engaged in the development of big data platform in the future or not, I sincerely hope that you can face the problems in your work and life, find solutions objectively and calmly, and win a full and valuable life for yourself with the correct methodology and values.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.